Dialogue text processing method and device, computer equipment, storage medium and product

Abstract

The embodiment of the invention discloses a dialogue text processing method and device, computer equipment, a storage medium and a product, and relates to the field of artificial intelligence. The method comprises the steps that a dialogue text is acquired, the dialogue text is extracted from dialogue audio, and the dialogue text comprises speaker texts corresponding to all speakers; inputting the dialogue text and the at least two kinds of prompt information into a large language model to obtain a corrected dialogue text output by the large language model; wherein different kinds of prompt information in the at least two kinds of prompt information are used for prompting and correcting errors of different levels in the dialogue text, the large language model is obtained based on the at least two kinds of prompt information and fine adjustment of the sample dialogue text, and the sample dialogue text has the errors of different levels. By adopting the method provided by the invention, the corrected dialogue text with relatively high quality can be obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, computer device, storage medium, and product for processing dialogue text. Background Technology

[0002] Speaker separation is a task that involves separating different speakers appearing in audio or video to obtain speaker texts for each speaker. Speaker texts can be used in various scenarios such as creating memorable quotes from videos and general editing.

[0003] In related technologies, speaker separation techniques can be used to achieve speaker separation tasks. For example, OSD (Overlapped Speech Detection) technology can be used to detect audio segments with multiple speakers overlapping, and SCD (Speaker Change Detection) technology can be used to detect audio segments where the speaker changes. Using these detection results, audio segments corresponding to different speakers can be separated, and then ASR (Automatic Speech Recognition) technology can be used to convert the audio segments into text, thus obtaining the speaker text corresponding to each speaker.

[0004] However, the accuracy of the speaker's text obtained using the above methods is not high and further optimization and correction are needed. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, storage medium, and product for processing dialogue text. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a method for processing dialogue text, the method comprising:

[0007] The dialogue text is obtained from the dialogue audio and includes the speaker text corresponding to each speaker.

[0008] The dialogue text and at least two types of prompts are input into the large language model to obtain the corrected dialogue text output by the large language model.

[0009] The different prompts in the at least two types of prompts are used to prompt correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on the at least two types of prompts and the sample dialogue text, which contains errors at different levels.

[0010] On the other hand, embodiments of this application provide a dialog text processing apparatus, the apparatus comprising:

[0011] The acquisition module is used to acquire dialogue text, which is extracted from dialogue audio and includes speaker text corresponding to each speaker.

[0012] The correction module is used to input the dialogue text and at least two kinds of prompt information into the large language model to obtain the corrected dialogue text output by the large language model.

[0013] The different prompts in the at least two types of prompts are used to prompt correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on the at least two types of prompts and the sample dialogue text, which contains errors at different levels.

[0014] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the method described above.

[0015] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the method described above.

[0016] On the other hand, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A terminal's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the terminal to perform the methods provided in the various optional implementations of the above aspects.

[0017] In this embodiment of the application, the dialogue text extracted from the dialogue audio may contain errors at different levels. Therefore, by fine-tuning the large language model with sample dialogue texts containing errors at different levels and at least two kinds of prompts, the large language model can be made capable of correcting errors at different levels. By inputting the dialogue text and at least two kinds of prompts into the large language model, the errors at different levels in the obtained corrected dialogue text are corrected, thereby reducing the text error rate and obtaining a high-quality corrected dialogue text. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the extraction of dialogue text using related technologies provided in an exemplary embodiment of this application;

[0020] Figure 2 This is an application architecture diagram provided by an exemplary embodiment of this application;

[0021] Figure 3 This is a flowchart of a dialogue text processing method provided in an exemplary embodiment of this application;

[0022] Figure 4 This is a schematic diagram of dialogue text provided in an exemplary embodiment of this application;

[0023] Figure 5 This is an exemplary embodiment of the present application, which provides a schematic diagram of correcting errors at least two levels in dialogue text through two correction stages.

[0024] Figure 6 This is a schematic diagram illustrating the correction of errors in word granularity provided in an exemplary embodiment of this application;

[0025] Figure 7 This is a flowchart provided by an exemplary embodiment of the present application for fine-tuning a large language model to improve the ability to correct word granularity errors;

[0026] Figure 8 This is a schematic diagram of an exemplary embodiment of the present application, illustrating the fine-tuning of a large language model to improve the ability to correct word granularity errors;

[0027] Figure 9 This is a schematic diagram illustrating an exemplary embodiment of the present application providing a method for correcting errors at the statement granularity.

[0028] Figure 10 This is a flowchart provided by an exemplary embodiment of the present application for fine-tuning a large language model to improve the ability to correct statement-level errors;

[0029] Figure 11 This is a schematic diagram of an exemplary embodiment of the present application, illustrating the fine-tuning of a large language model to improve the ability to correct statement-level errors;

[0030] Figure 12 This is a schematic diagram illustrating the correction of dialogue text through a first correction stage and a second correction stage, provided in an exemplary embodiment of this application.

[0031] Figure 13 This is a structural block diagram of a dialog text processing apparatus provided in an exemplary embodiment of this application;

[0032] Figure 14 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] First, the terms used in the embodiments of this application will be introduced.

[0035] Large Language Model (LLM): An artificial intelligence model designed to understand and generate human language. For example, a large language model can include a Generative Pre-trained Transformer (GPT) model and its descendant models. The training process of a large language model consists of two stages. The first stage is unsupervised pre-training using a large-scale unlabeled text corpus. The second stage is supervised fine-tuning using labeled text corpora for specific downstream tasks.

[0036] Prompt information: Information input to improve the capabilities of large language models. For example, prompts or instructions can be used. Prompt information can enhance the output quality of large language models.

[0037] Voice Activity Detection (VAD) is a technology that automatically detects significant segments of speech in audio. For example, VAD can detect whether someone is speaking in an audio file, as well as the start and end times of the speech.

[0038] Overlapped Speech Detection (OSD): A technique for detecting audio segments in which multiple speakers overlap.

[0039] Speaker Change Detection (SCD): A technique for detecting audio segments in which the speaker changes.

[0040] Automatic Speech Recognition (ASR): A technology that converts audio segments into corresponding text.

[0041] Among related technologies, techniques such as VAD, OSD, SCD, and ASR can be used to extract speaker text from audio for different speakers.

[0042] See Figure 1 , Figure 1 This is a schematic diagram illustrating the extraction of dialogue text using related technologies provided in an exemplary embodiment of this application.

[0043] like Figure 1 As shown, the relevant technology first extracts features using an audio input feature extraction network (such as a one-dimensional convolutional neural network); then, it uses VAD (Visual Audio Deposition) technology to locate audio intervals where speakers are actually speaking; next, it uses OSD (On-Side Deposition) technology to detect overlapping speaker intervals and SCD (Speaker Switching) technology to detect speaker switching intervals; then, based on the above detection results, it separates audio segments corresponding to each speaker from the audio intervals, extracts audio segment features using a segment feature extraction network (such as a one-dimensional convolutional neural network), and clusters them based on the similarity of the audio segment features to determine the audio segments corresponding to the same speaker. Finally, it uses ASR (Automatic Speech Recognition) technology to convert the audio segments corresponding to each speaker into speaker text to form dialogue text.

[0044] However, the dialogue text extracted using related technologies may contain various textual errors. For example, the dialogue text extracted using related technologies may contain errors in speaker identification in the speaker's text, as well as missing words, added words, typos, and disordered word order.

[0045] See Figure 2 , Figure 2 This is an application architecture diagram provided in an exemplary embodiment of this application. The application architecture diagram includes a terminal 210, a server 220, and a large language model 230.

[0046] Optionally, terminal 210 can be a portable mobile terminal, such as a smartphone, tablet computer, laptop computer, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, etc. The terminal can also be an in-vehicle terminal, desktop computer, smart TV, etc. The terminal may also be referred to as user equipment, portable terminal, or other names. This specification uses a smartphone as an example for illustration, but this does not constitute a limitation.

[0047] In some embodiments, terminal 210 sends the recorded dialogue video to server 220. Server 220 sends the corrected dialogue text back to terminal 210, so that terminal can add subtitles to the dialogue video based on the corrected dialogue text.

[0048] Server 220 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0049] In some embodiments, server 220 may acquire dialogue audio based on dialogue video and extract dialogue text from the dialogue audio to pass to large language model 230. Large language model 230 corrects errors at different levels in the dialogue text based on prompts, obtains the corrected dialogue text, and returns it to server 220.

[0050] Large language models230 refer to machine learning models that are pre-trained on large-scale text corpora. For example, generative pre-trained transformer (GPT) models, including GPT-1, GPT-2, GPT-3, GPT-3.5, GPT-4 and their subsequent iterations, or any other possible large-scale pre-trained models.

[0051] The training process of the large language model 230 is divided into two stages. The first stage is unsupervised pre-training using a large-scale unlabeled text corpus. The second stage is supervised fine-tuning using labeled text corpus for specific downstream tasks.

[0052] It should be noted that the above embodiments only illustrate the general architecture of the implementation environment. The system may also include more or fewer components, or combine certain components. This embodiment does not limit this.

[0053] To obtain high-quality corrected dialogue text, this application proposes a dialogue text processing method.

[0054] See Figure 3 , Figure 3 This is a flowchart of a dialogue text processing method provided in an exemplary embodiment of this application. In some embodiments, the method is performed by a computer device. Optionally, the computer device may be a terminal or a server. The method includes the following steps.

[0055] Step 301: Obtain the dialogue text. The dialogue text is extracted from the dialogue audio and includes the speaker text corresponding to each speaker.

[0056] Optionally, the dialogue audio can be the audio of the conversation between individual speakers extracted from the video. In some embodiments, the dialogue audio can be the audio of the conversation between individual speakers extracted from a short video or a long video. For example, dialogue audio extracted from a short video on a social media platform, or dialogue audio extracted from long video content captured by a terminal.

[0057] Optionally, the dialogue audio can also be audio of conversations between speakers recorded by computer equipment through a microphone device.

[0058] Optionally, the conversation audio can also be obtained from the user's dialogue records with customer service in the intelligent customer service system. For example, the conversation audio is the recording of a user's call to customer service.

[0059] Optionally, the conversation audio can also be audio from a voice call or video call between the user and another user.

[0060] Optionally, the dialogue audio can also be audio obtained from the Internet (e.g., from a podcast or talk show).

[0061] In addition, the dialogue audio can also be audio recordings of conversations between various speakers obtained through various possible means. It should be noted that during the collection of user-related data (such as call audio, recorded audio, or captured video data), this application may display a prompt interface, pop-up window, or output voice prompt information. This prompt interface, pop-up window, or voice prompt information is used to inform the user that their relevant data is being collected. This ensures that the application only begins the steps related to acquiring user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps related to acquiring user-related data end, meaning no user-related data is acquired. In other words, the information (including but not limited to user device information and user personal information), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the call audio, recorded audio, or captured video data involved in this application were all obtained with full authorization.

[0062] Dialogue text is extracted from the audio of a dialogue and consists of the speaker's text.

[0063] In one possible scenario, the dialogue text extracted from the audio may be correct or it may contain errors.

[0064] See Figure 4 , Figure 4 This is a schematic diagram of dialogue text provided in an exemplary embodiment of this application.

[0065] like Figure 4 As shown, in the correct dialogue text 410, the three speakers (Speaker1, Speaker2, and Speaker3) each have their corresponding speaker text.

[0066] In the erroneous dialogue text 420, the word "we two" at the end of the extracted Speaker3's text "Eh? How did you two know each other?" was incorrectly extracted from the beginning of the next speaker's text; the word "see" at the beginning of the extracted Speaker2's text "Yes, yes, it's fate that we met here." was incorrectly extracted from the end of the previous speaker's text.

[0067] In the erroneous dialogue text 430, the original speaker text "Hello!" corresponding to Speaker2 was incorrectly extracted as the speaker text corresponding to Speaker1; the original speaker text "Eh? How did you two meet?" corresponding to Speaker3 was incorrectly extracted as the speaker text corresponding to Speaker1; and the original speaker text "It's been many years since we last met!" corresponding to Speaker1 was incorrectly extracted as the speaker text corresponding to Speaker2.

[0068] In some embodiments, the dialogue text consists of speaker text corresponding to at least one speaker. For example... Figure 4 In the middle, the correct dialogue text 410 consists of the speaker texts corresponding to the three speakers.

[0069] Regarding the acquisition of dialogue text, one possible implementation is that the computer device can acquire the dialogue text using technologies such as VAD, OSD, and SCD. Another possible implementation is that the computer device can also use an audio-to-text tool to first acquire all the text in the dialogue audio, and then segment the text into speaker texts based on the text content and audio features, thus forming the dialogue text.

[0070] In another possible implementation, the dialogue text can be extracted from the dialogue audio by another device, or the computer device can directly obtain the dialogue text from the other device. For example, the computer device is a terminal, the dialogue text can be extracted from the dialogue audio by a server, and the terminal sends a dialogue text retrieval request to the server to obtain the dialogue text.

[0071] In some embodiments, a speaker may correspond to one or more speaker texts.

[0072] Optionally, the speaker's text may be a text consisting of one or more words, or a text consisting of one or more sentences, or a text consisting of one or more paragraphs; there are no restrictions on this.

[0073] In some embodiments, speaker text includes a speaker identifier and text content. For example, a speaker text corresponding to speaker A could be “Speaker A: Long time no see, how have you been?”, where “Speaker A” is the speaker identifier and “Long time no see, how have you been?” is the text content.

[0074] In some embodiments, the speaker text also includes a timestamp identifier to identify the timestamp corresponding to the speaker text in the dialogue audio. For example, the speaker text “Speaker A [08:32-08:36]: Long time no see, how have you been?” indicates that the audio segment corresponding to the text “Long time no see, how have you been?” appears from 8 minutes 32 seconds to 8 minutes 36 seconds in the dialogue audio.

[0075] Step 302: Input the dialogue text and at least two types of prompts into the large language model to obtain the corrected dialogue text output by the large language model; wherein, the different prompts in the at least two types of prompts are used to prompt the correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on the at least two types of prompts and the sample dialogue text, which contains errors at different levels.

[0076] Prompts are used to guide large language models to improve their output quality. Optionally, prompts can include either a prompt or an instruction. A prompt is typically a short text string that guides the large language model to better understand the requirements and generate correct output; an instruction is typically a more detailed text string that guides the model to perform specific operations or complete tasks.

[0077] Optionally, at least two types of prompts can be pre-set prompts or prompts generated by the computer device based on the dialogue text. For example, if the dialogue text contains timestamps, the computer device can generate prompts that adjust word order based on timestamps using the large language model; or, if the dialogue text contains speaker identifiers, the computer device can generate prompts that correct speaker identifiers using the large language model.

[0078] In one possible implementation, a large language model is deployed in the computer device, which can input the dialogue text and at least two kinds of prompts into the large language model; in another possible implementation, the computer device can provide the dialogue text and at least two kinds of prompts to the interface of the large language model, and receive the corrected dialogue text returned by the large language model through the interface.

[0079] At least two different prompts are used to prompt correction of errors at different levels in the dialogue text.

[0080] Optionally, errors at different levels can be errors at different semantic levels within the dialogue text. For example, if the dialogue text is a question-and-answer format, then errors at different levels can include errors at the question level and errors at the answer level.

[0081] Optionally, errors at different levels refer to errors at different text granularities within the dialogue text. Text granularity includes word granularity, sentence granularity, and word order granularity. Errors at different levels can include errors at the word granularity, sentence granularity, and word order granularity levels.

[0082] Optionally, different levels can have a hierarchical relationship. For example only, word order granularity is higher than sentence granularity, and sentence granularity is higher than word granularity.

[0083] In one possible implementation, the computer device first inputs the dialogue text and the corresponding prompts at the lowest level into a large language model, obtaining a corrected dialogue text output by the large language model after correcting errors at the lowest level. Next, the computer device inputs this corrected dialogue text and the corresponding prompts at the next lowest level into the large language model, obtaining a corrected dialogue text output by the large language model after correcting errors at the next lowest level, and so on, until the corrected dialogue text output by the large language model after correcting errors at the highest level is obtained.

[0084] In some embodiments, the computer device may pre-fine-tune the large language model before inputting the dialogue text and at least two types of prompts. In some embodiments, the computer device fine-tunes the large language model based on at least two types of prompts and sample dialogue text containing errors at different levels. The fine-tuning process of the large language model is described in the following embodiments and related descriptions, and will not be repeated here.

[0085] In summary, the dialogue text extracted from the dialogue audio may contain errors at different levels. Therefore, by fine-tuning the large language model using sample dialogue texts with errors at different levels and at least two types of prompts, the large language model can be made capable of correcting errors at different levels. By inputting the dialogue text and at least two types of prompts into the large language model, the errors at different levels in the obtained corrected dialogue text are corrected, thereby reducing the text error rate and obtaining a high-quality corrected dialogue text.

[0086] To further improve the quality of dialogue text and enhance the correction effect of large language models, in some embodiments, the computer device can correct errors at at least two levels in the dialogue text through at least two correction stages.

[0087] In some embodiments, during the first correction stage, the computer device concatenates the dialogue text and the first prompt information and inputs them into the large language model to obtain the first corrected dialogue text output by the large language model.

[0088] The first prompt message is used to prompt correction of the first level of errors in the dialogue text.

[0089] In some embodiments, the first level is the lowest level.

[0090] In some embodiments, during the (i+1)th correction stage, the i-th corrected dialogue text and the (i+1)th prompt information are concatenated and input into the large language model to obtain the (i+1)-th corrected dialogue text output by the large language model, where i is a positive integer.

[0091] The (i+1)th prompt is used to prompt for correction of errors at the (i+1)th level in the dialogue text, and the (i+1)th level is higher than the (i)th level.

[0092] See Figure 5 , Figure 5 This is an exemplary embodiment of the present application, illustrating the correction of errors at at least two levels in the dialogue text through two correction stages.

[0093] like Figure 5 As shown, in the first correction stage, the computer device concatenates the dialogue text 511 and the first prompt information 512 and inputs them into the large language model 520 to obtain the first corrected dialogue text 521 output by the large language model 520. The first corrected dialogue text 521 corrects the lowest level errors.

[0094] In the second correction stage, the computer device concatenates the first corrected dialogue text 521 and the second prompt message 522 and inputs them into the large language model 530 to obtain the second corrected dialogue text 531 output by the large language model 530. The second corrected dialogue text 531 corrects the second lowest level errors.

[0095] This process continues until the final correction stage, where the last corrected dialogue text is obtained from the output of the large language model. This corrects the highest-level errors and is used as the final corrected dialogue text.

[0096] The large language model corresponding to different correction stages can be the same model or different models.

[0097] In this embodiment, since lower-level errors can affect the large language model's correction of higher-level errors, in the earlier correction stage, the lower-level errors are corrected first based on the prompts corresponding to the lower-level errors using the large language model; then in the later correction stage, the higher-level errors are corrected based on the prompts corresponding to the higher-level errors using the large language model, thereby further reducing the error rate of the corrected dialogue text and improving the quality of the corrected dialogue text.

[0098] In some embodiments, different levels of error refer to errors at different text granularities within the dialogue text.

[0099] Among them, the text granularity includes at least two of word granularity, sentence granularity, and word order granularity, and the text granularity corresponding to the (i + 1)-th level is higher than the text granularity corresponding to the i-th level.

[0100] In some embodiments, the first prompt information corresponding to the first correction stage is used to correct errors at the word granularity, and the second prompt information corresponding to the second correction stage is used to correct errors at the sentence granularity.

[0101] In some scenarios, errors at the word granularity include that some words in the speaker's text are misidentified as other words. Exemplarily, errors at the word granularity include typos, such as "就是这样的" being misidentified as "久似这样的".

[0102] In some scenarios, errors at the word granularity include that some words in the speaker's text are missing. Exemplarily, the speaker's text "就是这样的" is wrongly extracted as "就是的".

[0103] In some scenarios, errors at the word granularity include that incorrect words are added to the speaker's text. Exemplarily, the speaker's text "就是这样的" is wrongly extracted as "这个就是这样的".

[0104] A specific scenario for correcting errors at the word granularity is described below.

[0105] In some embodiments, the dialogue text includes the first speaker's text and the second speaker's text, where the second speaker's text is the subsequent speaker's text of the first speaker's text.

[0106] See Figure 6 , Figure 6 which is a schematic diagram for correcting errors at the word granularity provided by an exemplary embodiment of the present application.

[0107] As Figure 6 shown, before the completion of the first correction stage, the dialogue text 610 with errors includes the speaker's text 611 to the speaker's text 614. Among them, the text content of the speaker's text 611 is "诶?你俩怎么认识的我俩?", the text content of the speaker's text 612 is "是同一个小学毕业的!", the text content of the speaker's text 613 is "也已经很多年没!", and the text content of the speaker's text 614 is "见了是的是的,在这碰到真是缘分。"

[0108] After the first correction phase is completed, the first corrected dialogue text 620 includes speaker texts 621 to 624. Among them, the text content of speaker text 621 is "Huh? How do you two know each other?", the text content of speaker text 622 is "We graduated from the same primary school!", the text content of speaker text 623 is "We haven't seen each other for many years!", and the text content of speaker text 624 is "Yes, yes. What a coincidence to meet here."

[0109] In some embodiments, after the first correction phase is completed, the first word at the beginning of the second speaker text in the first corrected dialogue text is the last word of the first speaker text in the dialogue text before the first correction phase is completed.

[0110] For example, in the incorrect dialogue text 610, the first speaker text is speaker text 611 and the second speaker text is speaker text 612; in the first corrected dialogue text 620, the first speaker text is speaker text 621 and the second speaker text is speaker text 622. Then, after the first correction phase is completed, the first word "We two" at the beginning of the second speaker text 622 in the first corrected dialogue text 620 is the last word "We two" of the first speaker text 611 in the incorrect dialogue text 610 before the first correction phase is completed. That is, the word "We two", which should be the first word of the second speaker text, is wrongly extracted to the end of the first speaker text. Through the first correction phase, the error at the word granularity can be corrected, improving the accuracy of the first corrected dialogue text.

[0111] In some embodiments, after the first correction phase is completed, the last word of the first speaker text in the first corrected dialogue text is the first word of the second speaker text in the dialogue text before the first correction phase is completed.

[0112] For example, in the incorrect dialogue text 610, the first speaker text is speaker text 613 and the second speaker text is speaker text 614; in the first corrected dialogue text 620, the first speaker text is speaker text 623 and the second speaker text is speaker text 624. Then, after the first correction phase is completed, the last word "seen" of the first speaker text 623 in the first corrected dialogue text 620 is the first word "seen" of the second speaker text 614 in the incorrect dialogue text 610 before the first correction phase is completed. That is, the word "seen", which should be the last word of the first speaker text, is wrongly extracted to the beginning of the second speaker text. Through the first correction phase, the error at the word granularity can be corrected, improving the accuracy of the first corrected dialogue text.

[0113] In some embodiments, the computer device concatenates the dialogue text and the first prompt information and inputs them into the large language model to obtain the first corrected dialogue text output by the large language model.

[0114] In some embodiments, the first prompt message corresponding to the first correction stage is used to correct errors at the word granularity level. Regarding the specific form of the first prompt message, which is only an example, the first prompt message could be: "I will provide dialogue between multiple characters, some of which may contain errors. You need to deduce and find the errors based on the character names and dialogue context information I provide, and then correct them. Please output the correct character names and dialogue at the end. The text I provide is as follows:\n"

[0115] In this embodiment, during the first correction stage, the dialogue text and the first prompt information are concatenated and input into the large language model. Since the first prompt information is used to correct errors in word granularity, the large language model will extract words at the end of the first speaker's text based on the first prompt information and restore them to the beginning of the second speaker's text, or extract words at the beginning of the second speaker's text and restore them to the end of the first speaker's text. This corrects the errors in word granularity in the dialogue text and yields a first corrected dialogue text with better text quality.

[0116] To improve the ability of large language models to correct errors at the word granularity level, one possible approach is to pre-fine-tune the large language model to optimize its ability to correct word granularity errors.

[0117] See Figure 7 , Figure 7 This is a flowchart illustrating an exemplary embodiment of the present application for fine-tuning a large language model to improve the ability to correct word-level errors. In one possible implementation, the fine-tuning is performed by a computer device or other device. The process includes the following steps.

[0118] Step 710, Data Collection.

[0119] In some embodiments, the computer device acquires a first real dialogue text, which does not contain word-level errors.

[0120] The first real dialogue text can be dialogue text extracted from the sample audio that matches the sample audio and does not contain any textual errors. For example, the first real dialogue text can be completely correct dialogue text extracted manually.

[0121] In one possible approach, the first authentic dialogue text can be collected from script data.

[0122] Step 710 may include steps 711 and 712.

[0123] Step 711: Collect script data.

[0124] Optionally, script data may include video files based on scripts, such as TV series, movies, or variety shows, or audio files based on scripts, such as radio programs, podcasts, or talk shows; there are no restrictions on this.

[0125] Step 712: Extract the script dialogue.

[0126] In one possible implementation, when video files such as TV series, movies, or variety shows contain subtitles, script dialogue corresponding to the video file can be created based on the subtitles.

[0127] In another possible implementation, where audio files such as radio programs, podcasts, and talk shows do not contain subtitles, script dialogue can be obtained based on language-to-text technology and manual correction.

[0128] The extracted script dialogue is the first real dialogue text, that is, the correct dialogue text that does not contain errors at the word level.

[0129] See Figure 8 , Figure 8 This is a schematic diagram of a fine-tuning of a large language model to improve the ability to correct word granularity errors, provided by an exemplary embodiment of this application.

[0130] like Figure 8 As shown, the first real dialogue text 810-1 (script dialogue) extracted from the script data is: "Character A: Mom and Dad, I'm home! Character B: Come and eat, Mom just made your favorite ribs. Character C: Remember to wash your hands first. Character A: I'm not hungry yet, I'm going to play soccer with my friends, you guys eat first. Character B: You've only been in school for a few days and you've already made good friends. Character C: Come back soon!"

[0131] Step 720, text preprocessing.

[0132] Step 720 may include steps 721 and 722.

[0133] Step 721, segment the speaker's text.

[0134] In one possible approach, if the speaker's text contained in the first real dialogue text has punctuation marks, the speaker's text can be segmented based on the punctuation marks.

[0135] For example, if the speaker text in the first real dialogue text is "Speaker B: Come and eat, Mom just made your favorite ribs.", then the text sub-contents "Come and eat" and "Mom just made your favorite ribs" can be obtained by segmenting based on punctuation marks.

[0136] Step 722: Concatenate the speaker's text.

[0137] In one possible approach, when two or more text sub-contents are obtained through segmentation, a speaker identifier can be added to each text sub-content, and the text sub-contents of the same speaker in the same speaker's text can be concatenated by semicolons.

[0138] For example, add speaker identifiers to the text sub-contents "Come eat quickly" and "Mom just made your favorite ribs" to get "Speaker B: Come eat quickly" and "Speaker B: Mom just made your favorite ribs", and then concatenate them with semicolons to get "Speaker B: Come eat quickly; Speaker B: Mom just made your favorite ribs".

[0139] like Figure 8 In the text, the first real dialogue text 810-2 after text preprocessing is "Character A: Mom and Dad, I'm home; Character B: Come and eat; Character B: Mom just made your favorite ribs; Character C: Remember to wash your hands first; Character A: I'm not hungry yet; Character A: I'm going to play football with my friends, you guys eat first; Character B: You've only been in school for a few days and you've already made good friends; Character C: Come back soon."

[0140] Step 730, text modification.

[0141] In some embodiments, the computer device moves the last word of the first sample speaker's text in the first real dialogue text to the beginning of the second sample speaker's text, or moves the first word of the second sample speaker's text to the end of the first sample speaker's text, to form a sample dialogue text.

[0142] The second sample speaker text is the speaker text following the first sample speaker text.

[0143] For example, the first real dialogue text (scripted dialogue) is "Character A: Mom and Dad, I'm home; Character B: Come and eat; Character B: Mom just made your favorite ribs; Character C: Remember to wash your hands first." The modified sample dialogue text is "Character A: Mom and Dad, I'm home; Character B: Come and eat; Character B: Mom just made your favorite; Character C: Remember to wash your hands first." In this modification, the computer moves the word "ribs" from the end of the third speaker's text to the beginning of the fourth speaker's text.

[0144] For example, the first real dialogue text (scripted dialogue) is: "Character A: I'm not hungry yet; Character A: I'm going to play soccer with my friends, you guys eat first; Character B: You've only been in school for a few days and you've already made good friends; Character C: Come back soon." The modified sample dialogue text is: "Character A: I'm not hungry yet; Character A: I'm going to play soccer with my friends, you guys eat first and go to school; Character B: You've only been in school for a few days and you've already made good friends; Character C: Come back soon." In this modification, the computer device moves the first word "go to school" from the third speaker's text to the end of the second speaker's text.

[0145] like Figure 8 In the Chinese text, the modified sample dialogue text 810-3 is: "Role A: Mom and Dad, I'm back; Role B: Come and eat; Role B: Mom just made your favorite dish; Role C: Remember to wash your hands before eating the ribs; Role A: I'm not hungry yet; Role A: I'm going to play football with my friends, you guys eat first and go to school; Role B: You've only been a few days and you've already made good friends; Role C: Come back soon."

[0146] In some embodiments, step 730 may include steps 731 and 732.

[0147] Step 731: Select the speaker text of the first proportion.

[0148] Optionally, the first percentage is a value preset by the computer device, such as 20%.

[0149] Step 732: Modify the words at the beginning or end of the sentence.

[0150] For example, the computer device modifies only 20% of the speaker text in the first real dialogue text, leaving the remaining 80% unchanged, to obtain the sample dialogue text. For instance, it moves the sentence-ending words of 10% of the speaker text to the sentence-beginning of the next speaker text, and moves the sentence-beginning words of 10% of the speaker text to the sentence-ending of the previous speaker text.

[0151] Step 740: Supervised fine-tuning.

[0152] Step 740 includes steps 741 and 742.

[0153] Step 741: Concatenate the sample dialogue text and the first prompt information and input them into the large language model.

[0154] In some embodiments, the computer device concatenates the sample dialogue text and the first prompt information and inputs them into the large language model to obtain the sample corrected dialogue text output by the large language model.

[0155] like Figure 8In the sample dialogue text and the first prompt information, the concatenated text 810-4 is: "I will provide dialogues between multiple characters, some of which may contain errors. You need to deduce and find the errors based on the character names and dialogue context information I provide, and then correct them. Please output the correct character names and dialogues at the end. The text I provide is as follows:\nCharacter A: Mom and Dad, I'm home; Character B: Come and eat; Character B: Mom just made your favorite dish; Character C: Remember to wash your hands first; Character A: I'm not hungry yet; Character A: I'm going to play soccer with my friends, you guys eat first and go to school; Character B: You've only been a few days and you've already made good friends; Character C: Come back soon."

[0156] After obtaining the concatenated text 810-4, the computer device inputs the concatenated text 810-4 into the large language model 820 to obtain the sample-corrected dialogue text output by the large language model 820.

[0157] Step 742: Fine-tune the large language model using the first real dialogue text as supervision.

[0158] In some embodiments, the computer device fine-tunes the large language model based on the difference between the sample-corrected dialogue text and the first real dialogue text.

[0159] For example, a computer device calculates a loss function based on the difference between the sample-corrected dialogue text and the first real dialogue text, and updates the parameters of the large language model based on the loss function to obtain a finely tuned large language model.

[0160] In one possible implementation, the computer device can calculate the loss function based on the sample-corrected dialogue text and the first real dialogue text 810-2 after text preprocessing, and update the parameters of the large language model through methods such as gradient descent and backpropagation; the fine-tuned large language model is obtained when the number of iterations exceeds the number threshold or the difference of the loss function is less than the difference threshold.

[0161] In this embodiment, by preprocessing and modifying the first real dialogue text, a sample dialogue text with word granularity errors can be obtained. Using the concatenated text of the sample dialogue text and the first prompt information as input, and using the first real dialogue text as supervision to fine-tune the large language model, the large language model can learn the ability to correct word granularity errors, thereby improving the quality of the dialogue text.

[0162] In some embodiments, the second prompt information corresponding to the second correction stage is used to correct errors at the statement granularity.

[0163] In some scenarios, speaker text includes speaker identifiers and text content. Errors at the statement level include incorrect speaker identifiers in speaker text (for example, the text content is actually what character A said, but the speaker identifier in speaker text is incorrectly extracted as character B).

[0164] In some scenarios, errors at the sentence level also include punctuation errors in the speaker's text. For example, a period corresponding to a declarative mood might be incorrectly extracted as a question mark corresponding to an interrogative mood, or the punctuation mark might be misplaced.

[0165] The following is an explanation of a specific scenario for correcting errors at the statement granularity level.

[0166] In some embodiments, before the second correction phase is completed, the third speaker text in the first corrected dialogue text contains an incorrect speaker identifier. After the second correction phase is completed, the incorrect speaker identifier in the second corrected dialogue text is corrected to the correct speaker identifier.

[0167] See Figure 9 , Figure 9 This is a schematic diagram illustrating an exemplary embodiment of the present application for correcting errors at the statement granularity.

[0168] like Figure 9 As shown, before the second revision stage is completed, the first revised dialogue text 910 contains speaker texts with incorrect speaker identifiers. For example, the speaker identifier "Speaker1" in speaker text 911 is an incorrect speaker identifier (the correct speaker identifier should be "Speaker2"). The speaker identifier "Speaker1" in speaker text 912 is an incorrect speaker identifier (the correct speaker identifier should be "Speaker3"). The speaker identifier "Speaker2" in speaker text 913 is an incorrect speaker identifier (the correct speaker identifier should be "Speaker1").

[0169] After the second revision stage is completed, the incorrect speaker identifiers in the second revised dialogue text 920 are corrected to the correct speaker identifiers. For example, the incorrect speaker identifier "Speaker1" in speaker text 921 is corrected to the correct speaker identifier "Speaker2", the incorrect speaker identifier "Speaker1" in speaker text 922 is corrected to the correct speaker identifier "Speaker3", and the incorrect speaker identifier "Speaker2" in speaker text 923 is corrected to the correct speaker identifier "Speaker1".

[0170] In some embodiments, during the second correction stage, the computer device concatenates the first corrected dialogue text and the second prompt information obtained in the first correction stage and inputs them into the large language model to obtain the second corrected dialogue text output by the large language model.

[0171] In one possible scenario, to enhance the correction effect, the computer device can first segment the speaker's text, and then concatenate the dialogue text and the first prompt information before inputting it into a large language model.

[0172] In one possible scenario, the speaker's text includes a speaker identifier and text content. If the number of characters in the text content of a single speaker's text exceeds a character threshold, the computer device segments the text content into at least two text sub-contents.

[0173] Optionally, the word count threshold is a value preset by the computer device, such as 10.

[0174] In one possible scenario, the speaker's text contains punctuation characters, and the computer device segments the text content based on the punctuation characters to obtain at least two text sub-contents.

[0175] For example, the speaker's text is "Speaker A: The weather is so nice today! It's perfect for a walk outside." The two sub-contents obtained by segmenting based on punctuation characters are "The weather is so nice today" and "It's perfect for a walk outside."

[0176] In some embodiments, the computer device synthesizes at least two new speaker texts based on at least two text sub-contents and a speaker identifier.

[0177] For example, based on two text sub-contents, “The weather is so nice today” and “It’s a great place to go for a walk”, and the speaker identifier “Speaker A”, two new speaker texts, “Speaker A: The weather is so nice today” and “Speaker A: It’s a great place to go for a walk”, are synthesized.

[0178] By dividing the text content into at least two sub-contents and adding speaker identifiers to each sub-content, the speaker identifiers of other speakers can be corrected in the second correction stage when text content corresponding to other speakers is mistakenly mixed in with long sentences. This avoids the situation of mixing in text content corresponding to other speakers and improves the quality of the corrected dialogue text.

[0179] In some embodiments, the second prompt message corresponding to the second correction stage is used to correct errors at the statement granularity level. Regarding the specific form of the second prompt message, given only as an example, the second prompt message could be: "I will provide dialogue between multiple characters, some of whose character names may be incorrect. You need to deduce and find the incorrect character names based on the character names and dialogue context information I provide, and then replace them with the correct character names. Please finally output the correct character names and dialogue. The text I provide is as follows:\n"

[0180] In this embodiment, during the second correction stage, the first corrected dialogue text and the second prompt information are concatenated and input into the large language model. Since the second prompt information is used to correct errors at the statement granularity level, the large language model will correct the incorrectly extracted speaker identifier to the correct speaker identifier based on the second prompt information, thereby obtaining a second corrected dialogue text with better text quality.

[0181] To improve the ability of large language models to correct errors at the granular level, one possible approach is to pre-fine-tune the large language model to improve its ability to correct granular errors at the granular level.

[0182] See Figure 10 , Figure 10 This is a flowchart illustrating an exemplary embodiment of the present application for fine-tuning a large language model to improve the ability to correct statement-level errors. In one possible implementation, the fine-tuning is performed by a computer device or other device. The process includes the following steps.

[0183] Step 1010, Data Collection.

[0184] In some embodiments, the computer device acquires a second real dialogue text that does not contain errors at the statement level.

[0185] The second real dialogue text can be dialogue text extracted from the sample audio that matches the sample audio and does not contain any textual errors. For example, the second real dialogue text can be completely correct dialogue text extracted manually.

[0186] In one possible approach, the second authentic dialogue text can be collected from script data.

[0187] Step 1010 may include steps 1011 and 1012.

[0188] Step 1011: Collect script data.

[0189] Step 1012: Extract the script dialogue.

[0190] The extracted script dialogue is the second real dialogue text, that is, the correct dialogue text that does not contain errors at the granular level of sentences.

[0191] Steps 1011 and 1012 can be similarly referred to steps 611 and 612, and will not be repeated here.

[0192] See Figure 11 , Figure 11 This is a schematic diagram of a fine-tuning of a large language model to improve the ability to correct statement-level errors, provided by an exemplary embodiment of this application.

[0193] like Figure 11 As shown, the second real dialogue text 1110-1 (script dialogue) extracted from the script data is as follows: "Character A: Mom and Dad, I'm home! Character B: Come and eat, Mom just made your favorite ribs. Character C: Remember to wash your hands first. Character A: I'm not hungry yet, I'm going to play soccer with my friends, you guys eat first. Character B: You've only been in school for a few days and you've already made good friends. Character C: Come back soon!"

[0194] Step 1020, text preprocessing.

[0195] Step 1020 may include steps 1021 and 1022.

[0196] Step 1021: Segment the speaker's text.

[0197] Step 1022: Concatenate the speaker's text.

[0198] Steps 1021 and 1022 can be similarly referred to steps 721 and 722, and will not be repeated here.

[0199] like Figure 11 In the text, the second real dialogue text 1110-2 after text preprocessing is "Character A: Mom and Dad, I'm home; Character B: Come and eat; Character B: Mom just made your favorite ribs; Character C: Remember to wash your hands first; Character A: I'm not hungry yet; Character A: I'm going to play football with my friends, you guys eat first; Character B: You've only been in school for a few days and you've already made good friends; Character C: Come back soon."

[0200] Step 1030, text modification.

[0201] In some embodiments, the computer device determines alternative speaker text from sample speaker text in a second real dialogue text.

[0202] The replaceable speaker text is the text for which the correct speaker identifier is to be replaced with the incorrect speaker identifier.

[0203] In some embodiments, step 1030 may include steps 1031 and 1032.

[0204] Step 1031: Determine the replaceable speaker text from the sample speaker text of the second proportion.

[0205] Optionally, the second percentage is a value preset by the computer device, such as 5%.

[0206] Regarding the specific method for determining replaceable speaker text, in some embodiments, if at least two sample speaker texts in the second real dialogue text contain the same speaker identifier, the computer device determines at least one sample speaker text from the at least two sample speaker texts corresponding to the same speaker identifier as replaceable speaker text.

[0207] For example, such as Figure 11 Among them, sample speaker texts 1112, 1113, and 1116 contain the same speaker identifier (all of which are role B), so the computer device identifies one of the sample speaker texts 1112 as the replaceable speaker text.

[0208] For example, such as Figure 11 In the sample speaker texts 1111, 1114, and 1115, the same speaker identifier (all role A) is contained, so the computer device identifies one of the sample speaker texts 1115 as the replaceable speaker text.

[0209] Step 1032: Modify the speaker identifier of the replaceable speaker text.

[0210] In some embodiments, the computer device modifies a first speaker identifier contained in the replaceable speaker text to a second speaker identifier to form sample dialogue text, wherein the second speaker identifier is an incorrect speaker identifier.

[0211] like Figure 11 In the process, the computer device changes the speaker identifier in the replaceable speaker text (sample speaker text 1112) from "role B" to "role A" to obtain sample speaker text 1122, and changes the speaker identifier in the replaceable speaker text (sample speaker text 1112) from "role A" to "role C" to obtain sample speaker text 1125.

[0212] Step 1040: Supervised fine-tuning.

[0213] Step 1040 includes steps 1041 and 1042.

[0214] Step 1041: Concatenate the sample dialogue text and the second prompt information and input them into the large language model.

[0215] In some embodiments, the computer device concatenates the sample dialogue text and the second prompt information and inputs them into the large language model to obtain the sample corrected dialogue text output by the large language model.

[0216] like Figure 11 In the sample dialogue text and the concatenated text of the second prompt, 1110-4, it reads: "I will provide dialogues between multiple characters, some of whose character names may be incorrect. You need to deduce and find the incorrect character names based on the character names I provide and the context information of the dialogue, and then replace them with the correct character names. Please output the correct character names and dialogues at the end. The text I provide is as follows:\nCharacter A: Mom and Dad, I'm home; Character A: Come and eat; Character B: Mom just made your favorite ribs; Character C: Remember to wash your hands first; Character A: I'm not hungry yet; Character C: I'm going to play football with my friends, you guys eat first; Character B: I've only been in school for a few days and I've already made good friends; Character C: Come back soon."

[0217] After obtaining the concatenated text 1110-4, the computer device inputs the concatenated text 1110-4 into the large language model 1120 to obtain the sample-corrected dialogue text output by the large language model 1120.

[0218] Step 1042: Fine-tune the large language model using the second real dialogue text as supervision.

[0219] In some embodiments, the computer device fine-tunes the large language model based on the difference between the sample-corrected dialogue text and the second real dialogue text.

[0220] For example, a computer device calculates a loss function based on the difference between the sample-corrected dialogue text and the second real dialogue text, and updates the parameters of the large language model based on the loss function to obtain a finely tuned large language model.

[0221] In one possible implementation, the computer device can calculate the loss function based on the sample-corrected dialogue text and the preprocessed second real dialogue text 1110-2, and update the parameters of the large language model through methods such as gradient descent and backpropagation; the fine-tuned large language model is obtained when the number of iterations exceeds the number threshold or the difference of the loss function is less than the difference threshold.

[0222] In this embodiment, by preprocessing and modifying the second real dialogue text, a sample dialogue text with sentence granularity errors can be obtained. Using the concatenated text of the sample dialogue text and the second prompt information as input, and using the second real dialogue text as supervision to fine-tune the large language model, the large language model can learn the ability to correct sentence granularity errors, thereby improving the quality of the dialogue text.

[0223] In one possible scenario, the extracted dialogue text may contain errors in word order granularity. For example, if different speakers overlap in the dialogue audio, the extracted dialogue text may contain word order errors in some speakers' texts.

[0224] For example, speaker text A corresponding to seconds 30-40 of the dialogue audio is "Speaker A: Oh, what a coincidence to run into you here!", while speaker text B corresponding to seconds 32-36 of the dialogue audio is "Speaker B: Long time no see, how have you been?". Because speaker text A and speaker text B contain overlapping speech from different speakers, the dialogue text extracted from the audio may contain errors in word order granularity, such as placing speaker text B before speaker text A.

[0225] In some embodiments, the speaker text includes a timestamp identifier, which represents the timestamp corresponding to the speaker text in the dialogue audio.

[0226] For example, the speaker text B, “Speaker B [00:32-00:36]: Long time no see, how have you been?” indicates that the audio segment corresponding to the text content “Long time no see, how have you been?” appears between the 32nd and 36th seconds of the dialogue audio.

[0227] To further improve the correction effect of the dialogue text, in some embodiments, during the third correction stage, the computer device inputs the second corrected dialogue text containing timestamps and the third prompt information into the large language model.

[0228] The third prompt information corresponding to the third correction stage is used to correct errors in word order granularity.

[0229] For example only, the third prompt could be: "I will provide dialogue between multiple characters, some of which may contain word order errors. You need to deduce and find the word order errors based on the dialogue content and context information I provide, and then correct them. Please output the dialogue with the correct word order at the end. The text I provide is as follows:\n".

[0230] In some embodiments, after the third correction phase is completed, the word order of each speaker's text in the third corrected dialogue text is determined by the large language model based on the second corrected dialogue text containing timestamps and the third cue information.

[0231] To improve the ability of large language models to correct word order granularity errors, one possible approach is to pre-fine-tune the large language model to improve its ability to correct word order granularity errors.

[0232] In some embodiments, the computer device acquires a third real dialogue text that does not contain word order granularity errors.

[0233] The third real dialogue text can be dialogue text extracted from the sample audio that matches the sample audio and does not contain any textual errors. For example, the third real dialogue text can be completely correct dialogue text extracted manually.

[0234] In some embodiments, text preprocessing such as segmenting speaker text and concatenating speaker text can also be applied to the third-party real dialogue text. For more information on text preprocessing, see [link to relevant documentation]. Figure 6 and Figure 9 The details and related descriptions will not be repeated here.

[0235] In some embodiments, a computer device alters the word order of at least one sample speaker's text within a third real dialogue text to form a sample dialogue text.

[0236] Optionally, the computer device can select a third proportion of sample speaker text from a third real dialogue text to change the word order.

[0237] Optionally, the computer device may swap at least two sample speaker texts to change the word order, or the computer device may randomly move the sample speaker texts between any two other adjacent sample speaker texts to change the word order, without any restrictions on the specific method of changing the word order.

[0238] In some embodiments, the computer device concatenates a sample dialogue text containing a timestamp and a third prompt message and inputs it into a large language model to obtain a sample-corrected dialogue text output by the large language model. Based on the difference between the sample-corrected dialogue text and the third real dialogue text, the large language model is fine-tuned.

[0239] For example, a computer device calculates a loss function based on the difference between the sample-corrected dialogue text and the third real dialogue text, and updates the parameters of the large language model based on the loss function to obtain a finely tuned large language model.

[0240] In this embodiment, by preprocessing and modifying the third real dialogue text, sample dialogue text with word order granularity errors can be obtained. Using the concatenated text of the sample dialogue text and the third prompt information as input, and using the third real dialogue text as supervision to fine-tune the large language model, the large language model can learn to correct word order granularity errors, thereby improving the quality of the dialogue text.

[0241] The dialogue text processing method proposed in this application has a wide range of applications, including but not limited to the following.

[0242] (1) Video production and video editing scenarios.

[0243] When a user captures video content using a terminal, the dialogue text processing method proposed in this application can be used to extract high-quality corrected dialogue text from the video content. This corrected dialogue text corrects errors at different levels. Therefore, using the corrected dialogue text to create subtitles for video content, or editing video content based on the corrected dialogue text, results in high-quality video production.

[0244] (2) Intelligent customer service and voice assistant scenarios.

[0245] After acquiring the audio of the communication between the user and the intelligent customer service or voice assistant, the dialogue text processing method proposed in this application can be used to extract high-quality corrected dialogue text from the audio. The corrected dialogue text corrects errors at different levels. Therefore, the intelligent customer service or voice assistant can provide accurate services to users based on the correct dialogue text, or use it for subsequent analysis and service optimization, thereby improving customer experience and service quality.

[0246] (3) Social media and content recommendation scenarios.

[0247] When users publish media content (such as video, audio, or live stream content) through social media, the corresponding backend server of the social media can use the dialogue text processing method proposed in this application to extract accurate corrected dialogue text from the media content, so as to perform content analysis and recommendation based on the corrected dialogue text. Since the corrected dialogue text has corrected errors at different levels, the corresponding backend server of the social media can better analyze the viewpoints and sentiments of the media content through the corrected dialogue text, thereby providing users with personalized content recommendations.

[0248] It should be noted that the above application scenarios are merely examples. The dialogue text processing method proposed in this application can also be applied to any other possible scenarios, such as office meeting scenarios, game voice scenarios, or virtual reality scenarios, without any limitation.

[0249] Video editing scenarios

[0250] In one optional example, the dialogue text processing method provided in the embodiments of this application is used as an example in a video editing scenario.

[0251] In some embodiments, the computer device acquires dialogue text, wherein the dialogue text is extracted from a dialogue video, the dialogue video being a video containing at least one speaker speaking, and the dialogue text including speaker text corresponding to each speaker.

[0252] As an example only, the speaker can be a video character shown in the dialogue video, or the speaker can be a voice-over or narrator in the dialogue video.

[0253] Regarding the acquisition of dialogue text, in one possible implementation, the computer device can extract the dialogue text from the video using technologies such as VAD, OSD, and SCD. In another possible implementation, the computer device can use an audio-to-text tool to first acquire all the text in the video, then segment the text into speaker texts based on the text content and audio features, thus forming the dialogue text. In yet another possible implementation, the dialogue text can be extracted from the video by another device, allowing the computer device to directly acquire the dialogue text from that device.

[0254] In some embodiments, the computer device inputs the dialogue text and at least two types of prompts into a large language model to obtain the corrected dialogue text output by the large language model.

[0255] Among them, at least two different prompts are used to prompt correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on at least two prompts and sample dialogue text, which contains errors at different levels.

[0256] In some embodiments, errors at different levels are errors at different text granularities in the dialogue text. The text granularity includes at least two of the following: word granularity, sentence granularity, and word order granularity. The text granularity corresponding to the (i+1)th level is higher than that corresponding to the i-th level.

[0257] In some embodiments, the computer device corrects word-level errors in the dialogue text through a first correction stage to obtain a first corrected dialogue text; and corrects sentence-level errors in the first corrected dialogue text through a second correction stage to obtain a second corrected dialogue text. The computer device then adjusts the format of the second corrected dialogue text to obtain the final corrected dialogue text.

[0258] See Figure 12 , Figure 12It is a schematic diagram for correcting a dialogue text through a first correction stage and a second correction stage provided by an exemplary embodiment of the present application.

[0259] As Figure 12 shown, there are errors at different text granularities in the speaker texts 1212 to 1215 in the dialogue text 1210-1 extracted from the dialogue video.

[0260] Among them, there are word granularity errors in the speaker text 1211 and the speaker text 1212. The first word "we two" at the beginning of the sentence in the speaker text 1212 is wrongly extracted to the end of the sentence in the speaker text 1211. There are also word granularity errors in the speaker text 1213 and the speaker text 1214. The last word "met" at the end of the sentence in the speaker text 1213 is wrongly extracted to the beginning of the sentence in the speaker text 1214.

[0261] Among them, there are sentence granularity errors in the speaker text 1212 and the speaker text 1215. The speaker identifier "Speaker1" in the speaker text 1212 is wrongly extracted as "Speaker2", and the speaker identifier "Speaker3" in the speaker text 1215 is wrongly extracted as "Speaker2".

[0262] In some embodiments, in the first correction stage, the computer device inputs the concatenated text 1210-2 of the dialogue text and the first prompt information into the large language model 1220, and obtains the first corrected dialogue text 1230-1 output by the large language model 1220.

[0263] Among them, the first prompt information is used to prompt the correction of the word granularity errors in the dialogue text 1210-2, and the word granularity errors in the obtained first corrected dialogue text 1230-1 have been corrected.

[0264] In some embodiments, in the second correction stage, the computer device inputs the concatenated text 1230-2 of the first corrected dialogue text and the second prompt information into the large language model 1240, and obtains the second corrected dialogue text 1250-1 output by the large language model 1240.

[0265] Among them, the second prompt information is used to prompt the correction of the sentence granularity errors in the first corrected dialogue text 1230-1, and the sentence granularity errors in the obtained second corrected dialogue text 1250-1 have been corrected.

[0266] In some embodiments, the computer device adjusts the format of the second corrected dialogue text 1250-1 (e.g., restores punctuation, splices consecutive speaker texts from the same speaker, etc.) to obtain the final corrected dialogue text 1250-2. The speaker texts in the corrected dialogue text 1250-2 (including speaker texts 1221, 1222, 1223, 1224, and 1225) do not contain errors at any level.

[0267] In some embodiments, the modified dialogue text may also include a timestamp identifier to represent the timestamp corresponding to the speaker's text in the dialogue video.

[0268] In some embodiments, a computer device may extract key text from the corrected dialogue text and, based on the timestamps corresponding to the key text, edit key video from the dialogue video.

[0269] For example, a 10-minute original dialogue video has a corresponding 1500-character revised dialogue text. The computer device extracts 150 characters of key text from the revised dialogue text and edits a 1-minute key video from the dialogue video.

[0270] See Figure 13 , Figure 13 This is a structural block diagram of a dialog text processing apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0271] The acquisition module 1301 is used to acquire dialogue text, which is extracted from dialogue audio and includes speaker text corresponding to each speaker.

[0272] The correction module 1302 is used to input the dialogue text and at least two kinds of prompt information into the large language model to obtain the corrected dialogue text output by the large language model.

[0273] The different prompts in the at least two types of prompts are used to prompt correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on the at least two types of prompts and the sample dialogue text, which contains errors at different levels.

[0274] Optionally, the correction module 1302 is used for:

[0275] In the first correction stage, the dialogue text and the first prompt information are concatenated and input into the large language model to obtain the first corrected dialogue text output by the large language model. The first prompt information is used to prompt the correction of the first level of error in the dialogue text.

[0276] In the (i+1)th correction stage, the i-th corrected dialogue text and the (i+1)th prompt information are concatenated and input into the large language model to obtain the (i+1)th corrected dialogue text output by the large language model, where i is a positive integer. The (i+1)th prompt information is used to prompt the correction of errors at the (i+1)th level in the dialogue text, wherein the (i+1)th level is higher than the i-th level.

[0277] Optionally, the errors at different levels are errors at different text granularities in the dialogue text. The text granularity includes at least two of the following: word granularity, sentence granularity, and word order granularity. The text granularity corresponding to the (i+1)th level is higher than the text granularity corresponding to the i-th level.

[0278] Optionally, the dialogue text includes a first speaker text and a second speaker text, wherein the second speaker text is the speaker text following the first speaker text;

[0279] After the first correction stage is completed, the first word of the second speaker's text in the first corrected dialogue text is the last word of the first speaker's text in the dialogue text before the first correction stage is completed; or,

[0280] After the first correction phase is completed, the sentence-ending words of the first speaker's text in the first corrected dialogue text are the sentence-initial words of the second speaker's text in the dialogue text before the first correction phase is completed.

[0281] Optionally, the device further includes a fine-tuning module for:

[0282] Obtain the first real dialogue text, which does not contain errors at the specified word granularity;

[0283] The sample dialogue text is formed by moving the last word of the first sample speaker text in the first real dialogue text to the beginning of the second sample speaker text, or by moving the first word of the second sample speaker text to the end of the first sample speaker text. The second sample speaker text is the speaker text that follows the first sample speaker text.

[0284] The sample dialogue text and the first prompt information are concatenated and input into the large language model to obtain the sample corrected dialogue text output by the large language model.

[0285] Based on the difference between the corrected dialogue text and the first real dialogue text, the large language model is fine-tuned.

[0286] Optionally, the speaker text includes a speaker identifier and text content;

[0287] Before the second correction phase is completed, the third speaker text in the first corrected dialogue text contains incorrect speaker identification;

[0288] After the second correction phase is completed, the incorrect speaker identifiers in the second corrected dialogue text are corrected to the correct speaker identifiers.

[0289] Optional, fine-tuning module, used for:

[0290] Obtain the second real dialogue text, which does not contain errors at the stated granularity;

[0291] From the sample speaker text in the second real dialogue text, determine the replaceable speaker text;

[0292] The first speaker identifier contained in the replaceable speaker text is modified to a second speaker identifier to form the sample dialogue text, wherein the second speaker identifier is an incorrect speaker identifier;

[0293] The sample dialogue text and the second prompt information are concatenated and input into the large language model to obtain the sample corrected dialogue text output by the large language model.

[0294] Based on the difference between the corrected dialogue text and the second real dialogue text, the large language model is fine-tuned.

[0295] Optional, fine-tuning module, used for:

[0296] If at least two of the sample speaker texts in the second real dialogue text contain the same speaker identifier, at least one of the sample speaker texts corresponding to the same speaker identifier is identified as the replaceable speaker text.

[0297] Optionally, the speaker text includes a speaker identifier and text content. The correction module 1302 is used for:

[0298] If the number of characters in the text content of a single speaker's text is greater than a character threshold, at least two text sub-contents are obtained by segmenting the text content; or, if the text content in the speaker's text contains punctuation characters, at least two text sub-contents are obtained by segmenting the text content based on the punctuation characters.

[0299] Based on at least two of the text sub-contents and the speaker identifier, at least two new speaker texts are synthesized.

[0300] Optionally, the speaker text includes a timestamp identifier, which represents the timestamp corresponding to the speaker text in the dialogue audio;

[0301] After the third correction phase is completed, the word order of each speaker's text in the third corrected dialogue text is determined by the large language model based on the second corrected dialogue text containing the timestamp identifier and the third prompt information.

[0302] Optional, fine-tuning module, used for:

[0303] Obtain a third real dialogue text, wherein the third real dialogue text does not contain errors at the grammatical order level;

[0304] The word order of at least one sample speaker text in the third real dialogue text is changed to form the sample dialogue text; the sample dialogue text containing the timestamp identifier and the third prompt information are concatenated and input into the large language model to obtain the sample corrected dialogue text output by the large language model;

[0305] Based on the differences between the corrected dialogue text and the third real dialogue text, the large language model is fine-tuned.

[0306] See Figure 14 , Figure 14 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0307] Optionally, the computer device can be a terminal or a server.

[0308] Specifically, the computer device 1400 includes a Central Processing Unit (CPU) 1401, a system memory 1404 including random access memory 1402 and read-only memory 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 also includes a basic input / output system (I / O system) 1406 to facilitate information transfer between various components within the computer, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0309] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0310] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer-readable media (not shown) such as a hard disk or drive.

[0311] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0312] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1401. The one or more programs contain computer instructions for implementing the methods described above. The central processing unit 1401 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0313] According to various embodiments of this application, the computer device 1400 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1400 can be connected to a network 1412 via a network interface unit 1411 connected to the system bus 1405, or the network interface unit 1411 can be used to connect to other types of networks or remote computer systems (not shown).

[0314] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0315] This application also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the method described in the above embodiments. Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0316] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above aspects.

[0317] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing dialogue text, characterized in that, The method includes: The dialogue text is obtained from the dialogue audio and includes the speaker text corresponding to each speaker. The dialogue text and at least two types of prompts are input into the large language model to obtain the corrected dialogue text output by the large language model. The different prompts in the at least two types of prompts are used to prompt correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on the at least two types of prompts and the sample dialogue text, which contains errors at different levels.

2. The method according to claim 1, characterized in that, The step of inputting the dialogue text and at least two types of prompting information into a large language model to obtain the corrected dialogue text output by the large language model includes: In the first correction stage, the dialogue text and the first prompt information are concatenated and input into the large language model to obtain the first corrected dialogue text output by the large language model. The first prompt information is used to prompt the correction of the first level of error in the dialogue text. In the (i+1)th correction stage, the i-th corrected dialogue text and the (i+1)th prompt information are concatenated and input into the large language model to obtain the (i+1)th corrected dialogue text output by the large language model, where i is a positive integer. The (i+1)th prompt information is used to prompt the correction of errors at the (i+1)th level in the dialogue text, wherein the (i+1)th level is higher than the i-th level.

3. The method according to claim 2, characterized in that, The different levels of errors refer to errors at different text granularities in the dialogue text. The text granularity includes at least two of the following: word granularity, sentence granularity, and word order granularity. The text granularity corresponding to the (i+1)th level is higher than the text granularity corresponding to the i-th level.

4. The method according to claim 3, characterized in that, The dialogue text includes a first speaker text and a second speaker text, wherein the second speaker text is the speaker text following the first speaker text; After the first correction stage is completed, the first word of the second speaker's text in the first corrected dialogue text is the last word of the first speaker's text in the dialogue text before the first correction stage is completed; or, After the first correction phase is completed, the sentence-ending words of the first speaker's text in the first corrected dialogue text are the sentence-initial words of the second speaker's text in the dialogue text before the first correction phase is completed.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the first real dialogue text, which does not contain errors at the specified word granularity; The sample dialogue text is formed by moving the last word of the first sample speaker text in the first real dialogue text to the beginning of the second sample speaker text, or by moving the first word of the second sample speaker text to the end of the first sample speaker text. The second sample speaker text is the speaker text that follows the first sample speaker text. The sample dialogue text and the first prompt information are concatenated and input into the large language model to obtain the sample corrected dialogue text output by the large language model. Based on the difference between the corrected dialogue text and the first real dialogue text, the large language model is fine-tuned.

6. The method according to any one of claims 3 to 5, characterized in that, The speaker text includes a speaker identifier and text content; Before the second correction phase is completed, the third speaker text in the first corrected dialogue text contains incorrect speaker identification; After the second correction phase is completed, the incorrect speaker identifiers in the second corrected dialogue text are corrected to the correct speaker identifiers.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the second real dialogue text, which does not contain errors at the stated granularity; From the sample speaker text in the second real dialogue text, determine the replaceable speaker text; The first speaker identifier contained in the replaceable speaker text is modified to a second speaker identifier to form the sample dialogue text, wherein the second speaker identifier is an incorrect speaker identifier; The sample dialogue text and the second prompt information are concatenated and input into the large language model to obtain the sample corrected dialogue text output by the large language model. Based on the difference between the corrected dialogue text and the second real dialogue text, the large language model is fine-tuned.

8. The method according to claim 7, characterized in that, The step of determining the replaceable speaker text from the sample speaker text in the second real dialogue text includes: If at least two of the sample speaker texts in the second real dialogue text contain the same speaker identifier, at least one of the sample speaker texts corresponding to the same speaker identifier is identified as the replaceable speaker text.

9. The method according to any one of claims 1 to 8, characterized in that, The speaker text includes a speaker identifier and text content, and the method further includes: If the number of characters in the text content of a single speaker's text is greater than a character threshold, at least two text sub-contents are obtained by segmenting the text content; or, if the text content in the speaker's text contains punctuation characters, at least two text sub-contents are obtained by segmenting the text content based on the punctuation characters. Based on at least two of the text sub-contents and the speaker identifier, at least two new speaker texts are synthesized.

10. The method according to any one of claims 3 to 9, characterized in that, The speaker's text includes a timestamp identifier, which represents the timestamp corresponding to the speaker's text in the dialogue audio. After the third correction phase is completed, the word order of each speaker's text in the third corrected dialogue text is determined by the large language model based on the second corrected dialogue text containing the timestamp identifier and the third prompt information.

11. The method according to claim 10, characterized in that, The method further includes: Obtain a third real dialogue text, wherein the third real dialogue text does not contain errors at the grammatical order level; The word order of at least one sample speaker text in the third real dialogue text is changed to form the sample dialogue text; the sample dialogue text containing the timestamp identifier and the third prompt information are concatenated and input into the large language model to obtain the sample corrected dialogue text output by the large language model; Based on the differences between the corrected dialogue text and the third real dialogue text, the large language model is fine-tuned.

12. A dialog text processing device, characterized in that, The device includes: The acquisition module is used to acquire dialogue text, which is extracted from dialogue audio and includes speaker text corresponding to each speaker. The correction module is used to input the dialogue text and at least two kinds of prompt information into the large language model to obtain the corrected dialogue text output by the large language model. The different prompts in the at least two types of prompts are used to prompt correction of errors at different levels in the dialogue text, and the large language model is fine-tuned based on the at least two types of prompts and the sample dialogue text, which contains errors at different levels.

13. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one computer instruction, which is executed by the processor to implement the dialog text processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the dialog text processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the dialog text processing method as described in any one of claims 1 to 11.

Images