Speech recognition method, device, equipment and medium based on large language model

Abstract

The present invention discloses a speech recognition method, device, equipment and medium based on a large language model. The method is applied to a management server, including inputting a received real-time input audio stream into a preset streaming recognition model to obtain a corresponding first-stage speech recognition text; dividing the audio stream into a plurality of sub-audio streams in chronological order according to the determined punctuation points, and inputting all the sub-audio streams into a preset first speech recognition model in sequence to obtain a corresponding second-stage sub-speech recognition text; inputting the second-stage sub-speech recognition texts related to different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts, so as to obtain a target speech recognition text corresponding to all the received audio streams. The method can improve the accuracy of speech recognition, correct and optimize the punctuation positions through the large language model, and improve the restoration of speech content.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method, device, equipment and medium based on a large language model. Background Art

[0002] In simultaneous interpretation scenarios, the speaker's speaking rhythm is not expressed in standard natural language, which will cause a lot of the speaker's audio to be acquired without pause for a long time. Therefore, it needs to be interrupted to ensure subsequent output. The current direct interruption method can easily make the acquired audio incomplete, making it impossible to accurately identify the interrupted audio position, affecting the speech recognition effect, and causing problems such as voice delay in subsequent translation results. Summary of the invention

[0003] The embodiments of the present invention provide a speech recognition method, apparatus, device and medium based on a large language model, aiming to solve the problem of inaccurate speech recognition in the prior art methods.

[0004] In a first aspect, an embodiment of the present invention provides a speech recognition method based on a large language model, wherein the method is applied to a management server, and the method includes:

[0005] Input the received real-time input audio stream into a preset streaming recognition model to obtain the corresponding first-stage speech recognition text until the preset audio stream end condition is triggered;

[0006] Determining whether the received audio stream needs to be segmented according to a preset segmentation condition and the first-stage speech recognition text, wherein the segmentation condition at least includes a forced segmentation condition;

[0007] If yes, then determine the corresponding punctuation point, and divide the audio stream into a plurality of sub-audio streams in chronological order according to the determined punctuation points, and input all the sub-audio streams into a preset first speech recognition model in turn to obtain the corresponding second-stage sub-speech recognition text; wherein the punctuation point corresponding to the mandatory punctuation condition is the first punctuation point;

[0008] Inputting the second-stage sub-speech recognition texts associated with different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts;

[0009] All the corrected two-stage sub-speech recognition texts and the remaining uncorrected two-stage sub-speech recognition texts are combined in time order to obtain target speech recognition texts corresponding to all received audio streams.

[0010] In a second aspect, an embodiment of the present invention further provides a speech recognition device based on a large language model, wherein the device is applied to a management server, and the device includes:

[0011] A streaming recognition unit, used to input the received real-time input audio stream into a preset streaming recognition model to obtain a corresponding first-stage speech recognition text until a preset audio stream end condition is triggered;

[0012] A sentence segmentation judgment unit, used for judging whether the received audio stream needs to be segmented according to a preset sentence segmentation condition and the first-stage speech recognition text, wherein the sentence segmentation condition at least includes a forced sentence segmentation condition;

[0013] A speech recognition unit, for determining a corresponding punctuation point if yes, and dividing the audio stream into a plurality of sub-audio streams in chronological order according to the determined punctuation points, and inputting all the sub-audio streams into a preset first speech recognition model in turn to obtain a corresponding second-stage sub-speech recognition text; wherein the punctuation point corresponding to the mandatory punctuation condition is the first punctuation point;

[0014] A text correction unit, used for inputting the second-stage sub-speech recognition texts associated with different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts;

[0015] The target determination unit is used to combine all the corrected two-stage sub-speech recognition texts and the remaining uncorrected two-stage sub-speech recognition texts in time sequence to obtain a target speech recognition text corresponding to all the received audio streams.

[0016] In a third aspect, an embodiment of the present invention further provides a computer device, wherein the device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0017] Memory, used to store computer programs;

[0018] The processor is used to implement the steps of the speech recognition method based on the large language model described in the first aspect when executing the program stored in the memory.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the speech recognition method based on a large language model as described in the first aspect above are implemented.

[0020] The embodiment of the present invention provides a speech recognition method, device, equipment and medium based on a large language model, the method comprising: inputting a received real-time input audio stream into a preset streaming recognition model to obtain a corresponding first-stage speech recognition text until a preset audio stream end condition is triggered; judging whether the received audio stream needs to be segmented according to a preset segmentation condition and the first-stage speech recognition text, wherein the segmentation condition at least includes a mandatory segmentation condition; if so, determining a corresponding segmentation point, and dividing the audio stream into a plurality of sub-audio streams in chronological order according to the determined segmentation point, and inputting all the sub-audio streams in sequence into a preset first speech recognition model to obtain a corresponding second-stage sub-speech recognition text; wherein the segmentation point corresponding to the mandatory segmentation condition is a first segmentation point; inputting the second-stage sub-speech recognition texts related to different first segmentation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts; combining all the corrected second-stage sub-speech recognition texts and the remaining uncorrected second-stage sub-speech recognition texts in chronological order to obtain a target speech recognition text corresponding to all the received audio streams. The above-mentioned speech recognition method based on a large language model is applied to a management server. This method can improve the accuracy of speech recognition, that is, by correcting and optimizing the sentence segmentation position through a large language model, thereby improving the restoration of the speech content and making the sentences in the recognized speech recognition text more fluent, so as to improve the accuracy and fluency of translation during simultaneous interpretation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0022] Figure 1 A method flow chart of a speech recognition method based on a large language model provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of an application scenario of a speech recognition method based on a large language model provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of another application scenario of the speech recognition method based on a large language model provided by an embodiment of the present invention;

[0025] Figure 4 A schematic block diagram of a speech recognition device based on a large language model provided by an embodiment of the present invention;

[0026] Figure 5A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0029] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] See also Figure 1 and Figure 2As shown in the figure, the embodiment of the present invention provides a speech recognition method based on a large language model, which is applied to a management server 10. If the management server 10 inputs the received real-time input audio stream into a preset streaming recognition model 20 to obtain a corresponding first-stage speech recognition text until the preset audio stream end condition is triggered; the management server 10 also determines whether the received audio stream needs to be segmented according to the preset segmentation condition and the first-stage speech recognition text; if so, the corresponding segmentation point is determined, and the audio stream is divided into a plurality of sub-audio streams in chronological order according to the determined segmentation point, and all the sub-audio streams are sequentially input into a preset first speech recognition model 30 to obtain a corresponding second-stage sub-speech recognition text; wherein the segmentation point corresponding to the forced segmentation condition is the first segmentation point; The management server 10 also inputs the second-stage sub-speech recognition texts related to different first punctuation points into the preset large language model 40 in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts; finally, the management server 10 also combines all the corrected second-stage sub-speech recognition texts and the remaining uncorrected second-stage sub-speech recognition texts in chronological order to obtain the target speech recognition text corresponding to all the received audio streams. Therefore, the above method not only improves the accuracy of speech recognition through the setting of the streaming recognition model, the first speech recognition model and the large language model, that is, the sentence punctuation position is corrected and optimized through the large language model, the restoration degree of the speech content is improved, and the sentences in the recognized speech recognition text are more fluent, and the accuracy and fluency of the translation can be improved in the simultaneous interpretation process; in addition, the speech recognition method based on the large language model is used for speech recognition during simultaneous interpretation, which can effectively improve the translation quality and reduce the voice delay after translation.

[0032] like Figure 1 As shown, in one embodiment, the speech recognition method based on the large language model is applied to a management server, which can be a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The method includes steps S110~S150.

[0033] Step S110, inputting the received real-time input audio stream into a preset streaming recognition model to obtain a corresponding first-stage speech recognition text until a preset audio stream end condition is triggered.

[0034] Among them, the audio stream refers to a real-time transmitted audio data stream. The streaming recognition model refers to a type of ASR (Automatic Speech Recognition) model that can support real-time return of recognition results in the process of processing the audio stream. In contrast, the non-streaming model must return the result after processing the complete sentence audio. Streaming ASR can be better used in scenarios where real-time acquisition of recognition results is required, such as live real-time subtitles, real-time recording of meetings, voice input, voice wake-up and other scenarios. In some embodiments, the streaming recognition model can be Zipformer, which is a new type of ASR model. Compared with mainstream ASR models such as Conformer, Squeezeformer, and E-Branchformer, Zipformer has the advantages of better effect, faster calculation, and more memory saving.

[0035] In this embodiment, the management server can input the received real-time input audio stream into a preset streaming recognition model, and output the corresponding first-stage speech recognition text through the streaming recognition model. The first-stage speech recognition text can be quickly displayed to the user for viewing, and is also convenient for subsequent process operations.

[0036] Step S120, judging whether the received audio stream needs to be segmented according to preset segmentation conditions and the first-stage speech recognition text, wherein the segmentation conditions at least include mandatory segmentation conditions.

[0037] In the present embodiment, the management server can also determine whether the received audio stream needs to be segmented according to the preset segmentation conditions and the obtained one-stage speech recognition text, that is, when the audio stream input is completed, the entire audio stream obtained is segmented, and the basis for the segmentation is the pre-set segmentation conditions. As an option, the segmentation conditions at least include a forced segmentation condition, which is of course not limited in the present embodiment. It can be generally believed that there is a certain error in the meaning between the adjacent voice texts obtained after the audio stream after segmentation by the forced segmentation condition is re-identified, which may lead to inaccurate recognition of the overall meaning, so it is necessary to continue with subsequent operations.

[0038] In one embodiment, the preset segmentation condition includes a forced segmentation condition, the segmentation point corresponding to the forced segmentation condition is the first segmentation point, the forced segmentation condition includes that the pause time between words does not exceed a preset time threshold and the cumulative number of recognized characters in the current stage reaches a preset number of characters, and the step S120 includes the following sub-steps:

[0039] Determine whether the pause duration between words in the received audio stream does not exceed a preset time threshold; if so, detect whether the cumulative number of characters in the first-stage speech recognition text reaches a preset number of characters according to the time sequence; if so, determine the time point corresponding to the last character in the accumulated characters as the first punctuation point of the corresponding audio stream for sentence segmentation, and return to the step of detecting whether the number of characters after the first punctuation point in the first-stage speech recognition text reaches a preset number of characters according to the time sequence, until all judgments on the received audio stream are completed.

[0040] The management server can determine whether the pause duration between words of the received audio stream does not exceed a preset time threshold according to the forced segmentation condition. Generally, the preset time threshold can be determined according to the actual usage of the user, and is not specifically limited in this embodiment.

[0041] If the pause duration between words in the received audio stream does not exceed the preset time threshold, it indicates that the speaker is still speaking a complete sentence. In order to avoid errors in speech recognition and improve recognition accuracy, it is necessary to detect whether the cumulative number of characters in the speech recognition text of the first stage has reached the preset number of characters according to the time sequence, that is, to distinguish whether the cumulative number of characters currently continuously input and recognized has reached the preset number of characters. If so, it indicates that the audio stream needs to be forcibly segmented at this time, and the time point corresponding to the last character in the accumulated characters is determined as the first segmentation point of the corresponding audio stream for segmentation.

[0042] Then, after completing this segmentation, it is necessary to continue to segment the audio stream after the first segmentation point. If the pause duration between words in the received audio stream does not exceed the preset time threshold, it is necessary to perform the step of detecting whether the number of characters after the first segmentation point in the first-stage speech recognition text reaches the preset number of characters according to the time sequence until all judgments on the received audio stream are completed; that is, the power-off judgment on the entire audio stream is completed so as to perform subsequent operations.

[0043] In a further embodiment, the sentence segmentation condition further includes a sentence end segmentation condition, the sentence end segmentation condition includes that the pause duration between words exceeds a preset time threshold, and the segmentation point corresponding to the sentence end segmentation condition is a second segmentation point. After the step of determining whether the pause duration between words of the received audio stream does not exceed the preset time threshold, the step further includes:

[0044] If not, the last speech time point before the pause of the received audio stream is taken as the second punctuation point, and the step of determining whether the pause duration between words of the received audio stream after the second punctuation point does not exceed the preset time threshold is returned.

[0045] Among them, if the pause duration between words of the received audio stream exceeds the preset time threshold, the last speech time point before the pause of the received audio stream is used as the second break point. After completing this break point, it is necessary to continue to return to the step of determining whether the pause duration between words of the received audio stream after the second break point does not exceed the preset time threshold. If it exceeds, continue to proceed to the sentence end point; if it does not exceed, continue to determine whether to perform forced break until all the audio stream break operations are completed.

[0046] For example, sentence segmentation can mainly rely on the following conditions, namely, the pause duration between words and the cumulative number of recognized characters in the current stage. When the pause duration between words in the audio stream exceeds 500ms, it is understood as a complete sentence and the end-of-sentence segmentation logic is executed; if the pause between words does not exceed 500ms and the cumulative number of recognized characters in the current stage exceeds 60 words, sentence segmentation is also executed, which is forced segmentation. The main difference between the two is that sentence segmentation based on the pause duration usually results in a complete sentence, and the recognition result sent to the first speech recognition model is relatively good, while forced segmentation usually does not result in a complete sentence, and it is possible that the sentence is cut in the middle of the pronunciation of a word or a single word. At this time, the middle and back parts of the recognition result sent to the first speech recognition model are relatively poor. By the same token, the beginning of the next audio segment after the segmentation is also a similar situation, so the relevant recognition results need to be corrected at this time.

[0047] Step S130, if yes, determine the corresponding punctuation points, and divide the audio stream into several sub-audio streams in chronological order according to the determined punctuation points, and input all the sub-audio streams into the preset first speech recognition model in turn to obtain the corresponding second-stage sub-speech recognition text; wherein the punctuation point corresponding to the forced punctuation condition is the first punctuation point.

[0048] In this embodiment, if it is determined that the received audio stream needs to be segmented based on the preset segmentation conditions and the first-stage speech recognition text, then the corresponding segmentation points need to be determined at this time, and the audio stream is divided into several sub-audio streams in chronological order according to the determined segmentation points, and all the sub-audio streams are input into the preset first speech recognition model in sequence to obtain the corresponding second-stage sub-speech recognition text.

[0049] For example, the model used by the first speech recognition model can be Paraformer, which is characterized by inputting a whole audio segment and recognizing the text result. The recognition accuracy is greatly improved compared with the streaming recognition model because it takes the content of the whole sentence into consideration and can be combined with semantic recognition, homophone recognition, near-phonetic character recognition, etc., so the recognition accuracy is much higher. However, due to the existence of forced sentence punctuation, the recognition result of the audio stream part after forced sentence punctuation needs to be subsequently corrected before it can be sent to the translation process with the recognition result of the sentence end punctuation part to realize simultaneous interpretation, which effectively reduces the delay time of simultaneous interpretation and improves the corresponding accuracy.

[0050] Step S140, the second-stage sub-speech recognition texts associated with different first punctuation points are input into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts.

[0051] In this embodiment, the two-stage sub-voice recognition text related to the first punctuation point is determined. Usually, the two-stage sub-voice recognition text related to the first punctuation point can be the two-stage sub-voice recognition text corresponding to the front sub-frequency stream located in the segment before the first punctuation point, or it can be the two-stage sub-voice recognition text corresponding to the rear sub-frequency stream located in the segment after the first punctuation point.

[0052] The two-stage sub-speech recognition texts related to different first punctuation points are input into the preset large language model in batches, so that a plurality of corresponding corrected two-stage sub-speech recognition texts can be obtained, that is, for each first punctuation point that appears, the corresponding correction of the two-stage sub-speech recognition text is completed.

[0053] For example, the preset Large Language Model (LLM) is an adjusted large language model. Specifically, in order to ensure the stability and accuracy of the recognition task, LoRA (Low-Rank Adaptation) is used to perform supervised fine-tuning (SFT) on the conventional large language model. Compared with full model fine-tuning, this method is more efficient, uses less training data, is less expensive, and can adapt to new tasks faster. In addition, we also use PromptEngineering to increase model stability, and adjust Prompt through multiple experiments to make the model's output more controllable.

[0054] Among them, SFT, namely Supervised Fine-Tuning, is a supervised fine-tuning technology. In large model fine-tuning technology, it refers to further training based on the pre-trained language model using a specific task dataset to improve the performance of the model on specific tasks.

[0055] LoRA is a method for updating weights during neural network fine-tuning training. It is called Low-Rank Adaptation and is a model fine-tuning technology, especially in the fine-tuning of large language models (such as BERT). The core idea of ​​LoRA is to introduce a low-rank structure into the model's weight matrix to achieve efficient adjustment of model parameters.

[0056] Specifically, LoRA implements model fine-tuning through the following steps:

[0057] Low-rank decomposition: The weight matrix in the model is decomposed into the product of two smaller matrices. These two smaller matrices have a lower rank, which means that they contain less information but are enough to capture the information that the model needs to be tuned for a specific task.

[0058] Parameter sharing: Sharing these low-rank matrices between different layers or parts of the model can further reduce the number of model parameters and improve training efficiency;

[0059] Fine-tuning process: During training, only the parameters of these low-rank matrices are updated, rather than the weight matrix of the entire model. In this way, most of the parameters of the model remain unchanged, and only the task-related parts are adjusted;

[0060] Maintaining model stability: Since only a small fraction of the model parameters are adjusted, LoRA helps preserve what the model has learned during the pre-training phase while enabling the model to adapt to new tasks.

[0061] From this we can see that the advantage of LoRA technology is that it can effectively fine-tune large models to perform better on specific tasks without significantly increasing the model size and computational cost.

[0062] In addition, Prompt Engineering is Prompt Engineering. Usually, when a request is sent to a big model, the content of the request is called Prompt. The key is how to describe the task to the big model so that the return of the big model is more stable. Therefore, the Prompt can be adjusted according to the required task so that the output of the big model can achieve the purpose of following the original meaning of the sentence and only modifying the part to make the sentence complete and smooth.

[0063] The training data set used when pre-training the language model to obtain the first speech recognition model can be obtained in the following way:

[0064] By writing the actual recognition results into a dataset and playing the audio with subtitles for recognition, we can obtain the inaccurate content obtained by forced sentence segmentation and the accurate content obtained through subtitles, so that the large model can learn how to splice and artificially introduce more audio interruptions to generate more datasets.

[0065] By crawling various long text contents on the Internet, randomly segmenting the text, and introducing noises such as homophones and modal particles, the recognition results after audio interruption are simulated.

[0066] Therefore, by introducing LLM and designing supervised fine-tuning and prompt adjustment, the semantic accuracy can be improved and the translation quality can be improved accordingly.

[0067] In one embodiment, the step S104 further includes the following sub-steps:

[0068] Determine the two-stage sub-speech recognition text corresponding to the front sub-audio stream and the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point; preprocess the two-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding first intermediate text; input the first intermediate text and the two-stage sub-speech recognition text corresponding to the back sub-audio stream into a preset large language model to output the revised two-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point.

[0069] The management server can determine the two-stage sub-speech recognition text corresponding to the front sub-audio stream and the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point, and pre-process the two-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to a pre-set pre-processing rule to obtain the corresponding first intermediate text. After the first intermediate text and the two-stage sub-speech recognition text corresponding to the back sub-audio stream are input into a preset large language model, the corrected two-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point can be obtained.

[0070] In a specific embodiment, the first intermediate text includes a first main content and an end content, and the step of preprocessing the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding first intermediate text includes the following sub-steps:

[0071] The two-stage sub-speech recognition texts corresponding to the front sub-audio stream of each first punctuation point are respectively input into the semantic filter to obtain the first main content and the end content corresponding to the front sub-audio stream of each first punctuation point; wherein the end content corresponding to the front sub-audio stream of each first punctuation point and the two-stage sub-speech recognition text corresponding to the back sub-audio stream are input into the preset large language model to output the revised end content corresponding to the front sub-audio stream of each first punctuation point; the first main content and the revised end content corresponding to the front sub-audio stream of each first punctuation point are combined to determine as the revised two-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point.

[0072] Among them, since there are several first punctuation points, the management server needs to input the second-stage sub-voice recognition text corresponding to the front sub-audio stream of each first punctuation point into the semantic filter respectively, so as to filter by the position and semantics of the content, and first preliminarily obtain the accurate first main content and inaccurate terminal content of the second-stage sub-voice recognition text corresponding to the front sub-audio stream of each first punctuation point, among which the first main content generally does not need to be processed, and the main purpose is to optimize the inaccurate terminal content, that is, to realize the inaccuracy caused by the existence of each first punctuation point, thereby improving the overall recognition accuracy.

[0073] Therefore, the management server inputs the end content corresponding to the front sub-audio stream of each first punctuation point and the two-stage sub-voice recognition text corresponding to the rear sub-audio stream into the preset large language model to output the revised end content corresponding to the front sub-audio stream of each first punctuation point; and combines the first main content and the revised end content corresponding to the front sub-audio stream of each first punctuation point to determine the revised two-stage sub-voice recognition text corresponding to the front sub-audio stream of the first punctuation point.

[0074] In a further embodiment, the step S104 further includes the following sub-steps:

[0075] The two-stage sub-speech recognition text corresponding to the latter sub-audio stream of each first punctuation point is preprocessed according to the preprocessing rules to obtain the corresponding second intermediate text; the second intermediate text and the two-stage sub-speech recognition text corresponding to the former sub-audio stream are input into a preset large language model to output the revised two-stage sub-speech recognition text corresponding to the latter sub-audio stream of the first punctuation point.

[0076] Similarly, the management server can determine the two-stage sub-speech recognition text corresponding to the front sub-audio stream and the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point, and pre-process the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point according to the pre-set pre-processing rules to obtain the corresponding second intermediate text. After the second intermediate text and the two-stage sub-speech recognition text corresponding to the back sub-audio stream are input into the preset large language model, the revised two-stage sub-speech recognition text corresponding to the back sub-audio stream of the first punctuation point can be obtained.

[0077] In a further specific embodiment, the second intermediate text includes a second main body content and a header content, and the step of preprocessing the second-stage sub-speech recognition text corresponding to the second-segment sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding second intermediate text includes:

[0078] The two-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point is respectively input into the semantic filter to obtain the first main content and header content corresponding to the rear-segment sub-audio stream of the first punctuation point; wherein the header content corresponding to the rear-segment sub-audio stream of the first punctuation point and the two-stage sub-speech recognition text corresponding to the front-segment sub-audio stream are input into the preset large language model to output the revised header content corresponding to the rear-segment sub-audio stream of the first punctuation point; the second main content and the revised header content corresponding to the rear-segment sub-audio stream of the first punctuation point are combined and determined as the revised two-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of the first punctuation point.

[0079] Similarly, since there are several first punctuation points, the management server needs to input the second-stage sub-voice recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point into the semantic filter respectively, so as to filter by the position and semantics of the content, and first preliminarily obtain the accurate second main content and inaccurate header content of the second-stage sub-voice recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point, among which the second main content generally does not need to be processed, and the inaccurate header content is mainly optimized, that is, the inaccuracy caused by the existence of each first punctuation point is realized, thereby improving the overall recognition accuracy.

[0080] Therefore, the management server inputs the header content corresponding to the rear-segment sub-audio stream of each first punctuation point and the two-stage sub-speech recognition text corresponding to the front-segment sub-audio stream into the preset large language model to output the revised header content corresponding to the rear-segment sub-audio stream of each first punctuation point; and combines the second main content and the revised header content corresponding to the rear-segment sub-audio stream of each first punctuation point to determine the revised two-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of the first punctuation point.

[0081] Step S150: combine all the corrected two-stage sub-speech recognition texts and the remaining uncorrected two-stage sub-speech recognition texts in chronological order to obtain a target speech recognition text corresponding to all the received audio streams.

[0082] In this embodiment, since the target speech recognition text corresponds to the received audio stream, after completing the sentence segmentation and speech recognition and the corresponding correction, it is necessary to combine the corrected two-stage sub-speech recognition text and the remaining uncorrected two-stage sub-speech recognition text according to the time sequence to obtain the target speech recognition text. The target speech recognition text can be input into a preset translation module for translation, that is, to achieve the so-called simultaneous interpretation, and the method can greatly improve its accuracy and efficiency.

[0083] For example, if the audio stream is interrupted due to an over-limit at the 10th second, the audio stream from 0 to 10 seconds is sent to the second-stage recognition model (i.e., the first speech recognition model) to obtain the text text1. At this time, the latter part of the text text1 may be inaccurate, so a semantic filter is required to filter the text text1 and obtain the accurate first main content and the inaccurate end content.

[0084] At the same time, since the audio stream after the 10th second triggers sentence punctuation at the 15th second (either end-of-sentence punctuation or forced punctuation is possible), the audio stream from the 10th to the 15th second is sent to the second-stage recognition model (i.e., the first speech recognition model) to obtain the text text2. At this time, the front content in the text text2 may not be accurately recognized, so a semantic filter is needed to filter the text text2 and obtain the accurate second main content and the inaccurate header content.

[0085] Finally, text1 and text2 are sent to the large language model to correct the content of the corresponding text around the 10th second; specifically, the end content of text1 and text2 are sent to the large language model to correct the end content of text1, and the beginning content of text1 and text2 are sent to the large language model to correct the beginning content of text2.

[0086] like Figure 3 In the application embodiment shown, it can be seen that after the audio stream is input into the streaming recognition model, the corresponding one-stage speech recognition text (one-stage ASR text) can be obtained until the preset audio stream end condition is triggered. The audio stream end condition can be set according to user needs and is not required in this embodiment.

[0087] When the audio stream input is finished, it enters the sentence segmentation state, that is, it determines whether the sentence is segmented. If it is not segmented, it continues to receive the audio stream input into the streaming recognition model. If the sentence is segmented, it is determined whether the segmentation type is the end-of-sentence segmentation or the over-limit forced segmentation (i.e. forced segmentation), and the audio stream after the segmentation is input into the two-stage model recognition (i.e. the first speech recognition model recognition). If it is a sentence-end segmentation, since the sentence-end segmentation is the situation where the speaker has finished speaking or paused, the audio input in this situation will be relatively complete, so after the two-stage model recognition processing, it can output a complete two-stage speech recognition text (i.e., the two-stage ASR text), and the sentence segmentation is also reasonable, and it can be directly output as a three-stage ASR text, that is, there is no need to correct and optimize the sentence-end segmentation. If it is an over-limit forced segmentation, since the situation is that the speaker speaks for a long time without pausing, because it is a forced segmentation, the audio at the end is incomplete, so the corresponding two-stage speech recognition text (i.e., the two-stage ASR text) is divided into two parts through a semantic filter, namely, the accurate main content and the inaccurate terminal content. Among them, the sentences and content in the main content are accurate and can be output as the three-stage ASR text; the terminal content will be inaccurate due to incomplete audio, and it is necessary to combine the next text to improve the content. A large language model is used to splice a piece of content to ensure content integrity, and the corrected text is output as the corrected three-stage ASR text corresponding to the terminal content. Finally, the contents of all the three-stage ASR texts are integrated together to obtain the target speech recognition text of the received audio stream.

[0088] In summary, the method of the above embodiment can improve the accuracy of speech recognition, that is, by correcting and optimizing the sentence segmentation position through a large language model, the restoration degree of the speech content is improved, and the sentences in the recognized speech recognition text are more fluent, so as to improve the accuracy and fluency of translation during simultaneous interpretation.

[0089] The embodiment of the present invention also provides a speech recognition device based on a large language model, which can be configured in a management server, and is used to execute any embodiment of the speech recognition method based on a large language model. Figure 4 , Figure 4 A schematic block diagram of a speech recognition device based on a large language model provided in an embodiment of the present invention.

[0090] like Figure 4 As shown, the speech recognition apparatus 100 based on the large language model includes a streaming recognition unit 110 , a sentence segmentation determination unit 120 , a speech recognition unit 130 , a text correction unit 140 and a target determination unit 150 .

[0091] The streaming recognition unit 110 is used to input the received real-time input audio stream into a preset streaming recognition model to obtain a corresponding first-stage speech recognition text until a preset audio stream end condition is triggered.

[0092] Among them, the audio stream refers to a real-time transmitted audio data stream. The streaming recognition model refers to a type of ASR (Automatic Speech Recognition) model that can support real-time return of recognition results in the process of processing the audio stream. In contrast, the non-streaming model must return the result after processing the complete sentence audio. Streaming ASR can be better used in scenarios where real-time acquisition of recognition results is required, such as live real-time subtitles, real-time recording of meetings, voice input, voice wake-up and other scenarios. In some embodiments, the streaming recognition model can be Zipformer, which is a new type of ASR model. Compared with mainstream ASR models such as Conformer, Squeezeformer, and E-Branchformer, Zipformer has the advantages of better effect, faster calculation, and more memory saving.

[0093] In this embodiment, the management server can input the received real-time input audio stream into a preset streaming recognition model, and output the corresponding first-stage speech recognition text through the streaming recognition model. The first-stage speech recognition text can be quickly displayed to the user for viewing, and is also convenient for subsequent process operations.

[0094] The segmentation determination unit 120 is used to determine whether the received audio stream needs to be segmented according to a preset segmentation condition and the first-stage speech recognition text, wherein the segmentation condition at least includes a forced segmentation condition.

[0095] In the present embodiment, the management server can also determine whether the received audio stream needs to be segmented according to the preset segmentation conditions and the obtained one-stage speech recognition text, that is, when the audio stream input is completed, the entire audio stream obtained is segmented, and the basis for the segmentation is the pre-set segmentation conditions. As an option, the segmentation conditions at least include a forced segmentation condition, which is of course not limited in the present embodiment. It can be generally believed that there is a certain error in the meaning between the adjacent voice texts obtained after the audio stream after segmentation by the forced segmentation condition is re-identified, which may lead to inaccurate recognition of the overall meaning, so it is necessary to continue with subsequent operations.

[0096] In one embodiment, the preset segmentation condition includes a forced segmentation condition, the segmentation point corresponding to the forced segmentation condition is the first segmentation point, the forced segmentation condition includes that the pause time between words does not exceed the preset time threshold and the cumulative number of recognized characters in the current stage reaches the preset number of characters, and the segmentation judgment unit 120 is further used to perform the following sub-steps:

[0097] Determine whether the pause duration between words in the received audio stream does not exceed a preset time threshold; if so, detect whether the cumulative number of characters in the first-stage speech recognition text reaches a preset number of characters according to the time sequence; if so, determine the time point corresponding to the last character in the accumulated characters as the first punctuation point of the corresponding audio stream for sentence segmentation, and return to the step of detecting whether the number of characters after the first punctuation point in the first-stage speech recognition text reaches a preset number of characters according to the time sequence, until all judgments on the received audio stream are completed.

[0098] The sentence segmentation judgment unit can judge whether the pause duration between words of the received audio stream does not exceed the preset time threshold according to the forced sentence segmentation condition. Generally, the preset time threshold can be determined according to the actual usage of the user, and is not specifically limited in this embodiment.

[0099] If the pause duration between words in the received audio stream does not exceed the preset time threshold, it indicates that the speaker is still speaking a complete sentence. In order to avoid errors in speech recognition and improve recognition accuracy, it is necessary to detect whether the cumulative number of characters in the speech recognition text of the first stage has reached the preset number of characters according to the time sequence, that is, to distinguish whether the cumulative number of characters currently continuously input and recognized has reached the preset number of characters. If so, it indicates that the audio stream needs to be forcibly segmented at this time, and the time point corresponding to the last character in the accumulated characters is determined as the first segmentation point of the corresponding audio stream for segmentation.

[0100] Then, after completing this segmentation, it is necessary to continue to segment the audio stream after the first segmentation point. If the pause duration between words in the received audio stream does not exceed the preset time threshold, it is necessary to perform the step of detecting whether the number of characters after the first segmentation point in the first-stage speech recognition text reaches the preset number of characters according to the time sequence until all judgments on the received audio stream are completed; that is, the power-off judgment on the entire audio stream is completed so as to perform subsequent operations.

[0101] In a further embodiment, the sentence segmentation condition further includes a sentence end segmentation condition, the sentence end segmentation condition includes that the pause duration between words exceeds a preset time threshold, and the segmentation point corresponding to the sentence end segmentation condition is a second segmentation point, and the sentence segmentation judgment unit 120 is further configured to perform the step of determining whether the pause duration between words of the received audio stream does not exceed the preset time threshold, and further includes a step of performing:

[0102] If not, the last speech time point before the pause of the received audio stream is taken as the second punctuation point, and the step of determining whether the pause duration between words of the received audio stream after the second punctuation point does not exceed the preset time threshold is returned.

[0103] Among them, if the pause duration between words of the received audio stream exceeds the preset time threshold, the last speech time point before the pause of the received audio stream is used as the second break point. After completing this break point, it is necessary to continue to return to the step of determining whether the pause duration between words of the received audio stream after the second break point does not exceed the preset time threshold. If it exceeds, continue to proceed to the sentence end point; if it does not exceed, continue to determine whether to perform forced break until all the audio stream break operations are completed.

[0104] For example, sentence segmentation can mainly rely on the following conditions, namely, the pause duration between words and the cumulative number of recognized characters in the current stage. When the pause duration between words in the audio stream exceeds 500ms, it is understood as a complete sentence and the end-of-sentence segmentation logic is executed; if the pause between words does not exceed 500ms and the cumulative number of recognized characters in the current stage exceeds 60 words, sentence segmentation is also executed, which is forced segmentation. The main difference between the two is that sentence segmentation based on the pause duration usually results in a complete sentence, and the recognition result sent to the first speech recognition model is relatively good, while forced segmentation usually does not result in a complete sentence, and it is possible that the sentence is cut in the middle of the pronunciation of a word or a single word. At this time, the middle and back parts of the recognition result sent to the first speech recognition model are relatively poor. By the same token, the beginning of the next audio segment after the segmentation is also a similar situation, so the relevant recognition results need to be corrected at this time.

[0105] The speech recognition unit 130 is used to determine the corresponding punctuation points, and divide the audio stream into a plurality of sub-audio streams in chronological order according to the determined punctuation points, and input all the sub-audio streams into a preset first speech recognition model in turn to obtain the corresponding second-stage sub-speech recognition text; wherein the punctuation point corresponding to the forced punctuation condition is the first punctuation point.

[0106] In this embodiment, if it is determined that the received audio stream needs to be segmented based on the preset segmentation conditions and the first-stage speech recognition text, then the corresponding segmentation points need to be determined at this time, and the audio stream is divided into several sub-audio streams in chronological order according to the determined segmentation points, and all the sub-audio streams are input into the preset first speech recognition model in sequence to obtain the corresponding second-stage sub-speech recognition text.

[0107] For example, the model used by the first speech recognition model can be Paraformer, which is characterized by inputting a whole audio segment and recognizing the text result. The recognition accuracy is greatly improved compared with the streaming recognition model because it takes the content of the whole sentence into consideration and can be combined with semantic recognition, homophone recognition, near-phonetic character recognition, etc., so the recognition accuracy is much higher. However, due to the existence of forced sentence punctuation, the recognition result of the audio stream part after forced sentence punctuation needs to be subsequently corrected before it can be sent to the translation process with the recognition result of the sentence end punctuation part to realize simultaneous interpretation, which effectively reduces the delay time of simultaneous interpretation and improves the corresponding accuracy.

[0108] The text correction unit 140 is used to input the second-stage sub-speech recognition texts related to different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts.

[0109] In this embodiment, the two-stage sub-voice recognition text related to the first punctuation point is determined. Usually, the two-stage sub-voice recognition text related to the first punctuation point can be the two-stage sub-voice recognition text corresponding to the front sub-frequency stream located in the segment before the first punctuation point, or it can be the two-stage sub-voice recognition text corresponding to the rear sub-frequency stream located in the segment after the first punctuation point.

[0110] The two-stage sub-speech recognition texts related to different first punctuation points are input into the preset large language model in batches, so that a plurality of corresponding corrected two-stage sub-speech recognition texts can be obtained, that is, for each first punctuation point that appears, the corresponding correction of the two-stage sub-speech recognition text is completed.

[0111] In one embodiment, the text correction unit 140 is further configured to perform the following sub-steps:

[0112] Determine the two-stage sub-speech recognition text corresponding to the front sub-audio stream and the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point; preprocess the two-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding first intermediate text; input the first intermediate text and the two-stage sub-speech recognition text corresponding to the back sub-audio stream into a preset large language model to output the revised two-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point.

[0113] The text correction unit 140 can determine the two-stage sub-speech recognition text corresponding to the front sub-audio stream and the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point, and pre-process the two-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the pre-set pre-processing rules to obtain the corresponding first intermediate text. After the first intermediate text and the two-stage sub-speech recognition text corresponding to the back sub-audio stream are input into the preset large language model, the corrected two-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point can be obtained.

[0114] In a specific embodiment, the first intermediate text includes a first main content and an end content, and the text correction unit 140 is further configured to perform the step of preprocessing the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding first intermediate text, specifically including performing the following sub-steps:

[0115] The two-stage sub-speech recognition texts corresponding to the front sub-audio stream of each first punctuation point are respectively input into the semantic filter to obtain the first main content and the end content corresponding to the front sub-audio stream of each first punctuation point; wherein the end content corresponding to the front sub-audio stream of each first punctuation point and the two-stage sub-speech recognition text corresponding to the back sub-audio stream are input into the preset large language model to output the revised end content corresponding to the front sub-audio stream of each first punctuation point; the first main content and the revised end content corresponding to the front sub-audio stream of each first punctuation point are combined to determine as the revised two-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point.

[0116] Among them, since there are several first punctuation points, the text correction unit 140 needs to input the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point into the semantic filter respectively, so as to filter by the position and semantics of the content, and first preliminarily obtain the accurate first main content and inaccurate terminal content of the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point, among which the first main content generally does not need to be processed, and the inaccurate terminal content is mainly optimized, that is, the inaccuracy caused by the existence of each first punctuation point is realized, thereby improving the overall recognition accuracy.

[0117] Therefore, the text correction unit 140 inputs the end content corresponding to the front sub-audio stream of each first punctuation point and the two-stage sub-voice recognition text corresponding to the rear sub-audio stream into the preset large language model to output the corrected end content corresponding to the front sub-audio stream of each first punctuation point; and combines the first main content and the corrected end content corresponding to the front sub-audio stream of each first punctuation point to determine the corrected two-stage sub-voice recognition text corresponding to the front sub-audio stream of the first punctuation point.

[0118] In a further embodiment, the text correction unit 140 is further configured to perform the following sub-steps:

[0119] The two-stage sub-speech recognition text corresponding to the latter sub-audio stream of each first punctuation point is preprocessed according to the preprocessing rules to obtain the corresponding second intermediate text; the second intermediate text and the two-stage sub-speech recognition text corresponding to the former sub-audio stream are input into a preset large language model to output the revised two-stage sub-speech recognition text corresponding to the latter sub-audio stream of the first punctuation point.

[0120] Similarly, the text correction unit 140 can determine the two-stage sub-speech recognition text corresponding to the front sub-audio stream and the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point, and pre-process the two-stage sub-speech recognition text corresponding to the back sub-audio stream of each first punctuation point according to the pre-set pre-processing rules to obtain the corresponding second intermediate text. After the second intermediate text and the two-stage sub-speech recognition text corresponding to the back sub-audio stream are input into the preset large language model, the corrected two-stage sub-speech recognition text corresponding to the back sub-audio stream of the first punctuation point can be obtained.

[0121] In a further specific embodiment, the second intermediate text includes a second main body content and a header content, and the text correction unit 140 is used to perform the step of preprocessing the second-stage sub-speech recognition text corresponding to the second-segment sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding second intermediate text, specifically for performing the following sub-steps:

[0122] The two-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point is respectively input into the semantic filter to obtain the first main content and header content corresponding to the rear-segment sub-audio stream of the first punctuation point; wherein the header content corresponding to the rear-segment sub-audio stream of the first punctuation point and the two-stage sub-speech recognition text corresponding to the front-segment sub-audio stream are input into the preset large language model to output the revised header content corresponding to the rear-segment sub-audio stream of the first punctuation point; the second main content and the revised header content corresponding to the rear-segment sub-audio stream of the first punctuation point are combined and determined as the revised two-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of the first punctuation point.

[0123] Similarly, since there are several first punctuation points, the text correction unit 140 needs to input the second-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point into the semantic filter respectively, so as to filter by the position and semantics of the content, and first preliminarily obtain the accurate second main content and inaccurate header content of the second-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point, among which the second main content generally does not need to be processed, and the inaccurate header content is mainly optimized, that is, the inaccuracy caused by the existence of each first punctuation point is realized, thereby improving the overall recognition accuracy.

[0124] Therefore, the text correction unit 140 inputs the header content corresponding to the rear sub-audio stream of each first punctuation point and the two-stage sub-speech recognition text corresponding to the front sub-audio stream into the preset large language model to output the corrected header content corresponding to the rear sub-audio stream of each first punctuation point; and combines the second main content and the corrected header content corresponding to the rear sub-audio stream of each first punctuation point to determine the corrected two-stage sub-speech recognition text corresponding to the rear sub-audio stream of the first punctuation point.

[0125] The target determination unit 150 is used to combine all the corrected two-stage sub-speech recognition texts and the remaining uncorrected two-stage sub-speech recognition texts in time order to obtain a target speech recognition text corresponding to all the received audio streams.

[0126] In this embodiment, since the target speech recognition text corresponds to the received audio stream, after completing the sentence segmentation and speech recognition and the corresponding correction, it is necessary to combine the corrected two-stage sub-speech recognition text and the remaining uncorrected two-stage sub-speech recognition text according to the time sequence to obtain the target speech recognition text. The target speech recognition text can be input into a preset translation module for translation, that is, to achieve the so-called simultaneous interpretation, and the method can greatly improve its accuracy and efficiency.

[0127] The above-mentioned speech recognition device based on the large language model can be implemented in the form of a computer program, and the computer program can be used in Figure 5 Runs on the computer device shown.

[0128] See also Figure 5 , Figure 5 : is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device may be a user terminal or a server for executing a speech recognition method based on a large language model to automatically compare prices according to a speech recognition module based on a large language model imported into an operating system.

[0129] See also Figure 5 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a communication bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0130] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a speech recognition method based on a large language model, wherein the storage medium 503 may be a volatile storage medium or a non-volatile storage medium.

[0131] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0132] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a speech recognition method based on a large language model.

[0133] The network interface 505 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] The processor 502 is used to run the computer program 5032 stored in the memory to implement the corresponding functions in the above-mentioned speech recognition method based on the large language model.

[0135] Those skilled in the art will understand that Figure 5The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 5 The embodiments shown are consistent and will not be described again here.

[0136] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0137] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps included in the above-mentioned speech recognition method based on a large language model are implemented.

[0138] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0139] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0141] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.

[0143] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A speech recognition method based on a large language model, characterized in that: The method is applied to a management server, and the method comprises: Input the received real-time input audio stream into a preset streaming recognition model to obtain the corresponding first-stage speech recognition text until the preset audio stream end condition is triggered; Determining whether the received audio stream needs to be segmented according to a preset segmentation condition and the first-stage speech recognition text, wherein the segmentation condition at least includes a forced segmentation condition; If yes, then determine the corresponding punctuation point, and divide the audio stream into a plurality of sub-audio streams in chronological order according to the determined punctuation points, and input all the sub-audio streams into a preset first speech recognition model in turn to obtain the corresponding second-stage sub-speech recognition text; wherein the punctuation point corresponding to the mandatory punctuation condition is the first punctuation point; Inputting the second-stage sub-speech recognition texts associated with different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts; Combining all the corrected two-stage sub-speech recognition texts and the remaining uncorrected two-stage sub-speech recognition texts in chronological order to obtain a target speech recognition text corresponding to all the received audio streams; The mandatory segmentation condition includes that the pause time between words does not exceed a preset time threshold and the cumulative number of characters recognized in the current stage reaches a preset number of characters. The step of determining whether the received audio stream needs to be segmented according to the preset segmentation condition and the first-stage speech recognition text includes: Determine whether the pause duration between words of the received audio stream does not exceed a preset time threshold; If so, detecting whether the cumulative number of characters in the speech recognition text of the first stage reaches a preset number of characters according to the time sequence; If yes, the time point corresponding to the last character in the accumulated characters is determined as the first punctuation point of the corresponding audio stream for punctuation, and the step of detecting whether the number of characters after the first punctuation point in the first-stage speech recognition text reaches the preset number of characters according to the time sequence is returned to, until all judgments on the received audio stream are completed; The step of inputting the two-stage sub-speech recognition texts associated with different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected two-stage sub-speech recognition texts includes: Determine the second-stage sub-speech recognition text corresponding to the front-segment sub-audio stream and the second-stage sub-speech recognition text corresponding to the rear-segment sub-audio stream of each first punctuation point; Preprocessing the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding first intermediate text; The two-stage sub-speech recognition texts corresponding to the first intermediate text and the second-stage sub-audio stream are input into a preset large language model to output a revised two-stage sub-speech recognition text corresponding to the first-stage sub-audio stream of the first punctuation point.

2. The speech recognition method based on a large language model according to claim 1, characterized in that: The sentence segmentation condition further includes a sentence end segmentation condition, wherein the sentence end segmentation condition includes that the pause duration between words exceeds a preset time threshold, and the segmentation point corresponding to the sentence end segmentation condition is a second segmentation point. After the step of determining whether the pause duration between words of the received audio stream does not exceed the preset time threshold, the step further includes: If not, the last speech time point before the pause of the received audio stream is taken as the second punctuation point, and the step of determining whether the pause duration between words of the received audio stream after the second punctuation point does not exceed the preset time threshold is returned.

3. The speech recognition method based on a large language model according to claim 1, characterized in that: The first intermediate text includes a first main body content and an end content, and the step of preprocessing the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding first intermediate text includes: Inputting the second-stage sub-speech recognition text corresponding to the front sub-audio stream of each first punctuation point into the semantic filter respectively to obtain the first main content and the end content corresponding to the front sub-audio stream of each first punctuation point; The end content corresponding to the front sub-audio stream of each first punctuation point and the two-stage sub-speech recognition text corresponding to the rear sub-audio stream are input into a preset large language model to output the corrected end content corresponding to the front sub-audio stream of each first punctuation point; The first main content corresponding to the front sub-audio stream of each first punctuation point and the modified tail content are combined to determine the modified second-stage sub-speech recognition text corresponding to the front sub-audio stream of the first punctuation point.

4. The speech recognition method based on a large language model according to claim 1, characterized in that: The method further comprises: Preprocessing the second-stage sub-speech recognition text corresponding to the sub-audio stream of the latter segment of each first punctuation point according to the preprocessing rule to obtain the corresponding second intermediate text; The second intermediate text and the second-stage sub-speech recognition text corresponding to the first sub-audio stream are input into a preset large language model to output a revised second-stage sub-speech recognition text corresponding to the second sub-audio stream of the first punctuation point.

5. The speech recognition method based on a large language model according to claim 4, characterized in that: The second intermediate text includes a second main body content and a header content, and the step of preprocessing the second-stage sub-speech recognition text corresponding to the second-segment sub-audio stream of each first punctuation point according to the preprocessing rule to obtain the corresponding second intermediate text includes: Inputting the second-stage sub-speech recognition text corresponding to the second-segment sub-audio stream of each first punctuation point into the semantic filter respectively to obtain the first main content and the header content corresponding to the second-segment sub-audio stream of the first punctuation point; The header content corresponding to the second segment of the sub-audio stream of the first punctuation point and the second-stage sub-speech recognition text corresponding to the first segment of the sub-audio stream are input into a preset large language model to output the corrected header content corresponding to the second segment of the sub-audio stream of the first punctuation point; The second main content corresponding to the second segment of the sub-audio stream at the first punctuation point and the modified header content combination are determined as the modified second-stage sub-speech recognition text corresponding to the second segment of the sub-audio stream at the first punctuation point.

6. A speech recognition device based on a large language model, characterized in that: The speech recognition device based on the large language model is used to execute the speech recognition method based on the large language model according to any one of claims 1 to 5, and the device is applied to a management server, and the device includes: A streaming recognition unit, used to input the received real-time input audio stream into a preset streaming recognition model to obtain a corresponding first-stage speech recognition text until a preset audio stream end condition is triggered; A sentence segmentation judgment unit, used for judging whether the received audio stream needs to be segmented according to a preset sentence segmentation condition and the first-stage speech recognition text, wherein the sentence segmentation condition at least includes a forced sentence segmentation condition; A speech recognition unit, for determining a corresponding punctuation point if yes, and dividing the audio stream into a plurality of sub-audio streams in chronological order according to the determined punctuation points, and inputting all the sub-audio streams into a preset first speech recognition model in turn to obtain a corresponding second-stage sub-speech recognition text; wherein the punctuation point corresponding to the mandatory punctuation condition is the first punctuation point; A text correction unit, used for inputting the second-stage sub-speech recognition texts associated with different first punctuation points into a preset large language model in batches to obtain a plurality of corresponding corrected second-stage sub-speech recognition texts; The target determination unit is used to combine all the corrected two-stage sub-speech recognition texts and the remaining uncorrected two-stage sub-speech recognition texts in time sequence to obtain a target speech recognition text corresponding to all the received audio streams.

7. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the speech recognition method based on a large language model described in any one of claims 1 to 5 when executing the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech recognition method based on a large language model as described in any one of claims 1 to 5 are implemented.

Images