A large model-based real-time english speech recognition method and system

Abstract

The present application relates to speech recognition technology, in particular to a real-time English speech recognition method and system based on a large model, the method comprising: splicing the current received audio segment with a length of T with historical audio segments and inputting the speech recognition model; if an end symbol appears in the result of speech recognition, the current recognition result is used to replace the last recognition result; using a dynamic linear programming algorithm, the timestamp of each word of the audio input to the speech recognition model is calculated, the end time of the word before the end symbol is used as the starting position, the audio before the starting position is discarded, all recognition results after the end symbol are discarded and used as historical audio segments, the end symbol and the previous recognition result are marked as splicable and fed back to the front end; if no end symbol appears, the current recognition result is spliced behind the last recognition result; the present application meets real-time while improving the accuracy of speech recognition.

Description

Technical Field

[0001] This invention relates to speech recognition technology, and in particular to a real-time English speech recognition method and system based on a large model. Background Technology

[0002] Currently, artificial intelligence (AI) technology is being used more and more widely. Applying AI to speech recognition can significantly improve accuracy. In the field of real-time speech recognition, to meet the requirements of response speed, a relatively small model is generally used to continuously recognize speech segments of less than one second transmitted from the front end, and then all the recognition results are spliced ​​together as the final result. However, the above methods generally suffer from low accuracy, insufficient consideration of context, and the need for other external models to predict punctuation marks and perform text normalization.

[0003] To improve accuracy, large models can be used for speech recognition. Due to their deeper network structure, large models can learn more knowledge during training than traditional models, and the prediction accuracy will be improved accordingly. However, the forward inference process of large models is slower. Unlike streaming speech recognition models, they cannot directly input all audio data into the model and obtain the output. They also cannot be connected to additional models to complete other necessary functions, such as timestamp prediction and text normalization. Otherwise, the already slow inference speed will be slowed down even more, making it impossible to meet real-time requirements.

[0004] Therefore, how to improve accuracy while ensuring the real-time performance of speech recognition is a problem that existing technologies urgently need to solve. Summary of the Invention

[0005] To address the shortcomings of existing English speech recognition models, such as low accuracy and limited functionality, this invention provides a method for applying a large model to real-time English speech recognition. This method employs a seq2seq architecture as the recognition model, which includes a decoder and an encoder. The speech recognition process using this model specifically includes the following steps:

[0006] 100. The front end sends audio to the speech recognition model at fixed intervals T, and the length of each audio segment is T;

[0007] 101. Concatenate the currently received audio segment of length T with historical audio segments and input it into the speech recognition model;

[0008] 102. If no end-of-speech character appears in the speech recognition result, proceed to step 103; otherwise, proceed to step 105.

[0009] 103. Determine whether the length of the audio segment of length T after splicing with the historical audio segment is greater than the threshold of the historical audio segment. If it is greater, obtain the start time of the word based on the timestamp of the last word in the recognition result, retain the audio segment from the start time of the word to the end of the spliced ​​audio as the historical audio segment, and mark the recognition result as "splicing", and proceed to step 107.

[0010] 104. Otherwise, directly concatenate the audio segment of length T with the historical audio segment as the historical audio segment, and mark the current recognition result as "can be replaced";

[0011] 105. Calculate the timestamp of the word preceding the end-of-word character, obtain the end time of the word based on the timestamp, and retain the audio segment from the end time of the word to the end of the concatenated audio as a historical audio segment.

[0012] 106. Discard the recognition results after the end symbol appears, and mark the current recognition result as "splicing";

[0013] 107. When the front end receives the current recognition result and its tag, if the previous tag was "can be replaced", then the current recognition result is used to replace the previous recognition result; if the previous tag was "concatenate", then the current recognition result is concatenated to the end of the previous recognition result.

[0014] 108. Receive the audio segment of the next time step and return to step 101.

[0015] Furthermore, the acquisition of the speech recognition model includes: using OpenAI's whisper-large model as a pre-trained model, and using the original speech data as training data to fine-tune the pre-trained model.

[0016] Furthermore, the format of the fine-tuned speech recognition model is transformed using the ctranslation2 framework.

[0017] Furthermore, the timestamps of each word are calculated sequentially backward from the last word in the decoder's predicted output, specifically including the following steps:

[0018] Calculate the timestamp of the last word in the decoder's predicted output. Start from the audio features of the last time step of the input encoder and traverse each time step of the encoder from the end to the beginning.

[0019] Each time a new encoder time step is reached, the normalized matching score between the current word and the audio features of the current encoder time step must be calculated through scaled dot product attention.

[0020] If the matching score between the current word and the audio feature at the current time step is less than 0.2, or the matching score between the previous word and the audio feature at the current time step is greater than the matching score between the current word and the audio feature of the segment, then stop calculating the matching score between the current word and other audio features, and record the audio feature at time step t in the encoder.

[0021] The audio features that meet the requirements are merged together to form an audio segment. The start time and end time of the audio segment are calculated to obtain the timestamp of the current word.

[0022] Using time step t as the starting time step for the next word, continue calculating the time steps that meet the requirements. Calculate the timestamp based on the time steps that meet the requirements, and repeat the above steps to calculate the timestamps for all words.

[0023] Furthermore, if there is silence of more than 1 second in the audio, the starting position is the position where the silence ends. The audio before the starting position is discarded, and the part from the starting position to the end of the audio is retained for the next audio splicing. At the same time, only the recognition result corresponding to the discarded audio is retained, and a period is added to the end of the word sequence of the recognition result. Then, the recognition result is marked as ready for splicing, and the result is returned to the front end.

[0024] This invention also proposes a real-time English speech recognition system based on a large model, including a speech acquisition module, a speech splicing module, a speech recognition module, a terminator detection module, a timestamp calculation module, a detection result push module, and a historical speech construction module, wherein:

[0025] The voice acquisition module is used to send audio to the voice recognition model at fixed intervals T, with each audio segment having a length of T.

[0026] The voice splicing module is used to splice the currently received audio segment of length T with historical audio segments;

[0027] The speech recognition module is used to perform speech recognition based on the concatenated speech obtained by the speech concatenation module. This model adopts a large seq2seq architecture, which includes a decoder and an encoder. The encoder extracts audio features from the input audio, and the decoder outputs the prediction result based on the audio features.

[0028] The terminator detection module is used to detect whether a terminator appears in the recognition result obtained by the speech recognition module.

[0029] The timestamp calculation module is used to generate a timestamp for each word output by the speech recognition module. The timestamp includes the start and end times of the word in the speech segment input to the speech recognition module.

[0030] The detection result push module replaces the previous recognition result with the current recognition result if an end-of-response character is present, and marks the end-of-response character and the previous recognition result as ready for concatenation and sends the result back to the front end; if no end-of-response character is present, the current recognition result is concatenated to the end of the previous recognition result, while discarding the last word of the recognition result.

[0031] In the historical audio construction module, if no end-of-speech character appears and the audio segment of the input speech model is not greater than the threshold of the set historical audio segment length, the audio segment of the input speech model is directly used as the historical audio segment. Otherwise, the start time of the last word in the recognition result is obtained based on the timestamp of the last word, and the audio segment from the start time of the last word to the end of the concatenated audio is retained as the historical audio segment. If an end-of-speech character appears, the timestamp of the word preceding the end-of-speech character is calculated, the end time of the last word is obtained based on the timestamp, and the audio segment from the end time of the last word to the end of the concatenated audio is retained as the historical audio segment.

[0032] This invention employs a large-size model and uses longer speech segments for speech recognition. The combination of these two approaches significantly improves the accuracy of speech recognition. Existing speech recognition technologies typically use a speech segment as input and output a clean text without punctuation or consideration of capitalization or numerical representation, requiring external models for word segmentation and other further processing. However, the speech recognition model constructed in this invention can directly output text with punctuation at one end, without the need for external modules. Attached Figure Description

[0033] Figure 1 This is a flowchart of a real-time English speech recognition method based on a large model according to the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This invention proposes a real-time English speech recognition method based on a large model. It employs a seq2seq architecture as the recognition model, which includes a decoder and an encoder. The speech recognition process using this model specifically includes the following steps:

[0036] 100. The front end sends audio to the speech recognition model at fixed intervals T, and the length of each audio segment is T;

[0037] 101. Concatenate the currently received audio segment of length T with historical audio segments and input it into the speech recognition model;

[0038] 102. If no end-of-speech character appears in the speech recognition result, proceed to step 103; otherwise, proceed to step 105.

[0039] 103. Determine whether the audio obtained by splicing the audio segment of length T with the historical audio segment is greater than the threshold of the historical audio segment. If it is greater, obtain the start time of the word based on the timestamp of the last word in the recognition result, retain the audio segment from the start time of the word to the end of the spliced ​​audio as the historical audio segment, and mark the recognition result as "splicing", and proceed to step 107.

[0040] 104. Otherwise, directly concatenate the audio segment of length T with the historical audio segment as the historical audio segment, and mark the current recognition result as "can be replaced";

[0041] 105. Calculate the timestamp of the word preceding the end-of-word character, obtain the end time of the word based on the timestamp, and retain the audio segment from the end time of the word to the end of the concatenated audio as a historical audio segment.

[0042] 106. Discard the recognition results after the end symbol appears, and mark the current recognition result as "splicing";

[0043] 107. When the front end receives the current recognition result and its tag, if the previous tag was "can be replaced", then the current recognition result is used to replace the previous recognition result; if the previous tag was "concatenate", then the current recognition result is concatenated to the end of the previous recognition result.

[0044] 108. Receive the audio segment of the next time step and return to step 101.

[0045] In this embodiment, the large model is a seq2seq architecture, which includes a decoder and an encoder. When predicting words, the decoder uses scaled dot product attention to calculate the normalized matching score between the word and the audio features of the input encoder at each time step. To prevent further slowing down the overall speech recognition system, this embodiment does not introduce an additional timestamp prediction model; it only uses these normalized matching scores to calculate the timestamp of each word. The specific method is as follows:

[0046] A dynamic linear programming algorithm is introduced to calculate the matching score between the last word predicted by the decoder and the audio features of the input encoder at each time step, starting from the last word predicted by the decoder.

[0047] Starting from the last time step of the encoder, iterate backwards. If the matching score is greater than 0.2, it is determined that the audio feature of that encoder time step participated in the decoding of the word. Then continue to iterate backwards until the matching score is less than 0.2, or the matching score of the previous word on the current audio feature is greater than the matching score of the current word on the current audio feature. At this point, stop iterating backwards, record the encoder time step t, merge all the audio features that meet the requirements into an audio segment, calculate the start and end times of the audio segment, and the result is the timestamp of the current word.

[0048] Next, calculate the timestamp of the second-to-last word predicted by the decoder. At this point, start from time step t in the encoder and traverse the audio features backward, repeating the above steps to calculate the timestamp until the timestamp of the first word predicted by the decoder is obtained.

[0049] By using the above method, we can determine which time step audio features participated in the decoding of the current predicted word, thereby confirming the timestamp of each predicted word.

[0050] For example, if the total length of the audio input to the encoder is 2 seconds, divided into 40 time steps, then the length of each audio feature segment is 50ms. For the last word predicted by the decoder, assuming that the audio feature matching score of the 30th to 40th encoder time steps meets the requirements, then the start time of this word is 30 * 50ms = 1.5s, and the end time is 40 * 50ms = 2s. The end time of the second to last word is calculated to be 1.5s. Starting from 1.5s, the second to last word is traversed backward to obtain the time steps that meet the requirements of the word. The timestamp is calculated based on the time steps that meet the requirements, and so on, to calculate the timestamps of all words.

[0051] Specifically, the present invention provides a process for calculating a matching score through scaled dot product attention, including the following steps: multiplying the semantic vector output by the last layer of the decoder at the current time step with the weight matrix of a query vector as the query vector Q;

[0052] The audio feature vectors output from the last layer of the encoder at each time step are put together to form an audio feature matrix. The value of multiplying this matrix by the weight matrix of a key vector is used as the key vector K.

[0053] The audio feature vectors output from the last layer of the encoder at each time step are put together to form an audio feature matrix. The value vector V is obtained by multiplying this matrix by the weight matrix of a value vector.

[0054] The matching score is calculated based on the query vector Q, key vector K, and value vector V, i.e.:

[0055]

[0056] Where Attention(Q,K,V) is the matching score; d k is the dimension of the audio feature vector.

[0057] Existing real-time speech recognition systems, in pursuit of extreme response speed, typically employ small-sized models, with input speech segments generally ranging from 300ms to 600ms in length. Furthermore, they can only perform speech recognition tasks, while punctuation prediction and text normalization are directly fed to subsequent models. This approach leads to three problems:

[0058] 1. Small-sized models have fewer parameters, so even with a lot of training data, they cannot fully learn speech features during training.

[0059] 2. Short input speech segments can cause the model to fail to fully consider the context when predicting and recognizing results. For words with non-standard pronunciation or ambiguity, it cannot give accurate results at all.

[0060] 3. Directly assigning punctuation prediction and text normalization tasks to subsequent models will incur additional time overhead and make the entire speech recognition chain too complex, which is not conducive to system maintenance.

[0061] In this embodiment, the pre-trained model used is OpenAI's Whisper-Large, which is based on the Transformer architecture, has 1.55G of parameters, and was pre-trained using 680,000 hours of multilingual speech data.

[0062] In this embodiment, 20,000 hours of English speech data were used for fine-tuning. Unlike other real-time speech recognition models, the text corresponding to this speech data was not removed for punctuation or underwent inverse text normalization before fine-tuning. Instead, the original text was used directly, which preserves the model's punctuation prediction and text normalization capabilities. Secondly, after fine-tuning, in order to enable the model to predict timestamps at the word level, we took the alignment matrix generated by the decoder part and used these normalized scores to perform dynamic linear programming to calculate a 0.01ms-level timestamp for each word recognized by the model. This part does not require fine-tuning.

[0063] Furthermore, after fine-tuning the model, this embodiment uses the ctranslation2 framework to convert the model's format. The converted model occupies half the GPU memory of the original model and has a response time of one-third, greatly improving performance.

[0064] In this embodiment, during actual use, the front end sends audio to the model at 1.5-second intervals, with each audio segment also lasting 1.5 seconds. After receiving new audio, it is concatenated with previously saved audio, and the concatenated audio is used to predict the recognition result. After obtaining the recognition result, some processing is performed before returning it to the front end and displaying it to the user. At the same time, the concatenated audio is saved for subsequent operations. The maximum length of the saved audio is 6 seconds. With the addition of the new audio, the maximum length of audio that the model needs to process is 7.5 seconds, at which point the model's response time is approximately 1.2 seconds. It can be assumed that when the user speaks, the recognition result of each 1.5-second audio segment, counting from the end of the audio, can be displayed to the user at the latest 1.2 seconds later (not considering data transmission time). Such a delay is acceptable.

[0065] Each time a new audio is obtained, it is concatenated with the saved audio and sent to the model to generate a recognition result. If the recognition result does not contain a period, question mark, exclamation mark, semicolon, or other ending punctuation, it is marked as new audio. If the previous recognition result was marked as being able to be concatenated, the current recognition result is used to replace the previous recognition result. If the previous recognition result was marked as being able to be concatenated, the current prediction result is concatenated to the end of the previous recognition result. After completing the above processing, the result is returned to the front end.

[0066] As an optional implementation, if the recognition result has an end punctuation mark, it is necessary to calculate the timestamp of each word in the entire audio segment. Taking the end time of the word before the end punctuation mark as the starting position, the audio before the starting position is discarded, and the part from the starting position to the end of the audio is retained for the next audio splicing. At the same time, all recognition results after the end punctuation mark are discarded, and the recognition results before the end punctuation mark are marked as splicing possible. Finally, the result is returned to the front end.

[0067] As an optional implementation, if the length of the spliced ​​audio is greater than 6 seconds and there is no end punctuation in the recognition result, it is necessary to calculate the timestamp of each word in the entire audio segment, take the start time of the last word as the starting position, discard the audio before the starting position, retain the part from the starting position to the end of the audio for the next audio splicing, discard the last word of the recognition result, mark the recognition result as ready for splicing, and finally return the result to the front end.

[0068] In this embodiment, a lightweight silence detection tool is also used to assist in punctuation prediction. If there is silence of more than 1 second in the audio, the audio before the start position is discarded, and the part from the start position to the end of the audio is retained for the next audio splicing. At the same time, only the recognition result corresponding to the discarded audio is retained, and a period is added to the end of the word sequence. Then the recognition result is marked as a splicing state, and finally the result is returned to the front end.

[0069] This embodiment also proposes a real-time English speech recognition system based on a large model, including a speech acquisition module, a speech splicing module, a speech recognition module, a terminator detection module, a timestamp calculation module, a detection result push module, and a historical speech construction module, wherein:

[0070] The voice acquisition module is used to send audio to the voice recognition model at fixed intervals T, with each audio segment having a length of T.

[0071] The voice splicing module is used to splice the currently received audio segment of length T with historical audio segments;

[0072] The speech recognition module is used to perform speech recognition based on the concatenated speech obtained by the speech concatenation module. This model adopts a large seq2seq architecture, which includes a decoder and an encoder. The encoder extracts audio features from the input audio, and the decoder outputs the prediction result based on the audio features.

[0073] The terminator detection module is used to detect whether a terminator appears in the recognition result obtained by the speech recognition module.

[0074] The timestamp calculation module is used to generate a timestamp for each word output by the speech recognition module. The timestamp includes the start and end times of the word in the speech segment input to the speech recognition module.

[0075] The detection result push module pushes the current recognition result based on the tag of the previous recognition result when the front end receives the current recognition result. That is, if the previous tag is "can be replaced", the current recognition result is used to replace the previous recognition result; if the previous tag is "to be spliced", the current recognition result is spliced ​​to the end of the previous recognition result.

[0076] In the historical audio construction module, if no end-of-speech character appears and the audio segment of the input speech model is not greater than the set threshold for the length of the historical audio segment, the audio segment of the input speech model is directly used as the historical audio segment. Otherwise, the start time of the last word in the recognition result is obtained based on its timestamp, and the audio segment from the start time of that word to the end of the concatenated audio is retained as the historical audio segment. If an end-of-speech character appears, the timestamp of the word preceding the end-of-speech character is calculated, and the end time of that word is obtained based on its timestamp. The audio segment from the end time of that word to the end of the concatenated audio is retained as the historical audio segment. Figure 1 This embodiment uses a 1.5s (unit: seconds / s) audio segment as an example to further illustrate the speech recognition process of the present invention. In this embodiment, the process of recognizing real-time acquired 1.5s interval audio includes the following steps:

[0077] If the front end sends audio to the model at 1.5-second intervals, the length of each audio segment is also 1.5 seconds;

[0078] After obtaining the latest 1.5s audio, it is spliced ​​together with the audio previously saved in the model. The audio saved locally is the previously received audio, and a maximum of 6s of historical audio is retained.

[0079] The retained audio is concatenated with the latest audio and used as the input to the speech recognition model. In this embodiment, the audio length input to the speech recognition model is 1.5s to 7.5s.

[0080] The speech recognition model has a response time of approximately 1.2 seconds for recognizing audio of 7.5 seconds in length. This means that if the recognition result of audio is obtained every 1.5 seconds while the user is speaking, it can be displayed to the user at the latest 1.2 seconds after the end of the audio (not considering data transmission time). Such a delay does not affect the user experience in this field.

[0081] Each piece of audio is input into the speech recognition model for recognition. If the recognition result does not contain a period, question mark, exclamation mark, semicolon, or other closing character, the recognition result is marked as "can be replaced". If the previous recognition result was marked as "can be replaced", then the current recognition result is used to replace the previous recognition result.

[0082] If the previous recognition result is marked as "sembroidery", then the current prediction result is appended to the previous recognition result; after completing the above processing, the result is returned to the front end.

[0083] After the feedback is sent to the front end, it is necessary to determine whether the length of the concatenated audio, i.e. the length of the input model, is greater than 6 seconds. If it is not greater, it can be directly saved locally and concatenated with the next 1.5-second input audio.

[0084] If it is longer than 6 seconds, the timestamp of each word in the entire audio needs to be calculated. The starting position is the end time of the word before the end punctuation mark. The audio before the starting position is discarded, and the part from the starting position to the end of the audio is kept for the next audio splicing. At the same time, all recognition results after the end punctuation mark are discarded, and the recognition results before the end punctuation mark are marked as "splicing". Finally, the result is returned to the front end.

[0085] If the recognition result does not contain an end character, it is necessary to locate the position of the word before the end character, calculate the end time of that word, retain the audio from the end time to the end, discard the recognition result after the end character, mark the recognition result as "concatenation", and finally return the result to the front end;

[0086] Before feeding back to the front end, the recognition results are summarized. If the previous recognition result was "can be replaced", the current recognition result is used to replace the previous recognition result and fed back to the front end. If the previous recognition result was "to be spliced", the current recognition result is spliced ​​to the end of the previous recognition result and fed back to the front end.

[0087] In this embodiment, historical audio is spliced ​​with the latest audio to improve recognition accuracy. In addition, the length of historical audio is appropriately trimmed to ensure that the recognition result within a single terminator is coherent. Furthermore, considering the amount of data processed by the recognition model, the audio stored locally is appropriately trimmed to ensure the response speed of the recognition model.

[0088] In the description of this invention, it should be understood that the terms "coaxial," "bottom," "one end," "top," "middle," "other end," "upper," "side," "top," "inner," "outer," "front," "center," "both ends," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0089] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time English speech recognition method based on a large model, characterized in that, A large seq2seq architecture model is used as the recognition model. The seq2seq architecture includes an encoder and a decoder. The process of using the recognition model for speech recognition specifically includes the following steps:

100. The front end sends audio to the speech recognition model at fixed intervals T, and the length of each audio segment is T; 101. Concatenate the currently received audio segment of length T with historical audio segments and input it into the speech recognition model; 102. If no end-of-speech character appears in the speech recognition result, proceed to step 103; otherwise, proceed to step 105.

103. Determine whether the length of the audio segment of length T after splicing with the historical audio segment is greater than the threshold of the historical audio segment. If it is greater, obtain the start time of the word based on the timestamp of the last word in the recognition result, retain the audio segment from the start time of the word to the end of the spliced ​​audio as the historical audio segment, and mark the recognition result as "splicing", and proceed to step 107.

104. Otherwise, directly concatenate the audio segment of length T with the historical audio segment as the historical audio segment, and mark the current recognition result as "can be replaced"; 105. Calculate the timestamp of the word preceding the end-of-word character, obtain the end time of the word based on the timestamp, and retain the audio segment from the end time of the word to the end of the concatenated audio as a historical audio segment.

106. Discard the recognition results after the end symbol appears, and mark the current recognition result as "concatenation"; 107. When the front end receives the current recognition result and its tag, if the previous tag was "can be replaced", then the current recognition result is used to replace the previous recognition result; if the previous tag was "concatenate", then the current recognition result is concatenated to the end of the previous recognition result.

108. Receive the audio segment of the next time step and return to step 101; When identifying word timestamps, the timestamps of each word are calculated sequentially backward from the last word predicted by the decoder. This involves the following steps: Calculate the timestamp of the last word predicted by the decoder, starting from the audio features of the last time step of the input encoder, and traverse each time step of the encoder from back to front. Each time a new encoder time step is reached, the normalized matching score between the current word and the audio features of the current encoder time step must be calculated by scaling dot product attention. If the matching score between the current word and the audio feature at the current time step is less than 0.2, or the matching score between the previous word and the audio feature at the current time step is greater than the matching score between the current word and the audio feature of the segment, then stop calculating the matching score between the current word and other audio features, and record the audio feature at time step t of the encoder. The audio features that meet the requirements are merged together to form an audio segment. The start time and end time of the audio segment are calculated to obtain the timestamp of the current word. Using time step t as the starting time step for the next word, continue calculating the time steps that meet the requirements. Calculate the timestamp based on the time steps that meet the requirements, and repeat the above steps to calculate the timestamps for all words.

2. The real-time English speech recognition method based on a large model according to claim 1, characterized in that, The acquisition of the speech recognition model includes: using OpenAI's whisper-large model as a pre-trained model, and using our own speech data as training data to fine-tune the pre-trained model.

3. The real-time English speech recognition method based on a large model according to claim 2, characterized in that, The ctranslation2 framework is used to transform the format of the fine-tuned speech recognition model.

4. The real-time English speech recognition method based on a large model according to claim 1, characterized in that, If there is silence of more than 1 second in the audio, the starting position is the position where the silence ends. The audio before the starting position is discarded, and the part from the starting position to the end of the audio is retained for the next audio splicing. At the same time, only the recognition result corresponding to the discarded audio is retained, and a period is added to the end of the word sequence of the recognition result. Then, the recognition result is marked as ready for splicing, and the result is returned to the front end.

5. A real-time English speech recognition system based on a large model, characterized in that, It includes a voice acquisition module, a voice splicing module, a voice recognition module, a terminator detection module, a timestamp calculation module, a detection result push module, and a historical voice construction module, among which: The voice acquisition module is used to send audio to the voice recognition model at fixed intervals T, with each audio segment having a length of T. The voice splicing module is used to splice the currently received audio segment of length T with historical audio segments; The speech recognition module is used to perform speech recognition based on the spliced ​​speech obtained by the speech splicing module. This model adopts a large model with a seq2seq architecture, which includes a decoder and an encoder. The encoder extracts audio features from the input audio, and the decoder outputs the prediction result based on the audio features. The terminator detection module is used to detect whether a terminator appears in the recognition result obtained by the speech recognition module. The timestamp calculation module generates a timestamp for each word output by the speech recognition module. This timestamp includes the start and end times of the word within the speech segment input to the speech recognition module. When generating a timestamp for each word, the timestamp calculation module calculates the timestamp for each word sequentially backwards from the last word predicted by the decoder. Specifically, this includes the following steps: Calculate the timestamp of the last word predicted by the decoder, starting from the audio features of the last time step of the input encoder, and traverse each time step of the encoder from back to front. Each time a new encoder time step is reached, the normalized matching score between the current word and the audio features of the current encoder time step must be calculated by scaling dot product attention. If the matching score between the current word and the audio feature at the current time step is less than 0.2, or the matching score between the previous word and the audio feature at the current time step is greater than the matching score between the current word and the audio feature of the segment, then stop calculating the matching score between the current word and other audio features, and record the audio feature at time step t of the encoder. The audio features that meet the requirements are merged together to form an audio segment. The start time and end time of the audio segment are calculated to obtain the timestamp of the current word. Use time step t as the starting time step for the next word and continue to calculate the time steps that meet the requirements. Calculate the timestamp based on the time steps that meet the requirements, and repeat the above steps to calculate the timestamps for all words. The detection result push module pushes the current recognition result based on the tag of the previous recognition result when the front end receives the current recognition result. That is, if the previous tag is "can be replaced", the current recognition result is used to replace the previous recognition result; if the previous tag is "to be concatenated", the current recognition result is concatenated to the end of the previous recognition result. In the historical audio construction module, if no end-of-speech character appears and the audio segment of the input speech model is not greater than the threshold of the set historical audio segment length, the audio segment of the input speech model is directly used as the historical audio segment. Otherwise, the start time of the last word in the recognition result is obtained based on the timestamp of the last word, and the audio segment from the start time of the last word to the end of the concatenated audio is retained as the historical audio segment. If an end-of-speech character appears, the timestamp of the word preceding the end-of-speech character is calculated, the end time of the last word is obtained based on the timestamp, and the audio segment from the end time of the last word to the end of the concatenated audio is retained as the historical audio segment.

6. A real-time English speech recognition system based on a large model according to claim 5, characterized in that, The speech recognition module uses OpenAI's whisper-large model as a pre-trained model and uses the original speech data as training data to fine-tune the pre-trained model.

7. A computer device, characterized in that, The computer device includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a real-time English speech recognition method based on a large model as described in claim 1.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the real-time English speech recognition method based on a large model as described in claim 1.

Images