Speech recognition method, device, system, electronic device and readable storage medium

Abstract

The application provides a speech recognition method, device, system, electronic equipment and readable storage medium, to obtain multiple audio segments of an audio stream in a preset time interval, recognize each audio segment to obtain multiple recognition results, splice to obtain multiple candidate recognition result sequences corresponding to the audio stream, and select a target recognition result sequence from the multiple candidate recognition result sequences. When correcting the target recognition result sequence, multiple candidate recognition result sequences are combined with the audio stream to be decoded in a first order and a second order. The first order is from left to right, so that each audio segment can be decoded and calculated in combination with the information before the audio segment. The second order is from right to left, so that each audio segment can be decoded and calculated in combination with the information after the audio segment, so that the complete context information of the audio stream can be used to correct the result, and the accuracy of the speech recognition model is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech recognition, and more particularly to a speech recognition method, apparatus, system, electronic device and readable storage medium. Background Technology

[0002] Streaming ASR, or online ASR, is a speech recognition method that provides real-time text results as the input speech data increases.

[0003] Most mainstream deep learning streaming speech recognition models are based on a self-attention encoder and decoder structure combined with a streaming prediction module, such as a Connectionist Temporal Classification (CTC) module. In real-time recognition, the streaming recognition result for each audio segment is first output in real-time by the CTC. After a sentence is completely recognized, the encoder output for the entire sentence and multiple candidate streaming results from the CTC are input into the decoder to obtain the corrected final non-streaming recognition result.

[0004] When correcting the streaming recognition result of a sentence, the above method can only use the audio information preceding each audio segment, so that the final non-streaming recognition result after correction is only the result of referring to the information above, which limits the accuracy of speech recognition. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide a speech recognition method, apparatus, system, electronic device and readable storage medium that overcomes or at least partially solves the above problems.

[0006] In a first aspect, embodiments of this application disclose a speech recognition method, the method comprising:

[0007] Multiple audio segments of an audio stream are acquired sequentially at preset time intervals, and audio features corresponding to each audio segment are generated.

[0008] According to the acquisition time order of the audio segments, multiple recognition results corresponding to each of the multiple audio features are obtained; based on the multiple recognition results, multiple candidate recognition result sequences corresponding to the audio stream are spliced ​​together, and a target recognition result sequence is selected from the multiple candidate recognition result sequences;

[0009] The candidate recognition result sequences and the audio stream are decoded in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments;

[0010] The candidate recognition result sequences and the audio stream are decoded in a second order to obtain a second recognition result, wherein the second order is the reverse of the first order.

[0011] The target identification result sequence is corrected based on the first and second identification results, and the corrected target identification result sequence is displayed.

[0012] Secondly, embodiments of this application disclose a voice recognition device, the device comprising:

[0013] The acquisition module is used to sequentially acquire multiple audio segments of an audio stream at preset time intervals and generate audio features corresponding to each audio segment;

[0014] The first recognition module is used to acquire multiple recognition results corresponding to each of the multiple audio features in the order of acquisition time of the audio segments; to concatenate the multiple recognition results to obtain multiple candidate recognition result sequences corresponding to the audio stream, and to select a target recognition result sequence from the multiple candidate recognition result sequences;

[0015] The second recognition module is used to decode the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments;

[0016] The third recognition module is used to decode the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result, wherein the second order is the reverse of the first order;

[0017] The correction module is used to correct the target recognition result sequence based on the first recognition result and the second recognition result, and to display the corrected target recognition result sequence.

[0018] Thirdly, this application also discloses a speech recognition system, the system comprising: a data acquisition device, a speech recognition model, and a display device, wherein the speech recognition model comprises: a first decoder, a second decoder, and a third decoder;

[0019] The acquisition device is used to sequentially acquire multiple audio segments of an audio stream at preset time intervals. The speech recognition model generates audio features corresponding to each audio segment based on the audio segments. The third decoder acquires multiple recognition results corresponding to each audio feature in the order of acquisition time of the audio segments. The speech recognition model concatenates the multiple recognition results to obtain multiple candidate recognition result sequences corresponding to the audio stream, and selects a target recognition result sequence from the multiple candidate recognition result sequences. The first decoder decodes the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result. The second decoder decodes the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result. After the speech recognition model corrects the target recognition result sequence based on the first and second recognition results, the display device displays the corrected target recognition result sequence.

[0020] Fourthly, embodiments of this application also disclose an electronic device, including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0021] Fifthly, embodiments of this application also disclose a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0022] In this embodiment, when correcting the results of streaming recognition, multiple candidate recognition results obtained during the streaming recognition process are decoded in a first order and a second order, respectively, in conjunction with the encoding of the audio stream. The first order is left-to-right decoding, that is, decoding according to the acquisition order of each audio segment in the audio stream, so that each audio segment can be decoded and calculated in combination with the information before the audio segment. The second order is right-to-left decoding, that is, decoding starts from the last audio segment of the audio stream, so that each audio segment can be decoded and calculated in combination with the information after the audio segment. Through two decoding operations in opposite orders, when correcting the target recognition results, the complete contextual information of the audio stream can be used to improve the accuracy of the speech recognition model, thereby improving the accuracy of the target recognition result correction. Attached Figure Description

[0023] Figure 1 This is a flowchart of a speech recognition method provided in an embodiment of the present invention;

[0024] Figure 2 This is a streaming speech recognition process provided in an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of another speech recognition method provided in an embodiment of the present invention;

[0026] Figure 4 This is a block diagram of a speech recognition device provided in an embodiment of the present invention;

[0027] Figure 5 This is a block diagram of a terminal according to another embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of the terminal structure according to another embodiment of the present invention. Detailed Implementation

[0029] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0030] refer to Figure 1 It illustrates a flowchart of the speech recognition method provided in an embodiment of this application, the method comprising:

[0031] Step 101: Sequentially acquire multiple audio segments of the audio stream at preset time intervals, and generate audio features corresponding to each audio segment.

[0032] In this embodiment of the invention, a streaming ASR model refers to a type of model that can support real-time return of recognition results during the processing of an audio stream. In contrast, a non-streaming model can only return results after processing the entire sentence of audio. Streaming models are better suited for scenarios requiring real-time recognition results, such as live streaming captions, real-time conference recording, voice input, and voice wake-up. Therefore, for streaming ASR applications, the typical approach is to sequentially acquire multiple audio segments from the audio stream at preset time intervals, generate audio features corresponding to each audio segment sequentially, and then recognize them sequentially.

[0033] Furthermore, the process of generating audio features from audio segments can be based on Fbank (Filter Bank, a front-end processing method that processes audio in a manner similar to the human ear). Specifically, it can include: 1. Pre-emphasis: The acquired audio segment's speech signal is pre-emphasized through a high-pass filter to compensate for the high-frequency components of the speech signal suppressed by the vocal system, and also to highlight the high-frequency formants; 2. Frame segmentation: Frame segmentation refers to dividing the audio segment file according to a fixed time length. Each sample after segmentation is called a frame. The signal sampling frequency used in speech recognition can be 8kHz or 16kHz. Taking 8kHz as an example, if the frame length is 256 sampling points, the corresponding time length is 256 / 8000×1000=32ms; 3. Windowing: After segmenting the audio into frames, each frame needs to be windowed to increase the continuity between the left and right ends of the frame and reduce spectral leakage; 4. Perform Discrete Fourier Transform on the signal: Since it is usually difficult to see the characteristics of a signal in the time domain, it can be converted into an energy distribution in the frequency domain for observation. Different energy distributions represent different speech characteristics. Therefore, after windowing, a Discrete Fourier Transform is needed to obtain the energy distribution in the spectrum; 5. Apply a Mel filter: The ordinary frequency scale can be converted into a Mel frequency scale, making it more consistent with human auditory perception, and the Mel spectrum is obtained as audio features; 6. Because different Mel filters have intersections, a Discrete Cosine Transform can be performed based on the previous step to obtain the final audio features.

[0034] Step 102: According to the acquisition time of the audio segment, acquire multiple recognition results corresponding to each of the multiple audio features; based on the multiple recognition results, concatenate to obtain multiple candidate recognition result sequences corresponding to the audio stream, and select the target recognition result sequence from the multiple candidate recognition result sequences.

[0035] In this embodiment of the invention, after obtaining audio features from audio segments, the audio features can be input into a speech recognition model to encode and decode the audio features of each audio segment in order to obtain the recognition result corresponding to each audio segment.

[0036] Furthermore, the speech recognition model can be a Transformer model based on a self-attention mechanism. The Transformer model structure includes an encoder and a decoder. The Transformer Encoder (or Transformer Encoder for short) is used to recognize speech information and generate an encoded sequence, while the Transformer Decoder (or Transformer Decoder for short) is used to recognize the encoded sequence and generate the recognition result. For streaming recognition scenarios, the encoder can be shared by the streaming decoder and the Transformer model's decoder. Specifically, the streaming decoder can be a CTC decoder, used to recognize audio features and output the recognition result of the audio features corresponding to each audio segment in real time.

[0037] For example, if audio features are encoded to generate an encoded sequence, such as the encoded sequence {x1,x2,....,xt}, whenever the encoder generates an output encoded feature, such as x4, the decoder of the CTC model can independently recognize x4, obtaining multiple recognition results. Then, it combines the multiple recognition results corresponding to x1, x2, and x3 respectively to obtain multiple recognition result sequences. After each audio segment is recognized, multiple candidate recognition result sequences can be obtained according to the above method. During real-time display of streaming recognition, one of the multiple candidate recognition result sequences can be selected as the target recognition result sequence.

[0038] Step 103: Decode the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments.

[0039] In this embodiment of the invention, the speech recognition model may include a first decoder. The first decoder may be a commonly used decoder that decodes in a first order, where the first order may be the acquisition time order of the audio segments, i.e., the first decoder may be a decoder that decodes from left to right. After the audio stream is acquired, multiple candidate result sequences corresponding to the audio stream and the complete encoding corresponding to the audio stream can be input together into the first decoder to obtain a non-streaming recognition result sequence after correcting the target result sequence.

[0040] Furthermore, during decoding, all audio feature encodings are only decoded and calculated with themselves and the encodings preceding them. The first encoded feature in the audio stream is decoded and calculated independently. That is, the first decoder can be a decoder that incorporates the preceding information during decoding. Through the decoding calculation of each candidate recognition result sequence by the first decoder, the first recognition result corresponding to each candidate recognition result can be obtained. Specifically, each candidate recognition result sequence undergoes the same decoding calculation process within the first decoder. The first recognition result obtained after the decoding calculation by the first decoder can be the result of re-determining the weight information for each recognition result in each candidate recognition result sequence.

[0041] Step 104: Decode the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result, wherein the second order is the reverse of the first order.

[0042] In this embodiment of the invention, the speech recognition model may include a second decoder. The second decoder may be a decoder whose decoding order is the reverse of the first order. That is, the second decoder may be a decoder that decodes from right to left, starting from the end of the audio stream and working backwards. After the audio stream is acquired, multiple candidate result sequences corresponding to the audio stream and the complete encoding corresponding to the audio stream may be input into the second decoder to obtain a non-streaming recognition result sequence after the target result sequence is corrected.

[0043] Furthermore, during decoding, all audio feature codes are only decoded and calculated with themselves and the codes following them. The last encoded feature in the audio stream is decoded and calculated independently. That is, the second decoder can be a decoder that incorporates the following information during decoding. Through the decoding calculation of each candidate recognition result sequence by the second decoder, the second recognition result corresponding to each candidate recognition result can be obtained. Specifically, each candidate recognition result sequence undergoes the same decoding calculation process within the second decoder. After the decoding calculation by the second decoder, the obtained second recognition result can be the result of re-determining the weight information for each recognition result in each candidate recognition result sequence.

[0044] Step 105: Correct the target identification result sequence based on the first identification result and the second identification result, and display the corrected target identification result sequence.

[0045] In this embodiment of the invention, after calculation by the first decoder and the second decoder, the candidate recognition result sequence can be obtained after correction using the context information and the background information respectively. This allows the target recognition result sequence to be corrected by combining the results of the first decoder and the second decoder. By making full use of the complete speech information, the recognition effect of the speech recognition model is improved.

[0046] refer to Figure 2 , Figure 2 A streaming speech recognition process is illustrated, wherein: S1, streaming speech is acquired; after extracting audio features from the streaming speech, step S2, the audio features are input into a streaming encoder to convert them into audio encoded features; step S3, the audio encoded features are input into a CTC decoder to decode and obtain multiple recognition results corresponding to audio segments; the multiple recognition results corresponding to each audio segment are concatenated to form multiple candidate recognition result sequences corresponding to the audio stream; step S4, the multiple candidate recognition result sequences obtained in step S3 and the complete speech encoded features corresponding to the audio stream are input into an Attention decoder, which includes a first decoder and a second decoder; the first decoder and the second decoder respectively calculate the decoding information corresponding to each candidate recognition result sequence and determine the matching value between each candidate recognition result sequence and the audio stream based on the decoding information (the matching value can be the attention score of each candidate recognition result sequence); step S5, the corrected target recognition result sequence is determined based on the matching value.

[0047] In summary, in this embodiment of the application, when correcting the results of streaming recognition, multiple candidate recognition results obtained during the streaming recognition process are decoded in a first order and a second order, respectively, in conjunction with the encoding of the audio stream. The first order is decoding from left to right, that is, decoding according to the acquisition order of each audio segment in the audio stream, so that each audio segment can be decoded and calculated in combination with the information before the audio segment. The second order is decoding from right to left, that is, decoding starts from the last audio segment of the audio stream, so that each audio segment can be decoded and calculated in combination with the information after the audio segment. Through two decoding operations in reverse order, when correcting the target recognition results, the complete contextual information of the audio stream can be used to improve the accuracy of the speech recognition model, thereby improving the accuracy of the target recognition result correction.

[0048] refer to Figure 2 It illustrates a flowchart of another speech recognition method provided in an embodiment of this application, the method comprising:

[0049] Step 201: Sequentially acquire multiple audio segments of the audio stream at preset time intervals, and generate audio features corresponding to each audio segment.

[0050] This step can be referred to in step 101, and will not be repeated here.

[0051] Step 202: According to the acquisition time of the audio segment, acquire multiple recognition results corresponding to each of the multiple audio features; based on the multiple recognition results, concatenate to obtain multiple candidate recognition result sequences corresponding to the audio stream, and select the target recognition result sequence from the multiple candidate recognition result sequences.

[0052] This step can be referred to in step 102, and will not be repeated here.

[0053] Optionally, step 202 specifically includes:

[0054] Sub-step 2021: Encode the multiple audio features to generate multiple speech coding features.

[0055] In this embodiment of the invention, the speech recognition model may include a streaming encoder and a third decoder. The streaming encoder is used to encode the speech features of an audio segment to obtain the encoded speech features. Specifically, the encoder is used to receive a sequence of variable length as input and convert it into an encoded state with a fixed shape.

[0056] Sub-step 2022 involves decoding the multiple speech coding features to generate multiple recognition results corresponding to each audio segment.

[0057] In this embodiment of the invention, the third decoder can be a streaming decoder, such as a CTC decoder. CTC decoding is based on a pre-trained CTC model to decode the input sequence and obtain the recognition result. For example, it is used to decode the encoded result output by a streaming encoder to obtain the streaming recognition result for each audio segment. The CTC decoder can use a prefix beam search algorithm to find the path with the highest probability for each speech coding feature, merge consecutive characters and remove whitespace tags, and finally fuse multiple paths with consistent results, retaining the result with the highest probability. The result with the highest probability can be used as the streaming output result at this time. This embodiment of the invention does not limit the CTC decoding method.

[0058] Optionally, step 202 specifically includes:

[0059] Sub-step 2023: Select any one of the multiple recognition results corresponding to each audio segment as an intermediate recognition result, and concatenate the intermediate recognition results corresponding to each audio segment as a candidate recognition result sequence.

[0060] In an embodiment of the present invention, the CTC model can obtain multiple recognition results for each audio segment, and each audio segment performs an independent decoding operation. For example, if the encoded sequence is {x1, x2, x3, x4}, then the recognition results corresponding to x1 can be a and b, the recognition results corresponding to x2 can be c and d, the recognition results corresponding to x3 can be e and f, and the recognition results corresponding to x4 can be g and h. Then, for x1, the multiple recognition results include a and b, and any one of a or b can be used as the intermediate recognition result corresponding to x1. The same applies to x2, x3, and x4. Then, the candidate recognition result sequences corresponding to the encoded sequence can include sequences such as: a, c, e, g; b, d, f, h; a, d, e, h, etc.

[0061] Sub-step 2024: Select the candidate recognition result sequence with the largest matching value with the audio stream from all the candidate recognition result sequences as the target recognition result sequence.

[0062] In an embodiment of the present invention, the target recognition result sequence can be the result with the highest matching value between the recognition result and the audio segment among the recognition results corresponding to each audio segment. The matching value can be the attention score corresponding to the recognition result.

[0063] Specifically, for example. The process of streaming speech recognition is as follows: When processing audio stream data, the speech recognition results obtained by streaming speech recognition of each audio segment in the audio stream are returned in real time. For example, when the user inputs "What's the weather like today" by voice in a certain interface, the streaming speech recognition can be: Call the streaming recognition model (CTC) to perform streaming speech recognition on the audio segment including "jin" in the voice input of "What's the weather like today", and display the speech recognition result of "jin" in the audio segment as the character "jin". Then call the streaming recognition model to perform streaming speech recognition on "tian" included in the audio segment of "What's the weather like today", and display the speech recognition result of "tian" included in the audio segment as the character level "tian"; and so on. Finally, the streaming recognition result of the audio stream is displayed in this interface, and the streaming recognition result of the audio stream is the character level sequence "What's the weather like today". Among them, each recognition result in the streaming recognition result can be the result with the highest matching value of the audio segment in this recognition process.

[0064] Optionally, sub-step 2024 specifically includes:

[0065] Sub-step 20241: Obtain the matching values of the multiple recognition results with the corresponding audio segments respectively.

[0066] In this embodiment of the invention, the matching value between each streaming recognition result sequence and the corresponding audio segment can be reflected by the attention score corresponding to the recognition result. The higher the attention score, the more similar the recognition result is to the audio segment. By obtaining the matching value of the recognition result, the final real-time display recognition result can be determined.

[0067] Sub-step 20242: Select the recognition result with the highest matching value as the real-time recognition result of the audio segment.

[0068] Sub-step 20243: The candidate recognition result sequence obtained by splicing the real-time recognition results corresponding to each audio segment is used as the target recognition result sequence.

[0069] In this embodiment of the invention, for each audio segment, there are multiple streaming recognition results, each with a matching value. The higher the matching value, the closer the recognition result is to the true meaning of the audio segment. Therefore, for multiple streaming recognition results corresponding to an audio segment, the recognition result with the highest matching value can be determined as the real-time recognition result for that audio segment. Each generated audio encoding feature generates a real-time recognition result, and outputting this real-time recognition result realizes streaming speech recognition, which can be presented to the user instantly. At this point, the target recognition result sequence can be a sequence formed by concatenating the real-time recognition results of each audio segment.

[0070] Step 203: Decode the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments.

[0071] This step can be referred to in step 103, and will not be repeated here.

[0072] Optionally, step 203 specifically includes:

[0073] Sub-step 20231: Traverse each recognition result in each candidate recognition result sequence in the first order, and perform decoding calculation based on the traversed recognition result, other recognition results before the traversed recognition result, and the encoding result of the audio stream to obtain the first recognition result corresponding to each candidate recognition result sequence.

[0074] In this embodiment of the invention, after the streaming recognition corresponding to the audio stream is completed, multiple candidate recognition result sequences obtained from the streaming recognition and the encoding result corresponding to the audio stream can be input into a first decoder. The first decoder can re-evaluate the attention score, i.e., the matching value, of each candidate streaming recognition result. The first decoder calculates the decoding information corresponding to each candidate recognition result sequence according to a first order, i.e., the encoding of each recognition result only refers to the encoding of the preceding one, using the complete speech encoding of the audio stream and multiple candidate recognition result sequences. Based on the decoding information, the matching value of each candidate recognition result sequence is calculated, which may specifically include adjusting the weight value of each recognition result in the candidate recognition result sequence to obtain the adjusted matching value.

[0075] Step 204: Decode the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result, wherein the second order is the reverse of the first order.

[0076] This step can be referred to in step 104, and will not be repeated here.

[0077] Optionally, step 204 specifically includes

[0078] Sub-step 20241: Traverse each recognition result in each candidate recognition result sequence in the second order, and perform decoding calculation based on the traversed recognition result, other recognition results after the traversed recognition result, and the encoding result of the audio stream to obtain the second recognition result corresponding to each candidate recognition result sequence.

[0079] In this embodiment of the invention, after the streaming recognition corresponding to the audio stream is completed, multiple candidate recognition result sequences obtained from the streaming recognition and the encoding result corresponding to the audio stream can be input into a second decoder. The second decoder can re-evaluate the attention score, i.e., the matching value, of each candidate streaming recognition result. The second decoder calculates the decoding information corresponding to each candidate recognition result sequence according to a second order, i.e., the encoding of each recognition result only refers to the encoding of the later one, based on the complete speech encoding of the audio stream and multiple candidate recognition result sequences. Based on the decoding information, the matching value of each candidate recognition result sequence is calculated, which may specifically include adjusting the weight value of each recognition result in the candidate recognition result sequence to obtain the adjusted matching value.

[0080] Step 205: Determine the target weight value for each candidate identification result sequence based on each candidate identification result sequence and the first and second identification results of each candidate identification result sequence.

[0081] In this embodiment of the invention, for each candidate recognition result sequence, the weight value corresponding to the candidate recognition result sequence can be re-evaluated by referring to the first recognition result obtained after the candidate recognition result sequence and the encoding features of the audio stream are decoded in a first order, and the second recognition result obtained after the second order is decoded, thereby realizing the correction of the target recognition result sequence.

[0082] Optionally, step 205 specifically includes:

[0083] Sub-step 2051: For each candidate recognition result sequence, based on the attention mechanism, obtain the weight information corresponding to the candidate recognition result sequence, the first recognition result and the second recognition result, and their matching values ​​with the audio stream.

[0084] In this embodiment of the invention, based on the attention mechanism, the weight information corresponding to the candidate recognition result sequence, the first recognition result and the second recognition result and their corresponding matching values ​​can be obtained. Specifically, the weight information can be determined through experiments, and the matching values ​​corresponding to the candidate recognition result sequence, the first recognition result and the second recognition result can be obtained based on the attention mechanism. The speech recognition model is a model based on the self-attention mechanism, which generates the attention score, i.e. the matching value, for each candidate recognition result sequence.

[0085] Furthermore, the Transformer model does not require recognition of the encoded sequence of the audio stream, and the CTC model recognizes one speech coding feature after generating it. When the last speech coding feature is generated, the CTC model only needs to recognize the last speech coding feature to generate multiple subsequent recognition result sequences. Therefore, the time taken by the CTC model to generate multiple candidate recognition result sequences is much less than the time taken by the Transformer model to recognize the speech coding features of the audio stream and generate the final recognition result. Moreover, the Transformer model's time to adjust the weights of multiple candidate recognition result sequences and correct the target recognition result sequence is very short. Therefore, the Transformer model based on the self-attention mechanism can effectively improve the performance and accuracy of speech recognition models.

[0086] Sub-step 2052: Based on the weight information, perform a weighted summation on the matching values ​​corresponding to the candidate streaming recognition result sequence, the first recognition result, and the second recognition result, respectively, to determine the target weight value for each candidate recognition result sequence.

[0087] In this embodiment of the invention, after obtaining the first recognition result and the second recognition result with adjusted weight values ​​by the calculation of the first decoder and the second decoder for each candidate recognition result sequence, the streaming matching value of each candidate recognition result sequence and the attention matching value after weight adjustment (the matching values ​​corresponding to the first recognition result and the second recognition result respectively) can be weighted and summed to obtain the target weight value of each candidate recognition result sequence. Then, the streaming recognition result with the highest target weight value is taken as the final recognition result of the audio stream.

[0088] Specifically, the target weight value corresponding to each candidate recognition result sequence can be determined using the following first expression:

[0089] First expression: S = a1 × S1 + a2 × (b1 × S2 + (1 - b1) × S3), where a1, a2, and b1 are preset weight information, S1 is the matching value of the candidate recognition result, S2 is the matching value of the first recognition result, S3 is the matching value of the second recognition result, and S is the target weight value of the candidate recognition result.

[0090] In this embodiment of the invention, the first expression can be used as an expression to determine the target weight value of each candidate recognition result sequence, where S1 is the matching value of the candidate recognition result, which can be obtained in the streaming recognition stage; S2 is the matching value of the first recognition result; and S3 is the matching value of the second recognition result. S2 and S3 can be obtained after the first decoder and the second decoder re-evaluate the weights of the candidate recognition result sequence. Furthermore, the weight value a1 corresponding to S1 can be set to 0, meaning the target weight value of the candidate recognition result sequence can be the matching value without referencing the streaming recognition stage. When correcting the target recognition sequence, by setting two decoders, decoding is performed in conjunction with the context of the speech information, thereby improving the accuracy of the correction of the target recognition result sequence.

[0091] Step 206: Select the candidate recognition result sequence with the largest target weight value as the corrected target recognition result sequence.

[0092] In this embodiment of the invention, the candidate recognition result sequence with the largest weight value is the candidate recognition result sequence with the largest target weight value obtained after integrating the context of speech information. This candidate recognition result sequence can be used as the corrected target recognition result sequence.

[0093] Optionally, the method further includes:

[0094] Step 208: Obtain the training dataset, which includes vocabulary and the corresponding pronunciation information of the vocabulary.

[0095] In the embodiments of the present invention, speech recognition generally includes end-to-end speech recognition and two-stage speech recognition split into an acoustic model (audio-pinyin) and a language model (pinyin-Chinese characters). For the training of the latter model, an efficient and correct Chinese character-pinyin dictionary is required to generate corresponding training data labels. The existing training sets label many words too formally, which does not match the actual spoken situation, especially for nouns, verbs, and adjectives, lacking the annotation of light tones, resulting in poor training of the speech recognition model. Therefore, when constructing the training data set in this application, a new training data set is obtained by correcting the defects of the existing training data set.

[0096] Optionally, step 208 specifically includes:

[0097] Sub-step 2081, obtaining the recognition error information of the words in the historical recognition result and the high-frequency words that appear in the historical recognition result.

[0098] In the embodiments of the present invention, the information of the recognition errors of the speech recognition model can be statistically obtained to get the recognition error information of the words in the recognition result. At the same time, information such as high-frequency words and common words that appear in the historical recognition result can also be obtained. The training data set can be expanded through these information.

[0099] Sub-step 2082, correcting the pronunciation information corresponding to the word according to the recognition error information of the word.

[0100] In the embodiments of the present invention, the pronunciation information corresponding to the word can be distinguished by numbers 1-5, that is, 1-4 represent the corresponding tones, and 5 represents the light tone. For example, the second character of words like sister (mei4 mei4), grandma (nai3 nai3), brother (ge1 ge1), master (shi1 fu4) is the light tone in daily spoken language, and the more appropriate pinyin should be sister (mei4mei5), grandma (nai3 nai5), brother (ge1 ge5), master (shi1 fu5). In addition to personal terms, some common verbs, nouns, and adjectives can also be marked as light tones, such as look (kan4 kan5), time (shi2 hou5), position (wei4 zhi5), temper (pi2 qi5), etc. By correcting the incorrect pronunciation information of the words, the training effect of the speech model can be improved.

[0101] Sub-step 2083, using the corrected word, the pronunciation information corresponding to the word, and the high-frequency words as the training data set.

[0102] In the embodiments of the present invention, the corrected words, the corresponding pronunciation information, common words, high-frequency words, etc. can be updated to the training data set to form a training data set with higher accuracy and improve the recognition accuracy of the speech model.

[0103] Step 209: Train the speech recognition model using the training dataset.

[0104] In this embodiment of the invention, the training dataset may include audio data and text annotations corresponding to the audio data. The speech recognition model is trained using the training dataset so that the speech recognition model can achieve streaming speech recognition and correction.

[0105] Optionally, the method further includes:

[0106] Step 210: Display each of the real-time recognition results, and after the last audio segment of the audio stream is recognized, display the target recognition result sequence corresponding to the audio stream.

[0107] In this embodiment of the invention, in order to realize speech recognition display in a streaming recognition scenario, the CTC module can recognize each audio segment in real time and determine the real-time recognition result from multiple recognition results corresponding to each audio segment for display. The real-time recognition result can be the one with the highest matching value among multiple recognition results corresponding to the audio segment.

[0108] Step 211: After the target recognition result sequence is corrected, the corrected target recognition result sequence is displayed.

[0109] In this embodiment of the invention, in order to prevent users from waiting during the real-time recognition process, the results of streaming recognition can be shown to the user first during the real-time recognition process. After the audio stream is acquired, the real-time displayed target recognition result sequence can be corrected, and after the correction is completed, the corrected non-streaming recognition result can be shown to the user to improve the accuracy of streaming recognition.

[0110] In summary, in this embodiment of the application, when correcting the results of streaming recognition, multiple candidate recognition results obtained during the streaming recognition process are decoded in a first order and a second order, respectively, in conjunction with the encoding of the audio stream. The first order is decoding from left to right, that is, decoding according to the acquisition order of each audio segment in the audio stream, so that each audio segment can be decoded and calculated in combination with the information before the audio segment. The second order is decoding from right to left, that is, decoding starts from the last audio segment of the audio stream, so that each audio segment can be decoded and calculated in combination with the information after the audio segment. Through two decoding operations in reverse order, when correcting the target recognition results, the complete contextual information of the audio stream can be used to improve the accuracy of the speech recognition model, thereby improving the accuracy of the target recognition result correction.

[0111] refer to Figure 4It illustrates a speech recognition device 30 provided in an embodiment of this application, the device comprising:

[0112] The acquisition module 301 is used to sequentially acquire multiple audio segments of the audio stream at preset time intervals and generate audio features corresponding to each audio segment;

[0113] The first recognition module 302 is used to acquire multiple recognition results corresponding to each of the multiple audio features in the order of acquisition time of the audio segment; to concatenate the multiple recognition results to obtain multiple candidate recognition result sequences corresponding to the audio stream, and to select a target recognition result sequence from the multiple candidate recognition result sequences;

[0114] The second recognition module 303 is used to decode the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments;

[0115] The third recognition module 304 is used to decode the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result, wherein the second order is the opposite of the first order;

[0116] The correction module 305 is used to correct the target recognition result sequence based on the first recognition result and the second recognition result, and to display the corrected target recognition result sequence.

[0117] Optionally, the first identification module includes:

[0118] The encoding submodule is used to encode multiple audio features to generate multiple speech encoding features;

[0119] The streaming decoding submodule is used to decode multiple speech coding features and generate multiple recognition results corresponding to each audio segment.

[0120] Optionally, the first identification module includes:

[0121] The splicing submodule is used to select any one of the multiple recognition results corresponding to each audio segment as an intermediate recognition result, and splice the intermediate recognition results corresponding to each audio segment to obtain a candidate recognition result sequence.

[0122] The determination submodule is used to select the candidate recognition result sequence with the largest matching value with the audio stream from all the candidate recognition result sequences as the target recognition result sequence.

[0123] Optionally, the determining submodule includes:

[0124] The acquisition submodule is used to acquire the matching values ​​of the multiple streaming recognition results with their corresponding audio segments;

[0125] The selection submodule is used to select the recognition result with the highest matching value as the real-time recognition result of the audio segment;

[0126] The target sequence determination submodule is used to take the candidate recognition result sequence obtained by splicing the real-time recognition results corresponding to each audio segment as the target recognition result sequence.

[0127] Optionally, the second identification module includes:

[0128] The first decoding submodule is used to traverse each recognition result in each candidate recognition result sequence in a first order, and perform decoding calculations based on the traversed recognition result, other recognition results before the traversed recognition result, and the encoding result of the audio stream to obtain the first recognition result corresponding to each candidate recognition result sequence.

[0129] The third identification module includes:

[0130] The second decoding submodule is used to traverse each recognition result in each candidate recognition result sequence in a second order, and perform decoding calculations based on the traversed recognition results, other recognition results after the traversed recognition results, and the encoding results of the audio stream to obtain the second recognition result corresponding to each candidate recognition result sequence.

[0131] Optionally, the correction module includes:

[0132] The weight determination submodule is used to determine the target weight value of each candidate recognition result sequence based on each candidate recognition result sequence and the first and second recognition results of each candidate recognition result sequence.

[0133] The correction submodule is used to select the candidate recognition result sequence with the largest target weight value as the corrected target recognition result sequence.

[0134] Optionally, the weight determination submodule includes

[0135] The weight acquisition submodule is used to acquire, based on an attention mechanism, the weight information corresponding to each candidate recognition result sequence, the first recognition result and the second recognition result, and their matching values ​​with the audio stream for each candidate recognition result sequence.

[0136] The calculation submodule is used to perform a weighted summation of the matching values ​​corresponding to the candidate streaming recognition result sequence, the first recognition result, and the second recognition result, based on the weight information, to determine the target weight value for each candidate recognition result sequence.

[0137] Optionally, the device further includes:

[0138] The first display module is used to display each of the real-time recognition results, and after the last audio segment of the audio stream is recognized, to display the target recognition result sequence corresponding to the audio stream;

[0139] The second display module is used to display the corrected target recognition result sequence after the target recognition result sequence has been corrected.

[0140] In summary, in this embodiment of the application, when correcting the results of streaming recognition, multiple candidate recognition results obtained during the streaming recognition process are decoded in a first order and a second order, respectively, in conjunction with the encoding of the audio stream. The first order is decoding from left to right, that is, decoding according to the acquisition order of each audio segment in the audio stream, so that each audio segment can be decoded and calculated in combination with the information before the audio segment. The second order is decoding from right to left, that is, decoding starts from the last audio segment of the audio stream, so that each audio segment can be decoded and calculated in combination with the information after the audio segment. Through two decoding operations in reverse order, when correcting the target recognition results, the complete contextual information of the audio stream can be used to improve the accuracy of the speech recognition model, thereby improving the accuracy of the target recognition result correction.

[0141] Figure 5 A block diagram of an electronic device 600 is shown according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0142] Reference Figure 5 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0143] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0144] Memory 604 is used to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0145] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0146] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0147] Audio component 610 is used to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) used to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0148] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0149] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0150] Communication component 616 facilitates wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0151] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a speech recognition method provided in the embodiments of this application.

[0152] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0153] Figure 6A block diagram of an electronic device 700 is shown according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. (Refer to...) Figure 6 The electronic device 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by a memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform a speech recognition method provided in embodiments of this application.

[0154] Electronic device 700 may also include a power supply component 726 configured to perform power management of electronic device 700, a wired or wireless network interface 750 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 758. Electronic device 700 may operate on an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0155] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0156] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A speech recognition method, characterized in that, The method includes: Multiple audio segments of an audio stream are acquired sequentially at preset time intervals, and audio features corresponding to each audio segment are generated. According to the acquisition time order of the audio segments, multiple recognition results corresponding to each of the multiple audio features are obtained; based on the multiple recognition results, multiple candidate recognition result sequences corresponding to the audio stream are spliced ​​together, and a target recognition result sequence is selected from the multiple candidate recognition result sequences; The candidate recognition result sequences and the audio stream are decoded in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments; The multiple candidate recognition result sequences and the audio stream are decoded in a second order to obtain a second recognition result, wherein the second order is the reverse of the first order; The target identification result sequence is corrected based on the first and second identification results, and the corrected target identification result sequence is displayed.

2. The method according to claim 1, characterized in that, The step of obtaining multiple recognition results corresponding to each of the multiple audio features includes: Encode the multiple audio features to generate multiple speech coding features; The multiple speech coding features are decoded to generate multiple recognition results corresponding to each audio segment.

3. The method according to claim 2, characterized in that, The step of concatenating multiple recognition results to obtain multiple candidate recognition result sequences corresponding to the audio stream, and selecting the target recognition result sequence from the multiple candidate recognition result sequences, includes: Select any one of the multiple recognition results corresponding to each audio segment as an intermediate recognition result, and concatenate the intermediate recognition results corresponding to each audio segment to form a candidate recognition result sequence; The candidate recognition result sequence with the largest matching value to the audio stream is selected from all the candidate recognition result sequences as the target recognition result sequence.

4. The method according to claim 3, characterized in that, The step of selecting the candidate recognition result sequence with the largest matching value to the audio stream from all the candidate recognition result sequences as the target recognition result sequence includes: Obtain the matching values ​​of the multiple recognition results with their corresponding audio segments; The recognition result with the highest matching value is selected as the real-time recognition result of the audio segment; The candidate recognition result sequence obtained by splicing the real-time recognition results corresponding to each audio segment is used as the target recognition result sequence.

5. The method according to claim 1, characterized in that, The step of decoding the multiple candidate recognition result sequences and the audio stream in a first order to obtain the first recognition result includes: The system iterates through each candidate recognition result sequence in the first order, and performs decoding calculations based on the traversed recognition result, other recognition results before the traversed recognition result, and the encoding result of the audio stream to obtain the first recognition result corresponding to each candidate recognition result sequence. The step of decoding the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result includes: The system iterates through each candidate recognition result sequence in the second order, and performs decoding calculations based on the traversed recognition results, other recognition results after the traversed recognition results, and the encoding results of the audio stream to obtain the second recognition result corresponding to each candidate recognition result sequence.

6. The method according to claim 1, characterized in that, The step of correcting the target identification result sequence based on the first identification result and the second identification result includes: Based on each candidate identification result sequence, and the first and second identification results of each candidate identification result sequence, determine the target weight value of each candidate identification result sequence; The candidate recognition result sequence with the largest target weight value is selected as the corrected target recognition result sequence.

7. The method according to claim 6, characterized in that, The step of determining the target weight value for each candidate identification result sequence based on each candidate identification result sequence and the first and second identification results of each candidate identification result sequence includes: For each candidate recognition result sequence, based on the attention mechanism, the weight information corresponding to the candidate recognition result sequence, the first recognition result and the second recognition result, and their matching values ​​with the audio stream are obtained respectively; Based on the weight information, the matching values ​​corresponding to the candidate streaming recognition result sequence, the first recognition result, and the second recognition result are weighted and summed to determine the target weight value for each candidate recognition result sequence.

8. The method according to claim 4, characterized in that, The method further includes: Each of the real-time recognition results is displayed, and after the last audio segment of the audio stream is recognized, the target recognition result sequence corresponding to the audio stream is displayed; After the target recognition result sequence is corrected, the corrected target recognition result sequence is displayed.

9. A voice recognition device, characterized in that, The device includes: The acquisition module is used to sequentially acquire multiple audio segments of an audio stream at preset time intervals and generate audio features corresponding to each audio segment; The first recognition module is used to acquire multiple recognition results corresponding to each of the multiple audio features in the order of acquisition time of the audio segments; to concatenate the multiple recognition results to obtain multiple candidate recognition result sequences corresponding to the audio stream, and to select a target recognition result sequence from the multiple candidate recognition result sequences; The second recognition module is used to decode the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result, wherein the first order is the acquisition time order of the audio segments; The third recognition module is used to decode the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result, wherein the second order is the reverse of the first order; The correction module is used to correct the target recognition result sequence based on the first recognition result and the second recognition result, and to display the corrected target recognition result sequence.

10. A speech recognition system, characterized in that, The system includes: a data acquisition device, a speech recognition model, and a display device; the speech recognition model includes: a first decoder, a second decoder, and a third decoder. The acquisition device is used to sequentially acquire multiple audio segments of an audio stream at preset time intervals. The speech recognition model generates audio features corresponding to each audio segment based on the audio segments. The third decoder acquires multiple recognition results corresponding to each audio feature in the order of acquisition time of the audio segments. The speech recognition model concatenates the multiple recognition results to obtain multiple candidate recognition result sequences corresponding to the audio stream, and selects a target recognition result sequence from the multiple candidate recognition result sequences. The first decoder decodes the multiple candidate recognition result sequences and the audio stream in a first order to obtain a first recognition result. The second decoder decodes the multiple candidate recognition result sequences and the audio stream in a second order to obtain a second recognition result. After the speech recognition model corrects the target recognition result sequence based on the first and second recognition results, the display device displays the corrected target recognition result sequence.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 8.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 8.

Images