Speech recognition methods, devices, electronic equipment and storage media

By encoding and attention-based encoding of speech feature sequences, and then extracting and decoding the target encoded feature sequences, the problem of real-time streaming recognition in existing technologies is solved, achieving real-time streaming speech recognition and high-accuracy recognition.

CN115910046BActive Publication Date: 2026-11-10IFLYTEK CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211353975.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-31
Publication Date
2026-11-10
Estimated Expiration
2042-10-31

AI Technical Summary

Technical Problem

Existing end-to-end speech recognition technology cannot achieve real-time streaming speech recognition; it requires the user to finish speaking before it can perform recognition.

Method used

By encoding the speech feature sequence, extracting the target encoded feature sequence and performing attention encoding, using the attention encoding network to extract the encoded feature sequence of the current recognition result to be determined from the speech encoded features, and then decoding it, streaming speech recognition is achieved.

Benefits of technology

It achieves real-time streaming speech recognition, adapts to the need for rapid recognition, and ensures the accuracy of recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115910046B_ABST
    Figure CN115910046B_ABST
Patent Text Reader

Abstract

This application proposes a speech recognition method, apparatus, electronic device, and storage medium. The method includes: encoding a speech feature sequence corresponding to the speech to be recognized to obtain a speech coded feature sequence; extracting a target coded feature sequence corresponding to the current recognition result to be determined from all speech coded features in the speech coded feature sequence, and performing attention encoding on the target coded feature sequence to obtain an attention encoding result; and decoding the attention encoding result to obtain a speech recognition result. Using the technical solution of this application, the target coded feature sequence to be recognized can be extracted from all speech coded features, so that attention encoding and decoding of the target coded feature sequence can be performed on the target coded feature sequence without waiting for a sentence to be finished before performing attention encoding and decoding on the entire speech coded feature sequence, thus realizing streaming speech recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, electronic device and storage medium. Background Technology

[0002] The task of speech recognition technology is to enable voice communication between humans and machines. It is the process of converting human speech into text or control signals using intelligent algorithms. Currently, automatic speech recognition technology has been widely applied in various fields, bringing great convenience to human society, such as in-vehicle voice systems and intelligent voice customer service.

[0003] In recent years, with the emergence and development of end-to-end speech recognition technology, end-to-end modeling methods have demonstrated the advantages of joint acoustic and language modeling in speech recognition tasks, surpassing traditional speech recognition methods that model acoustics and language separately. However, existing end-to-end speech recognition technologies typically perform offline speech recognition or only recognize speech after the user has finished speaking, failing to provide real-time speech recognition or achieve streaming speech recognition. Summary of the Invention

[0004] Based on the defects and shortcomings of the prior art, this application proposes a speech recognition method, device, electronic device and storage medium that can realize streaming speech recognition.

[0005] The first aspect of this application provides a speech recognition method, including:

[0006] The speech feature sequence corresponding to the speech to be recognized is encoded to obtain the speech encoded feature sequence;

[0007] Extract the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and perform attention coding on the target coding feature sequence to obtain the attention coding result;

[0008] The attention encoding result is decoded to obtain the speech recognition result.

[0009] Optionally, a target coding feature sequence corresponding to the current recognition result to be determined is extracted from all speech coding features of the speech coding feature sequence, and attention coding is performed on the target coding feature sequence to obtain an attention coding result, including:

[0010] The speech coding feature sequence is input into a pre-trained attention coding network, so that the attention coding network extracts the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and performs attention coding on the target coding feature sequence to obtain the attention coding result;

[0011] The attention encoding network is obtained by training the sample speech encoding feature sequence with attention encoding based on the sample speech encoding feature sequence and the attention label corresponding to the sample speech encoding feature sequence. The attention label corresponding to the sample speech encoding feature sequence includes the sample speech encoding feature sequence interval corresponding to each text character in the recognized text corresponding to the sample speech encoding feature sequence.

[0012] Optionally, the training process of the attention encoding network includes:

[0013] Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, the coding feature sequence to be identified corresponding to each word in the word tags is determined from all sample speech coding features; the word tags carried by the sample speech coding features are the text that matches the speech frame corresponding to the sample speech coding features;

[0014] Based on the word labels and the sequence of coded features to be identified corresponding to each word in the word labels, a sample coded feature matrix carrying attention labels is constructed;

[0015] The sample encoding feature matrix is ​​input into the attention encoding network to obtain the sample attention encoding result;

[0016] The parameters of the attention encoding network are adjusted using the loss function between the recognition result after decoding the sample attention encoding result and the attention label corresponding to the sample encoding feature matrix.

[0017] Optionally, based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, a sequence of coding features to be identified corresponding to each word in the word tags is determined from all sample speech coding features, including:

[0018] Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, determine the sample speech coding feature corresponding to each word in the word tags from all sample speech coding features;

[0019] For each word in the word tag, a first sample speech coding feature sequence is extracted from the sample speech coding feature sequence as the coding feature sequence to be identified for that word. The first sample speech coding feature sequence contains at least some sample speech coding features from the sample speech coding features corresponding to that word.

[0020] Optionally, the first sample speech coding feature sequence is a sample speech coding feature sequence of a set length that includes at least a portion of the sample speech coding features corresponding to the word.

[0021] or,

[0022] The first sample speech coding feature sequence is a sample speech coding feature sequence composed of the sample speech coding features corresponding to the word and the sample speech coding features preceding the sample speech coding features corresponding to the word.

[0023] Optionally, based on the sequence of encoded features to be identified, a sample encoded feature matrix carrying attention labels is constructed, including:

[0024] A masking operation is performed on the sample speech coding features other than the coding feature sequence to be identified in the sample speech coding feature sequence to obtain the sample coding feature matrix corresponding to the coding feature sequence to be identified. The sequence interval of the coding feature sequence to be identified in the sample speech coding feature sequence and all word labels corresponding to the sequence interval are used as the attention labels corresponding to the sample coding feature matrix.

[0025] Optionally, encoding the speech feature sequence corresponding to the speech to be recognized to obtain a speech encoded feature sequence includes:

[0026] The speech feature sequence corresponding to the speech to be recognized is input into a pre-trained speech encoder to obtain the speech coding feature sequence corresponding to the speech to be recognized.

[0027] The speech encoder is obtained by encoding the sample speech feature sequence based on an attention mechanism, using the sample speech feature sequence and the text annotations corresponding to each speech feature in the sample speech feature sequence.

[0028] Optionally, the training process of the speech encoder includes:

[0029] Extract speech features from each speech frame in the pre-collected sample speech data to obtain a sample speech feature sequence, and determine the text annotation sequence corresponding to the sample speech feature sequence based on the text annotation corresponding to the sample speech data.

[0030] The sample speech feature sequence is input into the speech encoder, and the speech encoder is used to encode each speech feature in the sample speech feature sequence based on an attention mechanism to obtain the sample speech encoded feature sequence.

[0031] Identify the word tags corresponding to each speech coding feature in the sample speech coding feature sequence; the word tags corresponding to the speech coding features are the text that matches the speech frame corresponding to the speech coding features;

[0032] The parameters of the speech encoder are adjusted based on the loss function between the word tags corresponding to the sample speech coding feature sequence and the text annotation sequence.

[0033] Optionally, identifying word tags corresponding to each speech coding feature in the sample speech coding feature sequence includes:

[0034] An acoustic model is used to force alignment of all speech coding features in the sample speech coding feature sequence to obtain phoneme labels corresponding to each speech coding feature; the phoneme labels corresponding to the speech coding features are the phonemes that match the speech frames corresponding to the speech coding features.

[0035] The phoneme tags corresponding to the speech coding features are mapped to word tags.

[0036] A second aspect of this application provides a voice recognition device, comprising:

[0037] The speech coding module is used to encode the speech feature sequence corresponding to the speech to be recognized, so as to obtain the speech coding feature sequence.

[0038] The attention coding module is used to extract the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and to perform attention coding on the target coding feature sequence to obtain the attention coding result;

[0039] The decoding module is used to decode the attention encoding result to obtain the speech recognition result.

[0040] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0041] The memory is connected to the processor and is used to store programs;

[0042] The processor is used to implement the above-described speech recognition method by running the program in the memory.

[0043] A fourth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described speech recognition method.

[0044] The speech recognition method proposed in this application encodes the speech feature sequence corresponding to the speech to be recognized, obtaining a speech coded feature sequence; extracts the target coded feature sequence corresponding to the current recognition result to be determined from all speech coded features in the speech coded feature sequence, and performs attention encoding on the target coded feature sequence to obtain an attention encoding result; and decodes the attention encoding result to obtain the speech recognition result. Using the technical solution of this application, the target coded feature sequence to be recognized can be extracted from all speech coded features, so that attention encoding and decoding of the target coded feature sequence can be performed on the target coded feature sequence, eliminating the need to wait until a sentence is finished before performing attention encoding and decoding on the entire speech coded feature sequence, thus achieving streaming speech recognition.

[0045] Furthermore, the speech recognition method based on the attention mechanism proposed in this application can determine the encoding feature sequence corresponding to the current recognition result to be determined from all speech encoding features, and encode the encoding feature sequence based on the attention mechanism. In other words, it can accurately grasp the attention range of each recognition step, so it can not only adapt to the fast recognition requirements of streaming recognition, but also ensure recognition accuracy by means of a reasonable attention range. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a schematic flowchart of a speech recognition method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the processing flow for training the attention encoding network provided in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram illustrating the extraction of a coded feature sequence to be identified, provided in an embodiment of this application.

[0050] Figure 4 This is a schematic diagram illustrating the extraction of another coded feature sequence to be identified, provided in an embodiment of this application.

[0051] Figure 5 This is a schematic diagram of the processing flow for training a speech encoder provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of this application are applicable to speech recognition scenarios, especially end-to-end speech recognition. Using the technical solutions of this application, streaming speech recognition can be achieved.

[0055] Speech recognition refers to the process of converting speech signals into corresponding text using a computer, and it is one of the main ways to achieve human-machine interaction. Traditional speech recognition technology establishes acoustic models and language models separately. The acoustic model is responsible for converting audio sequences into phoneme sequences, such as Chinese Pinyin and English phonetic symbols. The language model is responsible for converting these phoneme sequences into text sequences. However, the acoustic model and the language model need to use a pronunciation dictionary to determine the mapping relationship between the modeling units of the acoustic model and the modeling units of the language model, thereby connecting the acoustic model and the language model to form a search state space for the decoder to perform decoding work.

[0056] With the emergence and development of end-to-end speech recognition technology, end-to-end modeling methods offer the advantage of joint acoustic and language modeling in speech recognition tasks. They can directly convert speech feature sequences into text sequences without relying on a pronunciation dictionary. However, existing end-to-end speech recognition technologies typically perform offline speech recognition or only recognize speech after the user has finished speaking, failing to provide real-time speech recognition or enable streaming speech recognition.

[0057] In view of the shortcomings of the existing technology and the fact that existing end-to-end speech recognition technology cannot perform real-time speech recognition and cannot achieve streaming speech recognition, the inventors of this application have proposed a speech recognition method that can achieve streaming speech recognition after research and experimentation.

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] This application provides a speech recognition method, see [link to relevant documentation] Figure 1 As shown, the method includes:

[0060] S101. Encode the speech feature sequence corresponding to the speech to be recognized to obtain the speech coding feature sequence.

[0061] Specifically, for the speech to be recognized acquired by the speech acquisition device, the speech features of each speech frame in the speech to be recognized need to be extracted first, thereby obtaining the speech features corresponding to each speech frame. The speech feature sequence composed of the speech features corresponding to all speech frames in the speech to be recognized is the speech feature sequence corresponding to the speech to be recognized. The speech to be recognized can be an offline speech segment, in which case the speech feature sequence is the speech features of all speech frames in the offline speech. This embodiment can perform non-streaming recognition on offline speech. The speech to be recognized can also be acquired real-time speech, which may include one or several currently acquired speech frames and a segment of speech acquired previously and continuous with the currently acquired speech frame. This embodiment can perform real-time speech recognition on real-time speech, thereby achieving streaming speech recognition. The speech features extracted from each speech frame in this embodiment can be speech features such as PLP, MFCC, and FilterBank.

[0062] In this embodiment, after extracting the speech feature sequence corresponding to the speech to be recognized, each speech feature in the speech feature sequence is encoded to obtain the speech encoded features corresponding to each speech feature, thereby obtaining a speech encoded feature sequence composed of the encoded speech encoded features. In this embodiment, a pre-trained speech encoder can be used to encode the speech feature sequence, or existing linear predictive vocoders, formant vocoders, or cosine vocoders can be used to train the speech features. The pre-trained speech encoder is obtained by encoding the sample speech feature sequence corresponding to pre-collected sample speech, along with the corresponding text annotation sequence, using an attention-based encoding training mechanism. The text annotation sequence corresponding to the sample speech includes text annotations corresponding to each speech feature in the sample speech feature sequence. For example, the sample speech feature sequence corresponding to the sample speech can be represented as:

[0063] X = {x1, ..., x} j , ..., x J}

[0064] The text annotation sequence corresponding to the sample speech can be represented as:

[0065] Y = {y0, ..., y} i , ..., y I}

[0066] The sample speech encoded feature sequence obtained by encoding each sample speech feature in the sample speech feature sequence using a speech encoder can be represented as:

[0067] H = {h1, ..., h} j , ..., h J}

[0068] Where, x j y represents the sample speech features corresponding to the j-th sample speech frame in the sample speech feature sequence X, where J is the total number of sample speech frames in the sample speech, and y i Let yi represent the i-th character in the text annotation sequence Y, I+1 represent the total number of characters in the text annotation corresponding to the sample speech, and y0 represent the start character of the sentence corresponding to the sample speech. <s>” y I The terminator for the sentence corresponding to the sample speech.< / s> "h j The sample speech coding feature represents the encoded sample speech feature corresponding to the j-th sample speech frame in the sample speech coding feature sequence H.

[0069] In this embodiment, the extracted speech feature sequence of the speech to be recognized can be input into a pre-trained speech encoder, and the speech encoder can be used to encode each speech feature in the speech feature sequence to obtain a speech encoded feature sequence.

[0070] S102. Extract the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and perform attention coding on the target coding feature sequence to obtain the attention coding result.

[0071] Specifically, since the attention mechanism used in the existing technology to decode the encoded feature sequence is a global attention mechanism, it is necessary to perform a weighted summation of all outputs of the speech encoder for a whole sentence. This means that decoding can only begin after the user has finished speaking the whole sentence, making real-time speech recognition impossible, i.e., streaming speech recognition cannot be achieved.

[0072] Therefore, this embodiment first needs to extract the target coding feature sequence corresponding to the current recognition result to be determined from all the speech coding features of the speech coding feature sequence. The target coding feature sequence corresponding to the current recognition result to be determined is a coding feature sequence that can be encoded using an attention mechanism to achieve the recognition result. For example, after a user pronounces "morning" and part of "good," since the pronunciation of "good" is not complete, "morning" is taken as the current recognition result to be determined. The speech coding feature sequence composed of the speech coding features encoded from all the speech frames corresponding to "morning" is the target coding feature sequence.

[0073] In this embodiment, after extracting the target coding feature sequence from the speech coding feature sequence, attention coding is performed on the target coding feature sequence. At this time, a global attention mechanism can be used to perform attention coding on the target coding feature sequence to obtain the attention coding result.

[0074] In this embodiment, the attention encoding network can also be directly trained, then the speech encoding feature sequence is input into the pre-trained attention encoding network. The attention encoding network extracts the target encoding feature sequence corresponding to the current recognition result to be determined from all speech encoding features of the speech encoding feature sequence, and performs attention encoding on the target encoding feature sequence to obtain an attention encoding result. The attention encoding network is obtained by performing attention encoding training on the sample speech encoding feature sequence based on the pre-collected sample speech encoding feature sequence obtained by encoding the sample speech feature sequence of the sample speech and the attention label corresponding to the sample speech encoding feature sequence. The attention label corresponding to the sample speech encoding feature sequence includes a word label corresponding to each text character in the recognized text corresponding to the sample speech encoding feature sequence, and a sample speech encoding feature sequence interval corresponding to each text character. For example, for the sample speech "good morning", the sample speech encoding feature sequence includes: the sample speech encoding feature obtained by encoding the sample speech feature of the sample speech frame corresponding to "good", the sample speech encoding feature obtained by encoding the sample speech feature of the sample speech frame corresponding to "morn", and the sample speech encoding feature obtained by encoding the sample speech feature of the sample speech frame corresponding to "ing". The attention labels corresponding to the sample speech encoding feature sequence include: the sequence interval of the sample speech encoding feature corresponding to "good" in the sample speech encoding feature sequence and the word label corresponding to "good", the sequence interval of the sample speech encoding feature corresponding to "morn" in the sample speech encoding feature sequence and the word label corresponding to "morn", and the sequence interval of the sample speech encoding feature corresponding to "ing" in the sample speech encoding feature sequence and the word label corresponding to "ing".

[0075] S103. Decode the attention encoding result to obtain a speech recognition result.

[0076] Specifically, after the attention encoding result is obtained in this embodiment, the attention encoding result needs to be decoded, so as to obtain the speech recognition result corresponding to the speech to be recognized. The calculation steps of performing attention encoding (i.e., performing soft attention calculation) on the target encoding feature sequence and decoding the attention encoding result are as follows:

[0077] e i,j =a(s i-1 , h j )

[0078]

[0079]

[0080] s i =f(s i-1 , y i-1 , c i)

[0081] y i =g(s i c i )

[0082] Among them, e i,j The score representing the attention mechanism is equivalent to a relevance calculation, calculating e. i,j The function a() is a function that calculates the cosine distance, which reflects the hidden state s of the decoding network at the previous time step (time step i-1). i-1 and speech coding features h in the target coding feature sequence j The correlation between them, α i,j T represents the attention weight. i c represents the position of the last speech coding feature in the target coding feature sequence at the current time (time i). i s represents the attention encoding result. i This represents the hidden state of the decoding network at the current time (time i), and s is calculated. i The function f() is the tanh function, y i To decode the network's output at the current time step (time i), representing the posterior probability distribution of the character at the current time step (time i), calculate y. i The function g() is the softmax function. Here, the posterior probability dimension is the dictionary size, and the posterior probability reflects the probability that each word in the dictionary will be decoded in the current frame.

[0083] After the decoding network outputs the posterior probability distribution data of the characters corresponding to the target encoded feature sequence at the current moment, it is necessary to perform search decoding on the posterior probability distribution data output by the decoding network to obtain the final speech recognition result. In this embodiment, the search decoding methods that can be used include greedy search decoding or beam search decoding.

[0084] In this embodiment, the decoding network can also be pre-trained to ensure the accuracy of the output data. After obtaining the sample attention encoding result by performing soft attention calculation on the sample encoding feature sequence carrying text labels, the sample attention encoding result is input into the decoding network to obtain the sample posterior probability distribution data output by the decoding network. Then, the attention loss is calculated by comparing the sample posterior probability distribution data output by the decoding network with the text labels carried by the sample encoding feature sequence. The gradient is backpropagated based on the calculation result of the attention loss to update the parameters of the decoding network.

[0085] As described above, the speech recognition method proposed in this application encodes the speech feature sequence corresponding to the speech to be recognized, obtaining a speech coded feature sequence; extracts the target coded feature sequence corresponding to the current recognition result to be determined from all speech coded features in the speech coded feature sequence, and performs attention encoding on the target coded feature sequence to obtain an attention encoding result; and decodes the attention encoding result to obtain the speech recognition result. By adopting the technical solution of this embodiment, the target coded feature sequence to be recognized can be extracted from all speech coded features, so that attention encoding and decoding of the target coded feature sequence can be performed on the target coded feature sequence, eliminating the need to wait until a sentence is finished before performing attention encoding and decoding on the entire speech coded feature sequence, thus achieving streaming speech recognition.

[0086] As an optional implementation method, see [link to implementation details]. Figure 2 As shown in another embodiment of this application, the training process of the attention encoding network includes the following steps:

[0087] S201. Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, determine the coding feature sequence to be identified corresponding to each word in the word tags from all sample speech coding features.

[0088] Specifically, to train the attention encoding network, it is first necessary to obtain a sample speech encoding feature sequence. This sequence can be obtained by extracting and encoding speech features from pre-collected sample speech data. Specifically, first, speech features are extracted from each sample speech frame in the pre-collected sample speech data to obtain the corresponding sample speech features for each frame; then, the sample speech features corresponding to each frame are encoded to obtain the corresponding sample speech encoding features, which are then combined to form the sample speech encoding feature sequence. The method for encoding speech features has been provided in the above embodiments and will not be elaborated further in this embodiment.

[0089] After obtaining the sample speech coding feature sequence, it is necessary to force alignment of each sample speech coding feature in the sequence to determine the word label corresponding to each sample speech coding feature; that is, to align sample speech coding features with the same word label. Forced alignment can be performed using the Viterbi algorithm in an acoustic model, where the acoustic model can be a DNN-HMM model. When the sample speech is English, each word in the word label is an English word; when the sample speech is Chinese, each word in the word label is a Chinese character.

[0090] In this embodiment, it is necessary to determine the to-be-recognized coding feature sequence corresponding to each word in the word label from all sample speech coding features. For example, if the sample speech data is the speech of "good morning", and the word label corresponding to the sample speech coding feature sequence includes the word label corresponding to "morning", the word label corresponding to "good", and the word label corresponding to "you", then determining the to-be-recognized coding feature sequence corresponding to each word in the word label from all sample speech coding features is to determine the to-be-recognized coding feature sequence corresponding to the word label of "morning", the to-be-recognized coding feature sequence corresponding to the word label of "good", and the to-be-recognized coding feature sequence corresponding to the word label of "you" from all sample speech coding features.

[0091] Further, the specific steps are as follows:

[0092] First, according to the word label carried by each sample speech coding feature in the sample speech coding feature sequence, the sample speech coding feature corresponding to each word in the word label is determined from all sample speech coding features.

[0093] Each sample speech coding feature in the sample speech coding feature sequence is force-aligned with each word in the word label through forced alignment, so that the sample speech coding feature corresponding to each word in the word label can be determined. For example, for the three words "morning", "good" and "you" in the word label, the sample speech coding feature corresponding to "morning", the sample speech coding feature corresponding to "good", and the sample speech coding feature corresponding to "you" are extracted from all the sample speech coding features in the sample speech coding feature sequence. As shown in Figure 3 and Figure 4 , each row represents the sample speech coding feature sequence corresponding to the sample speech data "good morning", each circle marks one sample speech coding feature, Figure 3 and Figure 4 the sample speech coding features in the solid-line rectangular box in the first row of each of the two figures are the sample speech coding features corresponding to the word "morning" in the word label, the sample speech coding features in the solid-line rectangular box in the second row are the sample speech coding features corresponding to the word "good" in the word label, and the sample speech coding features in the solid-line rectangular box in the third row are the sample speech coding features corresponding to the word "you" in the word label.

[0094] Second, for each word in the word label, a first sample speech coding feature sequence is extracted from the sample speech coding feature sequence, and is used as the to-be-recognized coding feature sequence corresponding to the word.

[0095] This embodiment can extract a first sample speech coding feature sequence from the sample speech coding feature sequence corresponding to each word in the word tag as the coding feature sequence to be identified for each word. The first sample speech coding feature sequence corresponding to each word in the word tag contains at least a portion of the sample speech coding features corresponding to that word. This embodiment provides two methods for extracting the coding feature sequence to be identified for each word in the word tag from the sample speech coding feature sequence:

[0096] In the first approach, the first sample speech coding feature sequence corresponding to a word in the word tag can be a sample speech coding feature sequence of a set length that contains at least some of the sample speech coding features corresponding to that word.

[0097] If the length of the sequence consisting of all sample speech coding features corresponding to a word in the sample speech coding feature sequence is less than a set length, then a sample speech coding feature of the set length can be extracted from the sample speech coding features corresponding to the word and the sample speech coding features adjacent to the sample speech coding features corresponding to the word as the first sample speech coding feature sequence. In this case, the first sample speech coding feature sequence includes all sample speech coding features corresponding to the word and the sample speech coding features adjacent to it. Figure 3 In the example, the length is set to 3, but there are only 2 sample speech coding features corresponding to "early". At this time, the sample speech coding features corresponding to "early" and a sample speech coding feature adjacent to the sample speech coding feature are extracted as the first sample speech coding feature sequence corresponding to "early". That is, the sample speech coding feature sequence in the dashed rectangle in the first row is the coding feature sequence to be identified corresponding to "early".

[0098] If the length of the sequence consisting of all sample speech coding features corresponding to a word in the sample speech coding feature sequence is greater than a set length, and when the number of all sample speech coding features corresponding to the word is odd, then the sample speech coding feature of the center point of all sample speech coding features corresponding to the word is determined. The sample speech coding feature of the center point and the sample speech coding features to the left and right of the center point are taken as the first sample speech coding feature sequence corresponding to the word, where w is the set length. For example... Figure 3where there are 5 sample speech coding features corresponding to "shang", the length of the formed sequence is greater than the set length, the sample speech coding feature at the center point and one sample speech coding feature taken respectively from the left and right of the sample speech coding feature at the center point are taken as the first sample speech coding feature sequence corresponding to "shang", that is, the sample speech coding feature sequence in the dashed rectangular box in the second row is the to-be-identified coding feature sequence corresponding to "shang". When the number of all sample speech coding features corresponding to the word is an even number, the right one of the two sample speech coding features at the center of all sample speech coding features corresponding to the word is taken as the sample speech coding feature at the center point, then according to the above manner, the sample speech coding feature at the center point and (w-1) / 2 sample speech coding features taken respectively from the left and right of the sample speech coding feature at the center point are taken as the first sample speech coding feature sequence corresponding to the word, that is, the to-be-identified coding feature sequence.

[0099] If the length of the sequence formed by all sample speech coding features corresponding to the word in the sample speech coding feature sequence is equal to the set length, the sequence formed by all sample speech coding features corresponding to the word is directly taken as the first sample speech coding feature sequence, that is, the to-be-identified coding feature sequence. For example Figure 3 where there are 3 sample speech coding features corresponding to "hao", the length of the formed sequence is the same as the set length, the sequence formed by the sample speech coding features corresponding to "hao" is directly taken as the first sample speech coding feature sequence corresponding to "hao", that is, the sample speech coding feature sequence in the dashed rectangular box in the third row is the to-be-identified coding feature sequence corresponding to "hao".

[0100] In the second manner, the first sample speech coding feature sequence corresponding to the word in the word label may be a sample speech coding feature sequence formed by the sample speech coding features corresponding to the word and the sample speech coding features before the sample speech coding features corresponding to the word, that is, a sample speech coding feature sequence formed by the last sample speech coding feature among the sample speech coding features corresponding to the word and the sample speech coding features before it.

[0101] As Figure 4 shown, the sample speech coding feature sequence in the dashed rectangular box in the first row is the first sample speech coding feature sequence corresponding to "zao", that is, the to-be-identified coding feature sequence corresponding to "zao". The sample speech coding feature sequence in the dashed rectangular box in the second row is the first sample speech coding feature sequence corresponding to "shang", that is, the to-be-identified coding feature sequence corresponding to "shang". The sample speech coding feature sequence in the dashed rectangular box in the third row is the first sample speech coding feature sequence corresponding to "hao", that is, the to-be-identified coding feature sequence corresponding to "hao".

[0102] S202. Based on the word labels and the sequence of encoding features to be identified corresponding to each word in the word labels, construct a sample encoding feature matrix carrying attention labels.

[0103] Based on the word tags in the sample speech coding feature sequence and the identification feature sequence corresponding to each word in the word tags determined in the above steps, a sample coding feature matrix carrying attention tags is constructed. Specifically, for a word in the word tag, this embodiment can perform a masking operation on the sample speech coding features other than the identification feature sequence corresponding to that word in the sample speech coding feature sequence, thereby obtaining the mask sequence corresponding to that word. The mask sequences corresponding to all words in the word tag are combined into a mask matrix, and this mask matrix is ​​used as the sample coding feature matrix. Figure 3 and Figure 4 The mask matrix is ​​obtained by masking the sample speech coding features outside the target coding feature sequence in each row of the diagram (i.e., the sample speech coding features not within the dashed rectangle). Specifically, for... Figure 3 The matrix obtained after the masking operation is the sample encoding feature matrix obtained in the first method described above. Figure 4 The matrix obtained after the masking operation is the sample encoding feature matrix obtained in the second method of the above steps.

[0104] In this embodiment, the sequence interval of the coding feature sequence to be identified for each word in the sample speech coding feature sequence, and the word label corresponding to each sequence interval, are also used as the attention label corresponding to the sample coding feature matrix.

[0105] S203. Input the sample encoding feature matrix into the attention encoding network to obtain the sample attention encoding result.

[0106] Specifically, in this embodiment, the sample encoding feature matrix needs to be input into the attention encoding network so that the attention encoding network can perform soft attention calculation to obtain the sample attention encoding result. The soft attention calculation step has been provided in the above embodiment and will not be described in detail in this embodiment.

[0107] S204. Adjust the parameters of the attention encoding network using the loss function between the recognition result after decoding the sample attention encoding result and the attention label corresponding to the sample encoding feature matrix.

[0108] The sample attention encoding result is input into the decoding network. The posterior probability distribution data of the samples output by the decoding network is searched and decoded to obtain the decoded recognition result. The loss function between the recognition result and the attention label corresponding to the sample encoded feature matrix is ​​calculated. This loss function is used to adjust the parameters of the attention encoding network, and it can also be used to adjust the parameters of the decoding network. In this embodiment, the attention loss between the sample posterior probability distribution data output by the decoding network and the attention label can also be directly calculated, and this attention loss can be used to adjust the parameters of both the attention encoding network and the decoding network.

[0109] Since the sample speech coding features other than the coding feature sequence to be identified are masked in the sample coding feature matrix input into the attention coding network, the attention coding network can be trained to determine the speech coding features to be calculated more accurately when performing soft attention calculation, that is, the extracted target coding feature sequence is more accurate, thereby improving the accuracy of attention coding.

[0110] As an optional implementation method, see [link to implementation details]. Figure 5 As shown in another embodiment of this application, the training process of the speech encoder includes the following steps:

[0111] S501. Extract the speech features of each speech frame in the pre-collected sample speech data to obtain the sample speech feature sequence, and determine the text annotation sequence corresponding to the sample speech feature sequence based on the text annotation corresponding to the sample speech data.

[0112] Specifically, in order to train the speech encoder, this embodiment first needs to collect sample speech data, then extract speech features from each speech frame in the sample speech data to obtain the speech features corresponding to each speech frame, and combine them to obtain the sample speech feature sequence corresponding to the sample speech data, such as the sample speech feature sequence X in the above embodiment. The pre-collected sample speech data carries text annotations corresponding to the sample speech, and the characters in the text annotations are combined to form the text annotation sequence corresponding to the sample speech feature sequence, such as the text annotation sequence Y in the above embodiment.

[0113] S502. Input the sample speech feature sequence into the speech encoder, and use the speech encoder to encode each speech feature in the sample speech feature sequence based on the attention mechanism to obtain the sample speech encoded feature sequence.

[0114] Specifically, in this embodiment, the sample speech feature sequence is input into a speech encoder, and the speech encoder may perform encoding processing based on an attention mechanism on each speech feature in the sample speech feature sequence, so as to obtain the encoded sample speech encoding feature of each speech feature, and all the sample speech encoding features are combined to obtain a sample speech encoding feature sequence, such as the sample speech encoding feature sequence H in the foregoing embodiment.

[0115] S503. Identify the word label corresponding to each speech encoding feature in the sample speech encoding feature sequence.

[0116] In this embodiment, it is required to identify the word label corresponding to each speech encoding feature in the sample speech encoding feature sequence. Specifically, in this embodiment, an acoustic model may be used to perform forced alignment on all speech encoding features in the sample speech encoding feature sequence, so as to obtain the phoneme label corresponding to each speech encoding feature, where the phoneme label corresponding to a speech encoding feature is the phoneme matched with the speech frame corresponding to the speech encoding feature. The acoustic model may adopt a DNN-HMM model, which may use the Viterbi algorithm to perform forced alignment on all speech encoding features in the sample speech encoding feature sequence. The DNN-HMM model is an existing model, and the training and application of the DNN-HMM model belong to the prior art, which will not be specifically described in this embodiment.

[0117] In this embodiment, it is also required to map the phoneme label corresponding to each speech encoding feature to a word label according to the mapping relationship between phonemes and words. For example, in a Chinese recognition environment, the phonemes corresponding to the character "Zao" include two phonemes: z and ao, so it is necessary to map the phoneme labels of all speech encoding features whose phoneme labels are z and ao in the speech encoding features to the word label corresponding to "Zao".

[0118] S504. Adjust parameters of the speech encoder based on the loss function between the word label corresponding to the sample speech encoding feature sequence and the text annotation sequence.

[0119] Specifically, in this embodiment, the cross-entropy loss function between the word label corresponding to the sample speech encoding feature sequence and the text annotation sequence is calculated, and then the cross-entropy loss function is used to adjust the parameters of the speech encoder, so as to improve the encoding accuracy of the speech encoder.

[0120] Corresponding to the foregoing speech recognition method, an embodiment of the present application further provides a speech recognition apparatus, see Figure 6 as shown, the apparatus includes:

[0121] A speech encoding module 100, configured to encode a speech feature sequence corresponding to a to-be-recognized speech to obtain a speech encoding feature sequence;

[0122] Attention coding module 110 is used to extract the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and to perform attention coding on the target coding feature sequence to obtain the attention coding result;

[0123] The decoding module 120 is used to decode the attention encoding result to obtain the speech recognition result.

[0124] The speech recognition device proposed in this application uses a speech coding module 100 to encode the speech feature sequence corresponding to the speech to be recognized, obtaining a speech coding feature sequence; an attention coding module 110 extracts the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features in the speech coding feature sequence, and performs attention coding on the target coding feature sequence to obtain an attention coding result; a decoding module 120 decodes the attention coding result to obtain the speech recognition result. By adopting the technical solution of this embodiment, the target coding feature sequence to be recognized can be extracted from all speech coding features, so that attention coding and decoding of the target coding feature sequence can be performed on the target coding feature sequence, without waiting for a sentence to be finished before performing attention coding and decoding on the entire speech coding feature sequence, thus realizing streaming speech recognition.

[0125] As an optional implementation, another embodiment of this application also discloses an attention encoding module 110, specifically used for:

[0126] The speech coding feature sequence is input into a pre-trained attention coding network, so that the attention coding network extracts the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and performs attention coding on the target coding feature sequence to obtain the attention coding result;

[0127] The attention encoding network is trained by performing attention encoding on the sample speech encoding feature sequence based on the sample speech encoding feature sequence and the attention label corresponding to the sample speech encoding feature sequence. The attention label corresponding to the sample speech encoding feature sequence includes the sample speech encoding feature sequence interval corresponding to each text character in the recognized text corresponding to the sample speech encoding feature sequence.

[0128] As an optional implementation, another embodiment of this application also discloses that the speech recognition device of this embodiment further includes: a first determining module, a matrix construction module, a first input module, and a first parameter adjustment module.

[0129] The first determining module is used to determine the coding feature sequence to be identified from all sample speech coding features based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence; the word tags carried by the sample speech coding features are the text that matches the speech frame corresponding to the sample speech coding features.

[0130] The matrix construction module is used to construct a sample encoding feature matrix carrying attention labels based on word labels and the sequence of encoding features to be identified corresponding to each word in the word labels.

[0131] The first input module is used to input the sample encoding feature matrix into the attention encoding network to obtain the sample attention encoding result;

[0132] The first parameter adjustment module is used to adjust the parameters of the attention encoding network by using the loss function between the recognition result after decoding the sample attention encoding result and the attention label corresponding to the sample encoding feature matrix.

[0133] As an optional implementation, another embodiment of this application also discloses a first determining module, specifically used for:

[0134] Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, determine the sample speech coding feature corresponding to each word in the word tags from all sample speech coding features;

[0135] For each word in the corresponding word label, the first sample speech coding feature sequence is extracted from the sample speech coding feature sequence as the coding feature sequence to be identified for that word. The first sample speech coding feature sequence contains at least some of the sample speech coding features in the sample speech coding features corresponding to that word.

[0136] As an optional implementation, another embodiment of this application also discloses that the first sample speech coding feature sequence is a sample speech coding feature sequence of a set length that at least includes a portion of the sample speech coding features corresponding to the word.

[0137] or,

[0138] The first sample speech coding feature sequence is a sequence of sample speech coding features consisting of the sample speech coding features corresponding to the word and the sample speech coding features preceding the sample speech coding features corresponding to the word.

[0139] As an optional implementation, another embodiment of this application also discloses a matrix construction module, specifically used for:

[0140] Masking is performed on the sample speech coding features outside the coding feature sequence to be identified in the sample speech coding feature sequence to obtain the sample coding feature matrix corresponding to the coding feature sequence to be identified. The sequence interval of the coding feature sequence to be identified in the sample speech coding feature sequence and all word labels corresponding to the sequence interval are used as the attention labels corresponding to the sample coding feature matrix.

[0141] As an optional implementation, another embodiment of this application also discloses a speech coding module 100, specifically used for:

[0142] The speech feature sequence corresponding to the speech to be recognized is input into a pre-trained speech encoder to obtain the speech coding feature sequence corresponding to the speech to be recognized.

[0143] The speech encoder is trained by encoding the sample speech feature sequence based on an attention mechanism, using the sample speech feature sequence and the text annotations corresponding to each speech feature in the sample speech feature sequence.

[0144] As an optional implementation, another embodiment of this application also discloses that the voice recognition device of this embodiment further includes: a second determining module, a second input module, a tag recognition module, and a second parameter adjustment module.

[0145] The second determining module is used to extract the speech features of each speech frame in the pre-collected sample speech data to obtain the sample speech feature sequence, and determine the text annotation sequence corresponding to the sample speech feature sequence based on the text annotation corresponding to the sample speech data.

[0146] The second input module is used to input the sample speech feature sequence into the speech encoder, and use the speech encoder to encode each speech feature in the sample speech feature sequence based on the attention mechanism to obtain the sample speech encoded feature sequence.

[0147] The label recognition module is used to identify the word labels corresponding to each speech coding feature in the sample speech coding feature sequence; the word labels corresponding to the speech coding features are the text that matches the speech frame corresponding to the speech coding features.

[0148] The second parameter adjustment module is used to adjust the parameters of the speech encoder based on the loss function between the word labels corresponding to the sample speech coding feature sequence and the text annotation sequence.

[0149] As an optional implementation, another embodiment of this application also discloses a tag recognition module, specifically used for:

[0150] An acoustic model is used to force alignment of all speech coding features in the sample speech coding feature sequence to obtain the phoneme label corresponding to each speech coding feature; the phoneme label corresponding to the speech coding feature is the phoneme that matches the speech frame corresponding to the speech coding feature.

[0151] The phoneme labels corresponding to the speech coding features are mapped to word labels.

[0152] The speech recognition device provided in this embodiment belongs to the same concept as the speech recognition method provided in the above embodiments of this application. It can execute the speech recognition method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of executing the speech recognition method. Technical details not described in detail in this embodiment can be found in the specific processing content of the speech recognition method provided in the above embodiments of this application, and will not be repeated here.

[0153] Another embodiment of this application discloses an electronic device, see [link to relevant documentation] Figure 7 As shown, the device includes:

[0154] Memory 200 and processor 210;

[0155] The memory 200 is connected to the processor 210 and is used to store programs;

[0156] The processor 210 is used to implement the speech recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0157] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0158] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0159] A bus can include a pathway for transmitting information between various components of a computer system.

[0160] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0161] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0162] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0163] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0164] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0165] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0166] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the speech recognition method provided in the embodiments of this application.

[0167] Another embodiment of this application also provides a storage medium storing a computer program, which, when executed by a processor, implements the various steps of the speech recognition method provided in any of the above embodiments.

[0168] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0169] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0170] The steps in the methods of the various embodiments of this application can be adjusted, combined, or deleted according to actual needs.

[0171] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0172] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0173] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0174] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0175] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0176] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0178] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that, include: The speech feature sequence corresponding to the speech to be recognized is encoded to obtain the speech encoded feature sequence; The speech coding feature sequence is input into a pre-trained attention coding network, so that the attention coding network extracts the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and performs attention coding on the target coding feature sequence to obtain the attention coding result; The attention encoding result is decoded to obtain the speech recognition result; The training samples of the attention coding network include sample speech coding feature sequences, and the attention labels corresponding to the sample speech coding feature sequences include the sample speech coding feature sequence intervals corresponding to each text character in the recognized text corresponding to the sample speech coding feature sequences. The attention encoding network is obtained by training the sample speech encoding feature sequence with attention encoding based on the sample speech encoding feature sequence and the attention label corresponding to the sample speech encoding feature sequence.

2. The method according to claim 1, characterized in that, The training process of the attention encoding network includes: Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, the coding feature sequence to be identified corresponding to each word in the word tags is determined from all sample speech coding features; the word tags carried by the sample speech coding features are the text that matches the speech frame corresponding to the sample speech coding features; Based on the word labels and the sequence of coded features to be identified corresponding to each word in the word labels, a sample coding feature matrix carrying attention labels is constructed; The sample encoding feature matrix is ​​input into the attention encoding network to obtain the sample attention encoding result; The parameters of the attention encoding network are adjusted using the loss function between the recognition result after decoding the sample attention encoding result and the attention label corresponding to the sample encoding feature matrix.

3. The method according to claim 2, characterized in that, Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, determine the code feature sequence to be identified corresponding to each word in the word tags from all sample speech coding features, including: Based on the word tags carried by each sample speech coding feature in the sample speech coding feature sequence, determine the sample speech coding feature corresponding to each word in the word tags from all sample speech coding features; For each word in the word tag, a first sample speech coding feature sequence is extracted from the sample speech coding feature sequence as the coding feature sequence to be identified for that word. The first sample speech coding feature sequence contains at least some sample speech coding features from the sample speech coding features corresponding to that word.

4. The method according to claim 3, characterized in that, The first sample speech coding feature sequence is a sample speech coding feature sequence of a set length that contains at least a portion of the sample speech coding features corresponding to the word. or, The first sample speech coding feature sequence is a sample speech coding feature sequence composed of the sample speech coding features corresponding to the word and the sample speech coding features preceding the sample speech coding features corresponding to the word.

5. The method according to claim 2, characterized in that, Based on the sequence of encoded features to be identified, a sample encoded feature matrix carrying attention labels is constructed, including: A masking operation is performed on the sample speech coding features other than the coding feature sequence to be identified in the sample speech coding feature sequence to obtain the sample coding feature matrix corresponding to the coding feature sequence to be identified. The sequence interval of the coding feature sequence to be identified in the sample speech coding feature sequence and all word labels corresponding to the sequence interval are used as the attention labels corresponding to the sample coding feature matrix.

6. The method according to claim 1, characterized in that, The process of encoding the speech feature sequence corresponding to the speech to be recognized to obtain the speech encoded feature sequence includes: The speech feature sequence corresponding to the speech to be recognized is input into a pre-trained speech encoder to obtain the speech coding feature sequence corresponding to the speech to be recognized. The speech encoder is obtained by encoding the sample speech feature sequence based on an attention mechanism, using the sample speech feature sequence and the text annotations corresponding to each speech feature in the sample speech feature sequence.

7. The method according to claim 6, characterized in that, The training process of the speech encoder includes: Extract speech features from each speech frame in the pre-collected sample speech data to obtain a sample speech feature sequence, and determine the text annotation sequence corresponding to the sample speech feature sequence based on the text annotation corresponding to the sample speech data. The sample speech feature sequence is input into the speech encoder, and the speech encoder is used to encode each speech feature in the sample speech feature sequence based on an attention mechanism to obtain the sample speech encoded feature sequence. Identify the word tags corresponding to each speech coding feature in the sample speech coding feature sequence; the word tags corresponding to the speech coding features are the text that matches the speech frame corresponding to the speech coding features; The parameters of the speech encoder are adjusted based on the loss function between the word tags corresponding to the sample speech coding feature sequence and the text annotation sequence.

8. The method according to claim 7, characterized in that, Identifying word tags corresponding to each speech coding feature in the sample speech coding feature sequence, including: An acoustic model is used to force alignment of all speech coding features in the sample speech coding feature sequence to obtain phoneme labels corresponding to each speech coding feature; the phoneme labels corresponding to the speech coding features are the phonemes that match the speech frames corresponding to the speech coding features. The phoneme tags corresponding to the speech coding features are mapped to word tags.

9. A voice recognition device, characterized in that, include: The speech coding module is used to encode the speech feature sequence corresponding to the speech to be recognized, so as to obtain the speech coding feature sequence. The attention coding module is used to input the speech coding feature sequence into a pre-trained attention coding network, so that the attention coding network extracts the target coding feature sequence corresponding to the current recognition result to be determined from all speech coding features of the speech coding feature sequence, and performs attention coding on the target coding feature sequence to obtain the attention coding result; The decoding module is used to decode the attention encoding result to obtain the speech recognition result; The training samples of the attention coding network include sample speech coding feature sequences, and the attention labels corresponding to the sample speech coding feature sequences include the sample speech coding feature sequence intervals corresponding to each text character in the recognized text corresponding to the sample speech coding feature sequences. The attention encoding network is obtained by training the sample speech encoding feature sequence with attention encoding based on the sample speech encoding feature sequence and the attention label corresponding to the sample speech encoding feature sequence.

10. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the speech recognition method as described in any one of claims 1 to 8 by running a program in the memory.

11. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the speech recognition method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Real-time speech recognition method and device based on truncated attention, equipment and computer readable storage medium

    CN111429889A

  • Speech recognition model performance detection method and device, equipment and storage medium

    CN113257227A