A speech recognition method, device, equipment and storage medium

Abstract

The present application provides a speech recognition method, apparatus, device and storage medium. The method includes: obtaining encoded features obtained by encoding the acoustic features of the speech to be recognized by an encoder; wherein, the encoder is trained according to the first recognition result of a speech sample and the text label of the speech sample, and the first recognition result of the speech sample is determined according to the encoded features obtained by encoding the acoustic features of the speech sample by the encoder and the attention coefficients of each frame of encoded features output by the encoder and the recognition result of the speech sample; determining a speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized. By adopting the above technical solution, the speech recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of speech recognition, and more specifically, provides a speech recognition method, apparatus, device, and storage medium. Background Art

[0002] Automatic Speech Recognition (ASR) technology is a technology that enables a machine to convert a speech signal into a corresponding text or command through a recognition and understanding process, that is, to enable the machine to understand human speech.

[0003] Currently, end-to-end speech recognition is the mainstream solution, among which the recognition effect of the end-to-end speech recognition solution based on the attention mechanism is the best. However, it is difficult for the attention mechanism of a conventional end-to-end model based on the attention mechanism to ensure monotonicity. Specifically, the attention of the model is unconstrained and has no order, which makes it difficult to improve the recognition accuracy of the model. Especially when facing the requirement of streaming recognition, the recognition effect is often not good. Summary of the Invention

[0004] Based on the above technical status quo, the present application provides a speech recognition method, apparatus, device, and storage medium, which can constrain the attention of speech recognition, thereby improving the speech recognition accuracy.

[0005] A speech recognition method includes:

[0006] Obtaining encoded features obtained by an encoder encoding acoustic features of a speech to be recognized;

[0007] Wherein, the encoder is trained according to a first recognition result of a speech sample and a text label of the speech sample, and the first recognition result of the speech sample is determined according to the encoded features obtained by the encoder encoding the acoustic features of the speech sample and the attention coefficients of each frame of encoded features output by the encoder for the recognition result of the speech sample;

[0008] Determining a speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

[0009] Optionally, determining a speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized includes:

[0010] Decoding the encoded features of the speech to be recognized to obtain decoded features of the speech to be recognized;

[0011] Determining attention coefficients of the recognition result of the speech to be recognized for each frame of encoded features of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized;

[0012] Determine the speech recognition result of the speech to be recognized according to the encoding features and decoding features of the speech to be recognized, and the attention coefficients of the encoding features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized.

[0013] Optionally, the encoder encodes the acoustic features of the speech to be recognized to obtain encoding features, including:

[0014] Perform encoding processing based on the self-attention mechanism on each frame of the acoustic features of the speech to be recognized, so as to obtain the encoding features of the speech to be recognized;

[0015] Among them, performing encoding processing based on the self-attention mechanism on any frame of acoustic features includes: encoding the frame of acoustic features according to an acoustic feature sequence of a set length including this frame of acoustic features, to obtain the encoding features corresponding to this frame of acoustic features.

[0016] Optionally, decoding the encoding features of the speech to be recognized to obtain the decoding features of the speech to be recognized, including:

[0017] Decode the encoding features of each frame of the speech to be recognized and the decoding features corresponding to the recognized results of the speech to be recognized, to obtain the decoding features of the speech to be recognized;

[0018] Among them, the recognized result of the speech to be recognized is the recognition result of the speech to be recognized obtained before the current moment.

[0019] Optionally, determining the speech recognition result of the speech to be recognized according to the encoding features and decoding features of the speech to be recognized, and the attention coefficients of the encoding features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized, includes:

[0020] Determine the first decoding result of the speech to be recognized according to the encoding features of each frame of the speech to be recognized and the attention coefficients of the encoding features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized;

[0021] Determine the second decoding result of the speech to be recognized according to the decoding features of the speech to be recognized;

[0022] Determine the speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result.

[0023] Optionally, determining the first decoding result of the speech to be recognized according to the encoding features of each frame of the speech to be recognized and the attention coefficients of the encoding features of each frame of the speech to be recognized with respect to the decoding result of the speech to be recognized, includes:

[0024] Decode the encoded features of each frame of the speech to be recognized respectively to obtain the decoding results corresponding to the encoded features of each frame;

[0025] Use the attention coefficients of the encoded features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized as weights to weight the decoding results corresponding to the encoded features of each frame, so as to obtain the first decoding result of the speech to be recognized.

[0026] Optionally, the first decoding result and the second decoding result each include multiple decoding paths;

[0027] Determine the speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result, including:

[0028] Perform weighted summation on the scores of each decoding path of the first decoding result and the second decoding result, and determine the speech recognition result of the speech to be recognized from the decoding paths after the weighted summation of the scores.

[0029] Optionally, the training process of the encoder includes:

[0030] Obtain the encoded features obtained by the encoder encoding the acoustic features of the speech sample, and obtain the decoded features obtained by decoding the encoded features of the speech sample;

[0031] Determine the attention coefficients of the recognition result of the speech sample with respect to the encoded features of each frame of the speech sample according to the encoded features and decoded features of the speech sample;

[0032] Determine the first recognition result of the speech sample according to the encoded features of the speech sample and the attention coefficients;

[0033] Optimize the parameters of the encoder by comparing the first recognition result with the label of the speech sample.

[0034] Optionally, obtain the encoded features obtained by the encoder processing the acoustic features of the speech to be recognized, and determine the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized, including:

[0035] Input the acoustic features of the speech to be recognized into a pre-trained speech recognition model, so that the speech recognition model obtains the encoded features obtained by its encoder processing the acoustic features of the speech to be recognized, and determine the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

[0036] Optionally, the speech recognition model includes an encoder and a decoder, and the training process of the speech recognition model includes:

[0037] Input the acoustic features of the speech sample into an encoder to obtain the encoded features output by the encoder, and use a decoder to decode the encoded features to obtain decoded features;

[0038] Determine the attention coefficients of the decoded result of the speech sample for each frame of the encoded features of the speech sample according to the encoded features and the decoded features;

[0039] Determine a first recognition result for the speech sample according to the encoded features of the speech sample and the attention coefficients, and determine a first loss function by comparing the first recognition result with the label of the speech sample;

[0040] Determine a second recognition result for the speech sample according to the decoded features of the speech sample, and determine a second loss function by comparing the second recognition result with the label of the speech sample;

[0041] Optimize the parameters of the encoder and the decoder by using the first loss function and the second loss function.

[0042] Optionally, optimizing the parameters of the encoder and the decoder by using the first loss function and the second loss function includes:

[0043] Perform weighted summation on the first loss function and the second loss function to obtain a joint loss function;

[0044] Optimize the parameters of the encoder by using the joint loss function;

[0045] And,

[0046] Optimize the parameters of the decoder by using the second loss function, or by using the second loss function and the joint loss function.

[0047] A speech recognition device, comprising:

[0048] An encoding unit, configured to obtain the encoded features obtained by an encoder encoding the acoustic features of the speech to be recognized;

[0049] Wherein, the encoder is trained according to the first recognition result of the speech sample and the text label of the speech sample, and the first recognition result of the speech sample is determined according to the encoded features obtained by the encoder encoding the acoustic features of the speech sample and the attention coefficients of the recognition result of the speech sample for each frame of the encoded features output by the encoder;

[0050] A decoding unit, configured to determine a speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

[0051] A speech recognition device, comprising:

[0052] A memory and a processor;

[0053] Wherein, the memory is connected to the processor and is used for storing programs;

[0054] The processor is configured to implement the above-mentioned speech recognition method by running the program in the memory.

[0055] A storage medium, on which a computer program is stored, and when the computer program is run by a processor, the above-mentioned speech recognition method is implemented.

[0056] In the speech recognition method proposed in this application, the training process of the encoder combines the attention coefficients of each frame of encoded features output by the encoder with the speech recognition result. This training method realizes the regularization of the encoder, which is beneficial to promoting the monotonicity of the encoder attention mechanism. Based on the encoder trained by this training method, among the encoded features of the speech to be recognized output by it, the encoded feature frames concerned by the speech recognition attention are more accurate, so that the encoded features output by the encoder are more conducive to speech recognition, that is, the accuracy of the speech recognition result can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0058] Figure 1 is a schematic flowchart of a speech recognition method provided by an embodiment of the present application;

[0059] Figure 2 is a schematic structural diagram of a speech recognition model provided by an embodiment of the present application;

[0060] Figure 3 is a schematic structural diagram of a regularization module in a speech recognition model provided by an embodiment of the present application;

[0061] Figure 4 is a schematic structural diagram of an encoder of a speech recognition model provided by an embodiment of the present application;

[0062] Figure 5 is a schematic flowchart of another speech recognition method provided by an embodiment of the present application;

[0063] Figure 6 It is a schematic structural diagram of a decoder of a speech recognition model provided by an embodiment of the present application;

[0064] Figure 7 It is a schematic structural diagram of a speech recognition device provided by an embodiment of the present application;

[0065] Figure 8 It is a schematic structural diagram of a speech recognition device provided by an embodiment of the present application. Detailed implementation manners

[0066] The technical solution of the embodiment of the present application is applicable to the speech recognition application scenario. By adopting the technical solution of the embodiment of the present application, the accuracy and recognition efficiency of end-to-end speech recognition can be improved.

[0067] Speech recognition has a wide range of applications in the fields of home appliances, communications, automotive electronics, medical care, home services, consumer electronics, etc.

[0068] At present, end-to-end speech recognition is the most commonly used speech recognition solution. The mainstream end-to-end models mainly include three types: end-to-end ASR based on CTC (Connectionist Temporal Classification), encoder-decoder model based on attention (attention mechanism), and end-to-end ASR based on RNN-T (Recurrent Neural Network-Transducer). Although these three end-to-end models have shown excellent performance in the field of speech recognition, each has its own drawbacks. In order to improve efficiency, CTC assumes that the outputs between different frames are conditionally independent, and often requires an external language model during decoding. RNN-T solves the problem of conditional independence in CTC and integrates the language model into the system, but the model training is difficult, and there are some unreasonable decoding paths.

[0069] Relatively speaking, the speech recognition effect of the Encoder-Decoder model based on Attention is the best, but it is highly dependent on the length of the input sentence, and it is difficult to ensure monotonicity of attention. Specifically, the attention of the decoding end of the existing Encoder-Decoder model based on Attention to the encoding end is unconstrained and has no order, which makes it difficult to improve the recognition accuracy and efficiency of the model.

[0070] Based on the above-mentioned state of the art, the embodiments of this application propose a new speech recognition solution for an end-to-end model based on an attention mechanism. The end-to-end speech recognition implemented based on this solution can strengthen the constraints on the attention of the decoder on the encoder, so that the decoder's attention is focused on the correct encoding features, thereby regularizing the recognition process and further improving the recognition effect. At the same time, the above-mentioned attention regularization processing can ensure the monotonicity of the decoder's attention to the encoder, which has a positive effect on improving the accuracy of speech recognition, especially improving the accuracy of streaming recognition.

[0071] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0072] An embodiment of the present application proposes a speech recognition method, which is applicable to an end-to-end speech recognition model based on an attention mechanism.

[0073] Similar to conventional attention-based encoder-decoder models, this attention-based end-to-end speech recognition model also primarily consists of an encoder and a decoder. The encoder downsamples the original high-frame-rate acoustic feature sequence through convolution based on the characteristics of speech acoustic features (such as MFCCs or filterbanks) to a low-frame-rate feature sequence. Then, using a self-attention mechanism to focus on the context, it extracts speech-recognition-friendly hidden features, known as encoded features.

[0074] The decoder is generated by encoding the feature h output by the encoder e Attention and autoregressive mechanism, modeling conditional probability p(y t |h e ,y ＜t ). The decoder is optimized using the cross entropy loss L.

[0075] Based on the above end-to-end speech recognition model, the present application embodiment proposes a speech recognition method, see Figure 1 As shown, the method includes:

[0076] S101: Obtain encoding features obtained by encoding acoustic features of a speech to be recognized by an encoder.

[0077] Among them, the above encoder is trained according to the first recognition result of the speech sample and the text label of the speech sample. The first recognition result of the speech sample is determined according to the encoded features obtained by encoding the acoustic features of the speech sample by the encoder and the attention coefficients of each frame of encoded features output by the encoder for the recognition result of the speech sample.

[0078] Specifically, the acoustic features of the speech to be recognized can be any type of acoustic features obtained by extracting the acoustic features of the speech to be recognized. For example, they can be acoustic features of types such as MFCC and Filterbank.

[0079] Input the acoustic features of the speech to be recognized into the encoder of the above end-to-end speech recognition model. The encoder downsamples the original high-frame-rate acoustic feature sequence through convolution to obtain a low-frame-rate feature sequence, and then pays attention to the context through the self-attention mechanism to extract hidden layer features friendly to speech recognition, that is, the encoded features h are obtained. e Input the encoded feature h e into the decoder, and the decoder decodes it to obtain the decoded feature h d . By performing linear projection and softmax classification on the decoded feature h d , the decoding result at the current moment can be obtained.

[0080] In the conventional training scheme of the end-to-end speech recognition model, usually the recognition result of the speech sample output by the decoder is compared with the text label of the speech sample, the cross-entropy loss L is calculated, and the parameters of the encoder and decoder are optimized using the cross-entropy loss L.

[0081] In addition to the above training method, the embodiment of the present application also specially trains the encoder in the end-to-end speech recognition model.

[0082] See Figure 2 As shown, in the conventional end-to-end speech recognition model composed of an encoder and a decoder, the embodiment of the present application additionally adds a regularization module. The regularization module is used to perform MLP attention on the encoded feature h e output by the hidden layer of the encoder and the decoded feature h d output by the hidden layer of the decoder to obtain the attention coefficient. The attention coefficient is used to represent the attention coefficients of each frame of the encoded feature h d output by the hidden layer of the encoder by the decoded feature h e output by the hidden layer of the decoder. Since the decoded feature h d output by the hidden layer of the decoder is directly used to determine the decoding result, the attention coefficient can be used to represent the attention coefficients of the decoding result of the speech to each frame of encoded features of the speech.

[0083] The specific structure of the above regularization module is as Figure 3 shown. The encoded feature h of each frame of the speech e and the decoded feature h output by the decoder at the i-th moment d are each linearly projected and then added together. After passing through the tanh activation function, they pass through another linear projection, and finally the attention coefficient is calculated through softmax:

[0084] α i，t = softmax(Wtanh(uh e + vh d ))

[0085] where α i,t represents the attention coefficient of the decoded result of the speech at the i-th moment to the encoded feature of the t-th frame of the speech. W, u, and v are all parameters of the linear projection layer.

[0086] For the specific processing content of the above regularization module and the specific working content of the above encoder and decoder, reference can be made to the corresponding content in the following embodiments.

[0087] In the embodiments of the present application, the above regularization module is used for model training, especially for training the model encoder. During the training process, the acoustic features of the speech samples are input into the model encoder. Then, the encoded features of the speech samples output by the encoder are obtained, and the decoded features of the speech samples output by the decoder after decoding the encoded features output by the encoder are obtained. The MLP attention operation is performed on the encoded features and decoded features of the speech samples by using the regularization module. Through the operation of the above regularization module, the attention coefficients of the recognition results of the speech samples to the encoded features of each frame of the speech samples can be determined.

[0088] During the speech recognition process, the decoding end outputs the decoded results of partial speech frames of the speech samples at each moment. Therefore, the above determination of the attention coefficients of the recognition results of the speech samples to the encoded features of each frame of the speech samples is specifically to determine the attention coefficients of the decoded results at the current moment of the speech samples to the encoded features of each frame of the speech samples. These attention coefficients can be used to reflect the influence degree of the encoded features of each frame of the speech samples on the decoded results of the speech samples at the current moment. According to the above calculation, the attention coefficients of the decoded results of the speech samples at each moment to the encoded features of each frame of the speech samples can be determined.

[0089] Then, based on the encoded features of the speech sample and the calculated attention coefficients, determine the first recognition result for the speech sample. Specifically, after performing linear projection and softmax on the encoded features of each frame of the speech sample respectively, a V-dimensional probability distribution is obtained (V is the size of the dictionary). Then, using the above calculated attention coefficients, weight the probability distribution corresponding to the encoded features of each frame to obtain the probability distribution of each word in the dictionary, that is, obtain the first recognition result.

[0090] Finally, compare the first recognition result with the text label of the speech sample and calculate the cross-entropy loss L E , and then use this cross-entropy loss L E to optimize the parameters of the encoder.

[0091] Meanwhile, perform linear projection and softmax classification on the decoded features h d of the speech sample output by the decoder to obtain the decoding result, compare the decoding result with the text label of the speech sample and calculate the cross-entropy loss L D . This cross-entropy loss L D can also be used to optimize the parameters of the encoder and at the same time used to optimize the parameters of the decoder.

[0092] For the specific content of the above training process of the encoder, reference can also be made to the following training process of the end-to-end speech recognition model based on the attention mechanism.

[0093] It should be noted that training the encoder in the above manner makes the attention factors of each frame of the speech encoded features output by the encoder included in its loss function. Optimizing the parameters of the encoder based on this loss function can make the encoded features obtained by the encoder more accurate when encoding the speech acoustic features that the attention during decoding focuses on, so that the recognition result obtained based on this encoded feature is more accurate.

[0094] That is, the above training method that combines the attention coefficients of the recognition result for the encoded features plays a regularization role in the encoding process of the encoder, enabling the encoder to optimize the extraction of the encoded features that the speech recognition result focuses on, making the encoded features that the speech recognition result output by the encoder focuses on more accurate, which is conducive to improving the accuracy of the speech recognition result.

[0095] S102. Determine the speech recognition result for the speech to be recognized according to the encoded features of the speech to be recognized.

[0096] Exemplarily, after inputting the acoustic features of the speech to be recognized into the above-mentioned encoder, the encoded features obtained by encoding the acoustic features of the speech to be recognized output by the encoder are acquired. Then, the decoder decodes the encoded features to obtain decoded features. After performing linear projection and softmax classification on the decoded features, the decoding result at the current moment can be obtained. Sequentially concatenating the decoding results at each moment gives the complete speech recognition result of the speech to be recognized.

[0097] As can be seen from the above introduction, in the speech recognition method proposed in the embodiments of the present application, the training process of the encoder combines the attention coefficients of each frame of encoded features output by the encoder with the speech recognition result. This training method realizes the regularization of the encoder, which is beneficial to promoting the monotonicity of the encoder attention mechanism. For the encoder trained based on this training method, among the encoded features of the speech to be recognized output by it, the encoded feature frames focused by the speech recognition attention are more accurate, so that the encoded features output by the encoder are more conducive to speech recognition, that is, it can improve the accuracy of the speech recognition result.

[0098] As an optional implementation manner, the above-mentioned encoder adopts a network structure based on VGG-Transformer. Specifically, as Figure 4 shown, for the specific functions and names of each part in this network structure, reference can be made to the introduction of the network structures of VGG and Transformer in the prior art. Based on this network structure, the encoder first performs convolutional downsampling on the acoustic features of the speech to be recognized using the VGG network, and then uses the self-attention mechanism of the Transformer network structure to focus on the context and extract hidden layer features friendly to speech recognition, that is, the encoded features h <@ e .

[0099] Among them, Transformer is a model mechanism based on Attention. Before the advent of Transformer, in order to obtain long-distance dependency relationships, RNN was usually used for sequential encoding. Since the output of the RNN model depends on the current input and the previous hidden state each time, it cannot perform parallel computing, and the model efficiency is low. The multi-head self-attention mechanism proposed in Transformer enables the model to perform parallel computing.

[0100] Since self-attention has no sequential information, changing the order of the input sequence will not change the resulting output. To make the encoded result adapt to the input sequence, position encoding (PE) can be added during self-attention, giving each unit of the input sequence an embedding that is only related to the position. In this way, when the units of the sequence change order, the PE will also change accordingly, enabling self-attention to have position information and thus making the encoded result strictly match the input sequence. In the original Transformer, the PE is an absolute position information based on sin and cos. When the length of the samples in the test set is much larger than the common length in the training set, the obtained position encoding is not seen by the network, so the network will not get a robust result. Relative position encoding (relative position embedding) solves this problem well. Therefore, the Transformer network in the embodiments of this application adopts a multi-head self-attention mechanism with relative position encoding, that is Figure 4 the multi-head relative attention module in

[0101] It can be understood that the above encoder can implement the encoding function based on the self-attention mechanism. To meet the requirements of the streaming recognition scenario, in the embodiments of this application, it is set that when the encoder performs self-attention, it will intercept a segment and let the self-attention mechanism only focus on a small segment of the feature sequence before and after, rather than the entire speech sequence, so that it can be used for streaming recognition.

[0102] Specifically, when the above encoder encodes the acoustic features of the speech to be recognized, it specifically performs encoding processing based on the self-attention mechanism for each frame of the acoustic features of the speech to be recognized, so as to obtain the encoded features of the speech to be recognized.

[0103] Among them, when performing encoding processing based on the self-attention mechanism for any frame of acoustic features, the frame of acoustic features is encoded according to the set-length acoustic feature sequence containing the frame of acoustic features, and the encoded features corresponding to the frame of acoustic features are obtained. That is, the attention is limited to a small segment of the acoustic feature sequence containing the frame of acoustic features, and the frame of acoustic features is encoded to obtain the corresponding encoded features. Among them, a small segment of the acoustic feature sequence containing the frame of acoustic features can be a feature sequence composed of the frame of acoustic features and a small segment of the acoustic feature sequence before or after the frame of acoustic features.

[0104] Since the encoded attention is not on the entire speech sequence, encoded features for recognition can be obtained based on an acoustic feature sequence of a finite length, and then corresponding recognition results can be obtained, that is, streaming recognition effects can be achieved.

[0105] For the above-mentioned network structure of VGG-Transformer, the processing procedure of obtaining the encoded features of acoustic features by combining the self-attention mechanism based on this network structure, and the specific content of relative position encoding, reference can be made to the introduction of the network structure of VGG-Transformer and relative position encoding in the conventional technology, which will not be elaborated in the embodiments of this application.

[0106] In addition, the above-mentioned encoder can also adopt other network structures. For example, it can adopt a VGG-conformer network structure, a convolutional + conformer network structure, or a bidirectional LSTM network structure. The embodiments of this application do not make strict limitations, as long as it can implement the encoder functions introduced in the above embodiments of this application and can apply the encoder training scheme introduced in the above embodiments of this application.

[0107] As a preferred implementation manner, in addition to training the attention coefficients of the encoded features of each frame output by the encoder by means of the regularization module in combination with the recognition results of the speech samples as described above, in actual speech recognition applications, the speech recognition method proposed in the embodiments of this application also uses the regularization module to determine the speech recognition results of the speech to be recognized.

[0108] See Figure 5 As shown, for the speech recognition method proposed in the embodiments of this application, after inputting the acoustic features of the speech to be recognized into the encoder trained in the above manner, and after performing step S501 to obtain the encoded features obtained by the encoder encoding the acoustic features of the speech to be recognized, the speech recognition results of the speech to be recognized are obtained by performing the following steps S502 to S504:

[0109] S502. Decode the encoded features of the speech to be recognized to obtain the decoded features of the speech to be recognized.

[0110] Exemplarily, input the encoded features obtained by the encoder encoding the acoustic features of the speech to be recognized into the decoder, enable the decoder to decode it, and then obtain the feature vector output by the last hidden layer of the decoder, which is the decoding result of the encoded features, that is, the decoded features of the speech to be recognized.

[0111] It should be noted that the decoded feature obtained by decoding the above-mentioned encoded feature is specifically the decoded feature obtained by decoding the encoded feature and used to determine the decoding result at the current moment. Specifically, the "decoding result at the current moment" can be the complete recognition result of the entire speech to be recognized or the recognition result of some speech frames of the speech to be recognized.

[0112] When the speech to be recognized is short or in the offline speech recognition scenario, the complete recognition result of the speech to be recognized can be output at one time. At this time, the decoded feature obtained by decoding the encoded feature of the speech to be recognized is specifically the decoded feature used to determine the complete recognition result of the speech to be recognized.

[0113] When the speech to be recognized is long or in the streaming speech recognition scenario, the recognition result of the speech to be recognized is output in a streaming manner. At this time, the decoded feature obtained by decoding the encoded feature of the speech to be recognized is specifically the decoded feature used to determine the recognition result of the current speech frame to be recognized of the speech to be recognized.

[0114] Taking the streaming output of the speech recognition result as an example, this embodiment of the present application introduces the implementation process of recognizing the speech to be recognized and determining the decoding result at the current moment. The decoding results at other moments can also be obtained by referring to the description of this embodiment of the present application. Therefore, the decoded feature obtained by decoding the encoded feature is specifically the decoded feature used to determine the decoding result at the current moment, and this decoding result at the current moment is the recognition result of the current part of the speech frames to be recognized of the speech to be recognized.

[0115] When the speech recognition result is output in a streaming manner, the decoding results output by the decoding end are output in sequence, rather than outputting the entire recognition result of the speech to be recognized at one time.

[0116] For example, assume that the speech to be recognized has a total of T frames (frame 0 to frame T - 1), and correspondingly, T frames of encoded features can be obtained. When the decoder decodes the T frames of encoded features, at a certain moment, the decoded feature used to determine the recognition result of the current part of the speech frames to be recognized of the speech to be recognized is output, rather than the decoded feature used to determine the complete recognition result of the speech to be recognized at this moment.

[0117] For example, at the first moment, the decoder decodes the encoded features of the speech to be recognized to obtain decoded features. Based on these decoded features, the recognition result of the 0th frame of the speech to be recognized can be determined as the decoded result output by the decoder at the first moment. At the second moment, the decoder decodes the encoded features of the speech to be recognized to obtain decoded features. Based on these decoded features, the recognition result of the 1st frame of the speech to be recognized can be determined as the decoded result output by the decoder at the second moment. And so on. At the ith moment, the decoder decodes the encoded features of the speech to be recognized to obtain decoded features. Based on these decoded features, the recognition result of the (i - 1)th frame of the speech to be recognized can be determined as the decoded result output by the decoder at the ith moment. Finally, the decoded results output by the decoder at each moment are concatenated in sequence to obtain the complete recognition result of the speech to be recognized.

[0118] In the above example, the decoded features output by the decoder at a certain moment can also be the decoded features used to determine the recognition results of multiple speech frames to be recognized of the speech to be recognized. In this way, the decoded features output by the decoder each time are actually the decoded features used to determine the recognition results of one or more speech frames to be recognized of the speech to be recognized at the current moment.

[0119] As an exemplary implementation manner, the decoder in the embodiments of the present application is a decoder adopting a Transformer network structure, and its structure can be seen as Figure 6 shown. The attention in the decoder includes two parts. One part is the self-attention input to the decoder, which uses the Multi-head Relative Attention with relative position encoding here, the same as the encoder. The other part is the attention of the decoder to the encoder, and relative position is not required for this part, so a standard multi-head attention structure is used.

[0120] Based on the above decoder network structure, when obtaining the decoding features of the speech to be recognized with the help of the decoder, specifically, the encoded features of each frame of the speech to be recognized output by the encoder are input into the multi-head attention module of the decoder. At the same time, the decoding features corresponding to the recognized results of the speech to be recognized are input into the Multi-head RelativeAttention module. Among them, the recognized results of the speech to be recognized refer to the recognized results of the speech to be recognized obtained before the current moment. In the speech recognition scenario of streaming output of recognized results, in the order from front to back, the recognized results of some speech frames of the speech to be recognized are output at each moment. Correspondingly, the decoder outputs the decoding features used to determine the recognized results of some speech frames of the speech to be recognized at each moment. Therefore, the decoding features corresponding to the recognized results of the speech to be recognized are also the decoding features output by the decoder before the current moment. Therefore, the decoding features (outputs) output by the decoder before the current moment are input into the Multi-head Relative Attention module, so that the decoder jointly decodes the encoded features of each frame output by the encoder and the decoding features output by the decoder before the current moment, and obtains the decoding features of the current moment of the speech to be recognized.

[0121] It can be understood that when determining the decoding features of the current moment, the decoder refers to both the encoded features of the speech to be recognized and the decoding features output before the current moment. As a result, the decoding features of the current moment output by the decoder contain both all the information of the acoustic features of the speech to be recognized and the feature information of the previous recognized results, making the decoding feature information of the current moment richer and conducive to recognizing more accurate decoding results.

[0122] S503. Determine the attention coefficients of the recognized results of the speech to be recognized with respect to the encoded features of each frame of the speech to be recognized according to the encoded features and decoding features of the speech to be recognized.

[0123] Specifically, as can be seen from the above introduction, the decoding features output by the decoder are directly used to determine the decoding results of the current moment. Therefore, the attention coefficients of the decoding features output by the decoder at the current moment with respect to each frame of the encoded features of the speech to be recognized can represent the attention coefficients of the decoding results of the current moment of the speech to be recognized with respect to each frame of the encoded features of the speech to be recognized.

[0124] In the embodiments of the present application, through Figure 2 the regularization module of the end-to-end speech recognition model shown, for the encoded features h output by the encoder e and the decoding features h output by the decoder dPerform MLP attention operation to obtain an attention coefficient, which can represent the decoded feature h output by the decoder d For the encoded feature h output by the encoder e The attention coefficients for each frame, which are the attention coefficients of the decoded result of the speech to be recognized at the current moment to the encoded features of each frame of the speech to be recognized

[0125] The specific structure of the above regularization module is as follows Figure 3 As shown, project each frame of the encoded feature h of the speech to be recognized e and the decoded feature h output by the decoder at the i-th moment d through a linear projection respectively and then add them together. After passing through the tanh activation function, pass through another linear projection, and finally calculate the attention coefficient through softmax:

[0126] α i，t = softmax(Wtanh(uh e + vh d ))

[0127] where α i,t represents the attention coefficient of the decoded result (i.e., the recognition result) of the speech to be recognized at the i-th moment to the encoded feature of the t-th frame of the speech to be recognized. W, u, and v are all parameters of the linear projection layer

[0128] Through the above calculation, the attention coefficients of the recognition result of the speech to be recognized at the current moment to the encoded features of each frame of the speech to be recognized can be determined. This attention coefficient can be used to reflect the influence degree of the encoded features of each frame of the speech to be recognized on the recognition result of the speech to be recognized at the current moment

[0129] S504. Determine the speech recognition result of the speech to be recognized according to the encoded feature and decoded feature of the speech to be recognized, and the attention coefficient of the recognition result of the speech to be recognized to the encoded features of each frame of the speech to be recognized

[0130] Specifically, as a simple implementation method, perform linear projection and softmax classification on the decoded feature output by the decoder at the current moment (assumed to be the i-th moment), and the conditional probability distribution at the i-th moment can be obtained as follows

[0131]

[0132] where x represents the encoded feature of the acoustic feature of the speech to be recognized; W j and W j'are the parameters of the linear projection layer. The two values are actually the same, and only different symbols are used to distinguish the parameters in the dictionary dimension and the parameters at the current moment; V represents the dictionary dimension; y ＜i represents the decoding result before the i-th moment.

[0133] The dimension of the conditional probability distribution at the i-th moment mentioned above is the dictionary dimension, which can represent the probability of each word in the dictionary corresponding to the recognition result at the current moment, that is, the recognition result of the current moment of the speech to be recognized.

[0134] Alternatively, for each frame of encoded feature h output by the encoder e After performing linear projection and softmax respectively, a V-dimensional probability distribution (V is the dictionary size) is obtained, that is, the decoding results corresponding to each frame of encoded features are obtained. For example, after operating on the encoded feature of the t-th frame, its probability distribution is:

[0135]

[0136] Among them, W e represents the parameters of the linear projection layer; and are used to represent the encoded features of the speech to be recognized. The two values are actually the same, and only different symbols are used to distinguish the parameters in the dictionary dimension and the current parameters.

[0137] Then, using the attention coefficients of each frame of encoded features of the speech to be recognized with respect to the recognition result of the speech to be recognized as weights, the decoding results corresponding to each frame of encoded features are weighted to obtain the recognition result of the speech to be recognized.

[0138] Referring to the above introduction, assuming that the decoding result corresponding to the encoded feature of the t-th frame is p(y t |h e ), and the attention coefficient of the recognition result of the speech to be recognized at the i-th moment with respect to the encoded feature of the t-th frame is α i,t , then the decoding results corresponding to each frame of encoded features can be weighted according to the following formula to obtain the recognition result p i of the speech to be recognized at the i-th moment:

[0139]

[0140] The dimension of this recognition result p i is the dictionary dimension, that is, V-dimensional.

[0141] In the above two methods, the recognition result of the speech to be recognized is obtained through a single approach. Their implementation methods are simple, but the accuracy of the recognition result is not high enough.

[0142] To further improve the accuracy of the recognition result, the embodiments of the present application propose to combine the recognition results of the above two approaches to determine the recognition result of the speech to be recognized. Specifically, it can be implemented by performing the following steps A1 - A3:

[0143] A1. Determine the first decoding result of the speech to be recognized according to the encoded features of each frame of the speech to be recognized and the attention coefficients of the encoded features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized.

[0144] Specifically, for each frame of encoded feature h output by the encoder e After performing linear projection and softmax respectively, the decoding results corresponding to each frame of encoded feature are obtained. Then, using the attention coefficients of the encoded features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized as weights, the decoding results corresponding to each frame of encoded feature are weighted to obtain the recognition result of the speech to be recognized. For the sake of distinction, it is named the first decoding result of the speech to be recognized. For the specific process of the above processing, reference can be made to the introduction in the above embodiments.

[0145] A2. Determine the second decoding result of the speech to be recognized according to the decoding features of the speech to be recognized.

[0146] Specifically, perform linear projection and softmax classification on the decoding features output by the decoder to obtain the conditional probability distribution, which is the recognition result of the speech to be recognized at the i-th moment. For the sake of distinction, it is named the second decoding result of the speech to be recognized. For the specific processing process, reference can be made to the corresponding content introduction in the above embodiments.

[0147] A3. Determine the speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result.

[0148] Generally, whether it is the above first decoding result or the above second decoding result, there will not be only a single decoding path, but multiple decoding paths. For example, the first decoding result has 5 decoding paths, and the second decoding result also has 5 decoding paths. Moreover, there is a corresponding relationship between the decoding paths in the first decoding result and the decoding paths in the second decoding result. In each decoding path, the probability distribution of a certain recognized character for each character in the dictionary can represent the score of the recognized character.

[0149] On this basis, the embodiments of the present application perform weighted summation on the scores of the corresponding decoding paths in the first decoding result and the second decoding result, and then, based on the scores of each decoding path after weighted summation, select one or more decoding paths with the highest scores as the finally determined decoding result of the speech to be recognized.

[0150] Among them, the weight of the score of the decoding path in the first decoding result is between 0.1 and 0.3. Correspondingly, the weight of the score of the decoding path in the second decoding result is between 0.7 and 0.9, that is, the decoding path output by the decoder accounts for a relatively large proportion.

[0151] As can be seen from the above introduction, the speech recognition method proposed in the embodiments of the present application not only regularizes the model training to ensure the monotonicity of the model attention mechanism. At the same time, in the speech recognition application, fusing the recognition results at the encoding end and the recognition results at the decoding end also plays a role in regularizing the recognition process, which can further improve the speech recognition efficiency and accuracy.

[0152] As already described in the above embodiments, the speech recognition method proposed in the embodiments of the present application is applied to an end-to-end speech recognition model based on the attention mechanism. The speech recognition model includes an encoder and a decoder, and may also include a regularization module, as specifically introduced in the above embodiments.

[0153] Therefore, when the speech recognition method proposed in the embodiments of the present application is actually executed, specifically, the acoustic features of the speech to be recognized can be input into a pre-trained speech recognition model, so that the speech recognition model obtains the encoded features processed by its encoder for the acoustic features of the speech to be recognized, and determines the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

[0154] As a preferred implementation manner, the above speech recognition model is trained by the training method shown in the following steps B1 - B5:

[0155] B1. Input the acoustic features of the speech sample into the encoder to obtain the encoded features output by the encoder, and use the decoder to decode the encoded features to obtain decoded features.

[0156] B2. Determine the attention coefficients of each frame of encoded features of the speech sample for the decoding result of the speech sample according to the encoded features and the decoded features.

[0157] B3. Determine the first recognition result of the speech sample according to the encoded features of the speech sample and the attention coefficients, and determine the first loss function by comparing the first recognition result with the label of the speech sample.

[0158] B4. Determine the second recognition result of the speech sample according to the decoded features of the speech sample, and determine the second loss function by comparing the second recognition result with the label of the speech sample.

[0159] Specifically, the above processes of obtaining encoding features and decoding features, calculating attention coefficients, as well as the calculation processes of the first recognition result, the second recognition result, the first loss function, and the second loss function can all be executed with reference to the introduction in the above embodiments. For example, it can refer to the corresponding content of the training process of the encoder introduced in the above embodiments, which will not be elaborated here.

[0160] It should be particularly noted that after obtaining the above first loss function and second loss function, the embodiments of the present application train the speech recognition model by performing the following step B5:

[0161] B5. Optimize the parameters of the encoder and the decoder by using the first loss function and the second loss function.

[0162] Specifically, assume that the first loss function is L E , and the second loss function is L D . Weighted sum the first loss function L E and the second loss function L D to obtain the combined loss function L:

[0163] L = λL E + (1 - λ)L D

[0164] Where λ takes a value around 0.3.

[0165] Then, use the above combined loss function L to train the encoder of the speech recognition model. It can be known from referring to the introduction in the above embodiments that the combined loss function L contains the attention coefficient information of the recognition result of the speech sample to each frame of encoding features of the speech sample. Using this combined loss function to optimize the encoder parameters can regularize the encoder, which is beneficial to promoting the monotonicity of the encoder attention mechanism, improving the encoding accuracy of the encoding feature frames concerned by the recognition result, and further enabling a more accurate recognition result to be obtained based on the encoding features output by the encoder.

[0166] Meanwhile, optimize the parameters of the decoder by using the second loss function L D , or optimize the parameters of the decoder by using the second loss function L D and the above combined loss function L.

[0167] Specifically, optimizing the decoder parameters by using the second loss function L D is a conventional model training scheme.

[0168] In the embodiments of the present application, the second loss function and the joint loss function can also be combined to optimize the parameters of the decoder. The applications of these two loss functions can be sequential. For example, the second loss function can be first used to optimize the parameters of the decoder, and then the joint loss function can be used to optimize the parameters of the decoder.

[0169] Since the joint loss function contains the attention coefficient information of the recognition result for each frame of encoded features, using the joint loss function to optimize the parameters of the decoder is conducive to a certain extent to prompting the decoder to focus the decoding attention on the correct encoded features, so a more accurate decoding result can be obtained.

[0170] On the other hand, since the joint loss function contains the first loss function, the application of the first loss function may also affect the optimization direction of the second loss function for the decoder parameters, thus forming mutual interference and unable to achieve a better parameter optimization effect.

[0171] In view of the above situation, in the actual training process, according to the actual effect of the speech recognition result, it can be selected to use the second loss function to optimize the parameters of the decoder, or to use the second loss function and the joint loss function to optimize the parameters of the decoder.

[0172] Corresponding to the above speech recognition method, an embodiment of the present application also proposes a speech recognition device. Refer to Figure 7 As shown, the device includes:

[0173] An encoding unit 100, configured to obtain the encoded features obtained by the encoder encoding the acoustic features of the speech to be recognized;

[0174] Wherein, the encoder is trained according to the first recognition result of the speech sample and the text label of the speech sample. The first recognition result of the speech sample is determined according to the encoded features obtained by the encoder encoding the acoustic features of the speech sample and the attention coefficients of the recognition result of the speech sample for each frame of encoded features output by the encoder;

[0175] A decoding unit 110, configured to determine the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

[0176] As an optional implementation manner, determining the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized includes:

[0177] Decoding the encoded features of the speech to be recognized to obtain the decoded features of the speech to be recognized;

[0178] Determine the attention coefficients of the recognition result of the speech to be recognized for each frame of the encoded features of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized;

[0179] Determine the speech recognition result of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized, and the attention coefficients of the recognition result of the speech to be recognized for each frame of the encoded features of the speech to be recognized.

[0180] As an optional implementation manner, the encoder encodes the acoustic features of the speech to be recognized to obtain encoded features, including:

[0181] Perform encoding processing based on the self-attention mechanism on each frame of the acoustic features of the speech to be recognized, so as to obtain the encoded features of the speech to be recognized;

[0182] Among them, performing encoding processing based on the self-attention mechanism on any frame of acoustic features includes: encoding the frame of acoustic features according to an acoustic feature sequence of a set length including the frame of acoustic features to obtain encoded features corresponding to the frame of acoustic features.

[0183] As an optional implementation manner, decoding the encoded features of the speech to be recognized to obtain the decoded features of the speech to be recognized includes:

[0184] Decode each frame of the encoded features of the speech to be recognized and the decoded features corresponding to the recognized result of the speech to be recognized to obtain the decoded features of the speech to be recognized;

[0185] Among them, the recognized result of the speech to be recognized is the recognition result of the speech to be recognized obtained before the current moment.

[0186] As an optional implementation manner, determining the speech recognition result of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized, and the attention coefficients of the recognition result of the speech to be recognized for each frame of the encoded features of the speech to be recognized includes:

[0187] Determine the first decoding result of the speech to be recognized according to each frame of the encoded features of the speech to be recognized and the attention coefficients of the recognition result of the speech to be recognized for each frame of the encoded features of the speech to be recognized;

[0188] Determine the second decoding result of the speech to be recognized according to the decoded features of the speech to be recognized;

[0189] Determine the speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result.

[0190] As an alternative implementation, determining a first decoding result of the speech to be recognized for attention coefficients of the encoded features of each frame of the speech to be recognized, based on the encoded features of each frame of the speech to be recognized and a decoding result of the speech to be recognized, includes:

[0191] Decoding the encoded features of each frame of the speech to be recognized respectively to obtain decoding results corresponding to the encoded features of each frame;

[0192] Using the attention coefficients of the encoded features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized as weights, weighting the decoding results corresponding to the encoded features of each frame to obtain the first decoding result of the speech to be recognized.

[0193] As an alternative implementation, the first decoding result and the second decoding result respectively include multiple decoding paths;

[0194] Determining a speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result includes:

[0195] Performing weighted summation on scores of each decoding path of the first decoding result and the second decoding result, and determining the speech recognition result of the speech to be recognized from the decoding paths after the weighted summation of scores.

[0196] As an alternative implementation, the training process of the encoder includes:

[0197] Obtaining the encoded features obtained by the encoder encoding the acoustic features of a speech sample, and obtaining the decoded features obtained by decoding the encoded features of the speech sample;

[0198] Determining the attention coefficients of the recognition result of the speech sample with respect to the encoded features of each frame of the speech sample according to the encoded features and decoded features of the speech sample;

[0199] Determining a first recognition result of the speech sample according to the encoded features of the speech sample and the attention coefficients;

[0200] Optimizing the parameters of the encoder by comparing the first recognition result with the label of the speech sample.

[0201] As an alternative implementation, obtaining the encoded features obtained by the encoder processing the acoustic features of the speech to be recognized, and determining the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized, includes:

[0202] Input the acoustic features of the speech to be recognized into a pre-trained speech recognition model, so that the speech recognition model obtains the encoded features obtained by its encoder processing the acoustic features of the speech to be recognized, and determine the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

[0203] As an alternative implementation, the speech recognition model includes an encoder and a decoder. The training process of the speech recognition model includes:

[0204] Input the acoustic features of the speech sample into the encoder to obtain the encoded features output by the encoder, and use the decoder to decode the encoded features to obtain the decoded features;

[0205] According to the encoded features and the decoded features, determine the attention coefficients of the decoded result of the speech sample for each frame of encoded features of the speech sample;

[0206] According to the encoded features of the speech sample and the attention coefficients, determine the first recognition result of the speech sample, and determine the first loss function by comparing the first recognition result with the label of the speech sample;

[0207] According to the decoded features of the speech sample, determine the second recognition result of the speech sample, and determine the second loss function by comparing the second recognition result with the label of the speech sample;

[0208] Use the first loss function and the second loss function to optimize the parameters of the encoder and the decoder.

[0209] As an alternative implementation, using the first loss function and the second loss function to optimize the parameters of the encoder and the decoder includes:

[0210] Perform weighted summation on the first loss function and the second loss function to obtain a joint loss function;

[0211] Use the joint loss function to optimize the parameters of the encoder;

[0212] And,

[0213] Use the second loss function, or use the second loss function and the joint loss function to optimize the parameters of the decoder.

[0214] Specifically, for the specific working content of each part of the above speech recognition device, please refer to the corresponding content in the embodiments of the above speech recognition method, which will not be repeated here.

[0215] Another embodiment of the present application further provides a voice recognition device. Refer to Figure 8 As shown in

[0216] a memory 200 and a processor 210;

[0217] wherein, the memory 200 is connected to the processor 210 and is used for storing programs;

[0218] the processor 210 is configured to implement the voice recognition method disclosed in any of the above embodiments by running the programs stored in the memory 200.

[0219] Specifically, the above voice recognition device may further include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0220] The processor 210, the memory 200, the communication interface 220, the input device 230, and the output device 240 are interconnected via the bus. Among them:

[0221] The bus may include a path for transmitting information between various components of the computer system.

[0222] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0223] The processor 210 may include a main processor and may also include a baseband chip, a modem, etc.

[0224] The memory 200 stores programs for implementing the technical solution of the present invention and may also store an operating system and other key services. Specifically, the program may include program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash memory, etc.

[0225] The input device 230 may include devices for receiving data and information input by the user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0226] The output device 240 may include devices for allowing information to be output to the user, such as a display screen, a printer, a speaker, etc.

[0227] The communication interface 220 may include devices of any transceiver type for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0228] The processor 210 executes the programs stored in the memory 200 and calls other devices, and can be used to implement the various steps of the voice recognition method provided in the above embodiments of the present application.

[0229] Another embodiment of the present application also provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, it implements the various steps of the voice recognition method provided in the above embodiments of the present application.

[0230] Specifically, for the specific working content of each part of the above voice recognition device, and for the specific processing content when the computer program on the above storage medium is run by a processor, reference can be made to the content of the various embodiments of the above voice recognition method, which will not be elaborated here.

[0231] For the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0232] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0233] The steps in the methods of the various embodiments of the present application can be adjusted, combined, and deleted according to actual needs, and the technical features recorded in the various embodiments can be replaced or combined.

[0234] In the embodiments of the present application, the modules and sub-modules in the device and the terminal can be combined, divided, and deleted according to actual needs.

[0235] In several embodiments provided in the present application, it should be understood that the disclosed terminal, device, and method can be implemented in other ways. For example, the terminal embodiments described above are only illustrative. For example, the division of modules or sub-modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or module can be in electrical, mechanical, or other forms.

[0236] The modules or sub-modules described as separate components may or may not be physically separated. The components as modules or sub-modules may or may not be physical modules or sub-modules, that is, they can be located in one place, or can be distributed to multiple network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, in each embodiment of the present application, each functional module or sub-module can be integrated in a processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The above-mentioned integrated modules or sub-modules can be implemented in the form of hardware or in the form of software functional modules or sub-modules.

[0238] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0239] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software units executed by a processor, or a combination of the two. The software units can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0240] Finally, it should also be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0241] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A voice recognition method, characterized in that, Including: Obtaining the encoded features obtained by encoding the acoustic features of the speech to be recognized by an encoder; Wherein, the encoder is trained according to the first recognition result of the speech sample and the text label of the speech sample. The first recognition result of the speech sample is determined according to the encoded features obtained by encoding the acoustic features of the speech sample by the encoder and the attention coefficients of each frame of encoded features output by the encoder and the decoding result of the speech sample. Determining the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

2. The method according to claim 1, wherein Determining the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized includes: Obtaining the decoded features of the speech to be recognized by decoding the encoded features of the speech to be recognized; Determining the attention coefficients of the recognition result of the speech to be recognized for each frame of encoded features of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized; Determining the speech recognition result of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized and the attention coefficients of the recognition result of the speech to be recognized for each frame of encoded features of the speech to be recognized.

3. The method according to claim 1 or 2, characterized in that, The encoder encodes the acoustic features of the speech to be recognized to obtain encoded features, including: Performing encoding processing based on the self-attention mechanism on each frame of acoustic features of the speech to be recognized respectively, so as to obtain the encoded features of the speech to be recognized; Wherein, performing encoding processing based on the self-attention mechanism on any frame of acoustic features includes: encoding the frame of acoustic features according to the acoustic feature sequence with a set length including the frame of acoustic features to obtain the encoded features corresponding to the frame of acoustic features.

4. The method according to claim 2, characterized in that, Obtaining the decoded features of the speech to be recognized by decoding the encoded features of the speech to be recognized, including: Decoding each frame of encoded features of the speech to be recognized and the decoded features corresponding to the recognized result of the speech to be recognized to obtain the decoded features of the speech to be recognized; Wherein, the recognized result of the speech to be recognized is the recognition result of the speech to be recognized obtained before the current moment.

5. The method according to claim 2, wherein Determining the speech recognition result of the speech to be recognized according to the encoded features and decoded features of the speech to be recognized and the attention coefficients of the recognition result of the speech to be recognized for each frame of encoded features of the speech to be recognized includes: Determining the first decoding result of the speech to be recognized according to each frame of encoded features of the speech to be recognized and the attention coefficients of the recognition result of the speech to be recognized for each frame of encoded features of the speech to be recognized; Determining the second decoding result of the speech to be recognized according to the decoded features of the speech to be recognized; Determining the speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result.

6. The method according to claim 5, characterized in that Determining the first decoding result of the speech to be recognized according to each frame of encoded features of the speech to be recognized and the attention coefficients of the decoding result of the speech to be recognized for each frame of encoded features of the speech to be recognized includes: Decode the encoded features of each frame of the speech to be recognized respectively to obtain the decoding results corresponding to the encoded features of each frame; Use the attention coefficients of the encoded features of each frame of the speech to be recognized with respect to the recognition result of the speech to be recognized as weights to weight the decoding results corresponding to the encoded features of each frame, and obtain the first decoding result of the speech to be recognized.

7. The method according to claim 5, wherein The first decoding result and the second decoding result each include a plurality of decoding paths; Determine the speech recognition result of the speech to be recognized according to the first decoding result and the second decoding result, including: Perform weighted summation on the scores of each decoding path of the first decoding result and the second decoding result, and determine the speech recognition result of the speech to be recognized from the decoding paths after the weighted summation of the scores.

8. The method according to claim 1, wherein The training process of the encoder includes: Obtain the encoded features obtained by the encoder encoding the acoustic features of the speech sample, and obtain the decoded features obtained by decoding the encoded features of the speech sample; Determine the attention coefficients of the decoded result of the speech sample with respect to the encoded features of each frame of the speech sample according to the encoded features and the decoded features of the speech sample; Determine the first recognition result of the speech sample according to the encoded features of the speech sample and the attention coefficients; Optimize the parameters of the encoder by comparing the first recognition result with the label of the speech sample.

9. The method according to claim 1, characterized in that Obtain the encoded features obtained by the encoder processing the acoustic features of the speech to be recognized, and determine the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized, including: Input the acoustic features of the speech to be recognized into a pre-trained speech recognition model, so that the speech recognition model obtains the encoded features obtained by its encoder processing the acoustic features of the speech to be recognized, and determine the speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

10. The method according to claim 9, wherein The speech recognition model includes an encoder and a decoder. The training process of the speech recognition model includes: Input the acoustic features of the speech sample into the encoder to obtain the encoded features output by the encoder, and use the decoder to decode the encoded features to obtain the decoded features; Determine the attention coefficients of the decoded result of the speech sample with respect to the encoded features of each frame of the speech sample according to the encoded features and the decoded features; Determine the first recognition result of the speech sample according to the encoded features of the speech sample and the attention coefficients, and determine the first loss function by comparing the first recognition result with the label of the speech sample; Determine the second recognition result of the speech sample according to the decoded features of the speech sample, and determine the second loss function by comparing the second recognition result with the label of the speech sample; Optimize the parameters of the encoder and the decoder using the first loss function and the second loss function.

11. The method according to claim 10, characterized in that, Optimize the parameters of the encoder and the decoder using the first loss function and the second loss function, including: Perform a weighted sum of the first loss function and the second loss function to obtain a joint loss function; Use the joint loss function to optimize the parameters of the encoder; And, Use the second loss function, or use the second loss function and the joint loss function, to optimize the parameters of the decoder.

12. A voice recognition device, characterized in that, Comprising: An encoding unit, configured to obtain encoded features obtained by encoding the acoustic features of the speech to be recognized by an encoder; Wherein, the encoder is trained according to the first recognition result of the speech sample and the text label of the speech sample, and the first recognition result of the speech sample is determined according to the encoded features obtained by encoding the acoustic features of the speech sample by the encoder and the attention coefficients of each frame of encoded features output by the encoder and the decoding result of the speech sample; A decoding unit, configured to determine a speech recognition result of the speech to be recognized according to the encoded features of the speech to be recognized.

13. A voice recognition device, characterized in that, Comprising: A memory and a processor; Wherein, the memory is connected to the processor and is configured to store programs; The processor is configured to implement the speech recognition method according to any one of claims 1 to 11 by running the programs in the memory.

14. A storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is run by a processor, the speech recognition method according to any one of claims 1 to 11 is implemented.

Images