Speech recognition method, device, storage medium and equipment

Abstract

The present application discloses a speech recognition method, apparatus, storage medium and equipment, the method comprising: first obtaining a target speech to be recognized, then inputting the target speech into a pre-constructed speech recognition model, and recognizing and obtaining a probability distribution of characters corresponding to the target speech; wherein the speech recognition model is a neural network model obtained by training a neural network model based on an attention mechanism in combination with an RnnT loss constraint function; then, the target speech can be recognized according to the probability distribution of characters corresponding to the target speech, and a recognition result of the target speech can be obtained. Since the pre-constructed speech recognition model in the present application is obtained by training a neural network model based on an attention mechanism in combination with an RnnT loss constraint function, the RnnT loss constraint function can be used to explicitly perform alignment constraints between input features and output characters during model training, thereby improving the generalization ability of the model and also improving the speech recognition effect and accuracy.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a speech recognition method, apparatus, storage medium and device. Background Art

[0002] With continuous breakthroughs in artificial intelligence (AI) technology and the increasing popularity of various smart devices, human-computer interaction is becoming increasingly common in our daily lives and work. As one of the most convenient and efficient methods of interaction, voice recognition has become a crucial component of human-computer interaction. With the increasing number of voice users, traditional speech recognition methods that combine acoustic, pronunciation, and language models are no longer able to achieve high recognition accuracy. However, by leveraging the power of internet big data, end-to-end speech recognition systems that utilize a combination of acoustic, pronunciation, and language models have gained widespread adoption.

[0003] Currently, the most effective end-to-end speech recognition model is mainly based on the neural network model structure of the attention mechanism. This structure can convert the speech sequence word by word into the corresponding recognition result sequence. Figure 1 As shown in the figure, the structure consists of three parts: encoder module, decoder module and attention module. The attention module can convert the high-order features extracted by the encoder into a probability distribution of a set of character sequences, and then obtain the recognition sequence through this probability distribution. To further ensure the effect, the recognition result is re-applied through the traditional language model to obtain the final recognition result. However, in the existing Figure 1 The training process of the end-to-end speech recognition model based on the attention mechanism shown here does not explicitly enforce alignment constraints between input features and output characters. Because this is an end-to-end model, if the constraints are not strong enough during training, the learned patterns will be biased, resulting in insufficient generalization and poor overall speech recognition performance.

[0004] Therefore, how to optimize the end-to-end speech recognition model based on the attention mechanism to obtain more accurate speech recognition results through the optimized model is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to provide a speech recognition method, apparatus, storage medium and equipment that can optimize the end-to-end speech recognition model based on the attention mechanism so as to improve the speech recognition effect and accuracy when performing speech recognition.

[0006] The present invention provides a speech recognition method, including:

[0007] Obtain the target speech to be recognized;

[0008] Input the target speech into a pre-built speech recognition model to identify and obtain the character probability distribution corresponding to the target speech; the speech recognition model is a neural network model trained on the basis of a neural network model based on an attention mechanism and a recursive neural network transformer RnnT loss constraint function;

[0009] The target speech is recognized according to the probability distribution of characters corresponding to the target speech to obtain a recognition result of the target speech.

[0010] In one possible implementation, the speech recognition model is constructed as follows:

[0011] Get sample speech;

[0012] The sample speech and the RnnT loss constraint function are used to train the initial speech recognition model to obtain the speech recognition model.

[0013] In one possible implementation, the initial speech recognition model includes an input layer, an encoder layer, a classification layer, an attention mechanism layer, a pseudo-joint network layer, and an output layer.

[0014] In one possible implementation, inputting the target speech into a pre-built speech recognition model to identify and obtain a probability distribution of characters corresponding to the target speech includes:

[0015] Inputting the acoustic features of the target speech into the encoder layer through the input layer to obtain encoding features;

[0016] Inputting the coding features into the classification layer to obtain an initial probability distribution of characters corresponding to each speech frame in the target speech;

[0017] Inputting the encoded features into the attention mechanism layer to obtain an attention coefficient for each speech frame in the target speech;

[0018] Inputting the initial probability distribution of characters and the attention coefficient corresponding to each speech frame in the target speech into the pseudo-joint network layer to obtain the intermediate probability distribution of characters corresponding to each speech frame in the target speech; and concatenating the intermediate probability distribution of characters with the probability of a one-dimensional empty character to obtain an updated intermediate probability distribution;

[0019] The updated intermediate probability distribution is constrained by using the RnnT loss constraint function to obtain a final probability distribution of characters corresponding to each speech frame in the target speech.

[0020] In one possible implementation, inputting the encoded features into the attention mechanism layer to obtain an attention coefficient for each speech frame in the target speech includes:

[0021] Decoding the encoded features to obtain decoded features;

[0022] The encoding features and the decoding features are input into the attention mechanism layer together, and attention calculation is performed to obtain the attention weighted coefficient of each speech frame in the target speech.

[0023] In a possible implementation, the method further includes:

[0024] Get verification voice;

[0025] Inputting the verification speech into the speech recognition model to obtain a character recognition result corresponding to the verification speech;

[0026] When the character recognition result of the verification speech is inconsistent with the character label result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

[0027] In one possible implementation, the RnnT loss constraint function is used to explicitly constrain the relationship between the acoustic features of the target speech and the characters corresponding to the target speech, so as to improve the generalization ability and recognition effect of the speech recognition model.

[0028] The present application also provides a speech recognition device, including:

[0029] A first acquisition unit, configured to acquire a target speech to be recognized;

[0030] A first recognition unit is configured to input the target speech into a pre-built speech recognition model to identify and obtain a probability distribution of characters corresponding to the target speech; the speech recognition model is a neural network model trained on the basis of an attention-based neural network model and a recursive neural network transformer (RnnT) loss constraint function;

[0031] The second recognition unit is used to recognize the target speech according to the probability distribution of characters corresponding to the target speech to obtain a recognition result of the target speech.

[0032] In a possible implementation, the apparatus further includes:

[0033] A second acquiring unit, configured to acquire a sample speech;

[0034] The training unit is used to train the initial speech recognition model using the sample speech and the RnnT loss constraint function to obtain the speech recognition model.

[0035] In one possible implementation, the initial speech recognition model includes an input layer, an encoder layer, a classification layer, an attention mechanism layer, a pseudo-joint network layer, and an output layer.

[0036] In a possible implementation, the first identification unit includes:

[0037] A first input subunit, configured to input the acoustic features of the target speech into the encoder layer through the input layer to obtain encoded features;

[0038] A second input subunit is configured to input the coding feature into the classification layer to obtain an initial probability distribution of characters corresponding to each speech frame in the target speech;

[0039] A third input subunit is used to input the encoded features into the attention mechanism layer to obtain an attention coefficient for each speech frame in the target speech;

[0040] a fourth input subunit, configured to input the initial probability distribution of characters and the attention coefficient corresponding to each speech frame in the target speech into the pseudo-joint network layer to obtain an intermediate probability distribution of characters corresponding to each speech frame in the target speech; and concatenate the intermediate probability distribution of characters with the probability of a one-dimensional empty character to obtain an updated intermediate probability distribution;

[0041] The constraint subunit is used to constrain the updated intermediate probability distribution using the RnnT loss constraint function to obtain the final probability distribution of the characters corresponding to each speech frame in the target speech.

[0042] In a possible implementation, the third input subunit includes:

[0043] a decoding subunit, configured to decode the encoding feature to obtain a decoding feature;

[0044] The computing subunit is used to input the encoding features and the decoding features into the attention mechanism layer, perform attention calculation, and obtain the attention weighting coefficient of each speech frame in the target speech.

[0045] In a possible implementation, the apparatus further includes:

[0046] A third acquiring unit, configured to acquire a verification voice;

[0047] An input unit, configured to input the verification speech into the speech recognition model to obtain a character recognition result corresponding to the verification speech;

[0048] An updating unit is configured to, when a character recognition result of the verification speech is inconsistent with a character label result corresponding to the verification speech, use the verification speech as the sample speech again to update the speech recognition model.

[0049] In one possible implementation, the RnnT loss constraint function is used to explicitly constrain the relationship between the acoustic features of the target speech and the characters corresponding to the target speech, so as to improve the generalization ability and recognition effect of the speech recognition model.

[0050] The embodiment of the present application also provides a speech recognition device, comprising: a processor, a memory, and a system bus;

[0051] The processor and the memory are connected via the system bus;

[0052] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any one implementation of the above-mentioned speech recognition method.

[0053] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0054] An embodiment of the present application further provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementations of the above-mentioned speech recognition method.

[0055] The embodiment of the present application provides a speech recognition method, apparatus, storage medium and device, which first obtains the target speech to be recognized, and then inputs the target speech into a pre-built speech recognition model to identify the character probability distribution corresponding to the target speech; wherein the speech recognition model is a neural network model obtained by training a recursive neural network converter RnnT loss constraint function based on a neural network model based on an attention mechanism; then, the target speech can be recognized according to the character probability distribution corresponding to the target speech to obtain the recognition result of the target speech. It can be seen that since the pre-built speech recognition model in the embodiment of the present application is obtained by training a recursive neural network converter RnnT loss constraint function based on a neural network model based on an attention mechanism, the RnnT loss constraint function can be used to explicitly make alignment constraints between input features and output characters during model training, thereby realizing the optimization of the end-to-end speech recognition model based on the attention mechanism, improving the generalization ability and recognition effect of the model, and also improving the speech recognition effect and accuracy when performing speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 This is a diagram of the network structure of the current end-to-end speech recognition model based on the attention mechanism;

[0058] Figure 2 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the network structure of the speech recognition model provided in the embodiment of the present application;

[0060] Figure 4 A schematic diagram of the composition of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] With the rapid development of intelligent recognition technology, speech recognition is required in an increasing number of scenarios. In recent years, end-to-end models have replaced the traditional combination of acoustic, pronunciation, and language models with a single neural network. These models, optimized with a unified goal, have greatly simplified the construction and maintenance of speech recognition systems. Furthermore, leveraging the unique advantages of internet big data, end-to-end speech recognition systems that jointly model acoustic, pronunciation, and language models have replaced traditional speech recognition systems and have become the most sought-after speech recognition model framework in recent years.

[0062] Currently, existing end-to-end speech recognition model construction methods generally include the following three methods:

[0063] The first is the Connectionist Temporal Classification (CTC) model. However, due to the conditional independence assumption, the CTC model cannot model the dependencies between model outputs. Moreover, due to the lack of language modeling capabilities, it cannot integrate language models for joint optimization.

[0064] The second is the Recurrent Neural Network Transducer (RnnT) model. Although this model overcomes the shortcomings of the CTC model and enables the recognition model to have language modeling capabilities, facilitating end-to-end joint optimization and real-time speech recognition, it can cause information loss during the recognition process.

[0065] The third is a neural network model based on the attention mechanism. This model simulates the human attention mechanism to determine which parts of the model encoder output need more attention, extract relevant information from it and pass it to the decoder for decoding and output. Since all context information is taken into account, its accuracy is generally higher than the first two recognition models. It is also the best end-to-end speech recognition model currently. The specific model structure is as follows: Figure 1 However, during the training process of this recognition model, there are no explicit alignment constraints between the input features and the output characters. Because it is an end-to-end model, if the constraints are not strong enough during model training, the learned patterns will be biased, resulting in insufficient generalization and poor overall speech recognition performance.

[0066] Therefore, how to optimize the end-to-end speech recognition model based on the attention mechanism to obtain more accurate speech recognition results through the optimized model is a technical problem that needs to be solved urgently.

[0067] In order to solve the above-mentioned defects, the present application provides a speech recognition method, which first obtains the target speech to be recognized, and then inputs the target speech into a pre-built speech recognition model to identify the character probability distribution corresponding to the target speech; wherein the speech recognition model is a neural network model obtained by training the recursive neural network converter RnnT loss constraint function on the basis of a neural network model based on an attention mechanism; then, the target speech can be recognized according to the character probability distribution corresponding to the target speech to obtain the recognition result of the target speech. It can be seen that since the speech recognition model pre-built in the embodiment of the present application is obtained by training the recursive neural network converter RnnT loss constraint function on the basis of a neural network model based on an attention mechanism, the RnnT loss constraint function can be used to explicitly make alignment constraints between the input features and the output characters during model training, thereby realizing the optimization of the end-to-end speech recognition model based on the attention mechanism, improving the generalization ability and recognition effect of the model, and also improving the speech recognition effect and accuracy when performing speech recognition.

[0068] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0069] First embodiment

[0070] See also Figure 2 , is a flow chart of a speech recognition method provided in this embodiment, the method comprising the following steps:

[0071] S201: Acquire target speech to be recognized.

[0072] In this embodiment, any speech recognized by this embodiment is defined as a target speech. Furthermore, this embodiment does not limit the language type of the target speech; for example, the target speech can be Chinese or English. Furthermore, this embodiment does not limit the length of the target speech; for example, the target speech can be a single sentence or multiple sentences.

[0073] It is understandable that the target voice can be obtained through recording or other means according to actual needs. For example, the telephone conversation voice or meeting recording in people's daily life can be used as the target voice. After obtaining the target voice, the present embodiment can be used to realize the recognition of the target voice.

[0074] S202: Input the target speech into a pre-built speech recognition model to identify the character probability distribution corresponding to the target speech; wherein the speech recognition model is a neural network model obtained by training the recursive neural network converter RnnT loss constraint function based on the neural network model based on the attention mechanism.

[0075] In this embodiment, after the target speech to be recognized is obtained through step S201, in order to quickly and accurately identify the character recognition result of the target speech, the target speech can be further input into a pre-built speech recognition model, thereby identifying the character probability distribution corresponding to the target speech, so as to execute the subsequent step S203. It should be noted that the specific format of the character probability distribution corresponding to the target speech can be set according to actual conditions (such as the architecture of the selected recognition network model or the language to which the character belongs, etc.), and this embodiment does not limit this. For example, when the character recognition result corresponding to the target speech is Chinese, and the number of all characters in the Chinese dictionary is M (M is a positive integer greater than 1), the character probability distribution corresponding to the target speech can be an M-dimensional vector, etc.

[0076] Among them, the speech recognition model is a neural network model obtained by training on the basis of a neural network model based on the attention mechanism and combined with the RnnT loss constraint function. Therefore, in the existing attention model training process, the RnnT loss constraint function can be used to explicitly perform alignment constraints between input features and output characters to improve the generalization ability and recognition effect of the model, so that the speech recognition effect and accuracy can also be improved.

[0077] like Figure 3 As shown in the figure, the speech recognition model includes an input layer, an encoder layer (Encoder), a classification layer (for outputting frame-level probability (Frame Prob)), an attention mechanism layer (Attention), a pseudo joint network layer (Pseudo Joint Net), and an output layer.

[0078] On this basis, an optional implementation manner is that the specific implementation process of step S202 may include the following steps AE:

[0079] Step A: Input the acoustic features of the target speech into the encoder layer through the input layer to obtain the encoded features.

[0080] In this implementation, after obtaining the target speech to be recognized, it is first necessary to perform frame processing on the target speech to obtain the corresponding speech frame sequence, and then pre-emphasize the framed speech frame sequence; and then extract the acoustic features of each speech frame in turn, where the acoustic feature refers to the feature data used to characterize the voiceprint information of the corresponding speech frame, for example, it can be Mel-scale Frequency Cepstral Coefficients (MFCC) features or Log Mel filterbank (FBANK) features, etc.

[0081] It should be noted that the embodiments of the present application do not limit the method for extracting the acoustic features of the target speech, nor do they limit the specific extraction process. An appropriate extraction method can be selected according to actual conditions, and corresponding feature extraction operations can be performed.

[0082] Specifically, the total number of audio frames contained in the target speech can be defined as T, and the acoustic feature sequence of the target speech (here defined as x) can be expressed as the following formula:

[0083] x=(x1,x2,...,x T ) (1)

[0084] Furthermore, the acoustic feature x of the target speech can be used as an audio feature and input into the encoder layer through the input layer to obtain the encoded feature (here defined as h), which is specifically expressed as follows:

[0085] h=En(x)=(h1,h2,...,h T ) (2)

[0086] Step B: Input the encoded features into the classification layer to obtain the initial probability distribution of characters corresponding to each speech frame in the target speech.

[0087] In this implementation, after obtaining the encoding feature h corresponding to the target speech through step A, it can be further input into a classification layer (such as Dnn) connected to the back end of the encoder to obtain the initial probability distribution of characters corresponding to each speech frame in the target speech (defined here as g), which is specifically expressed as follows:

[0088] g=soft max(Dnn(h))=(g1,g2,...,g T ) (3)

[0089] Among them, g represents the probability of the character corresponding to each speech frame when directly classified through the encoder layer output; g1,g2,...,g T The dimensions of each are M, where M refers to the total number of characters in the Chinese dictionary when the character recognition result corresponding to the target speech is Chinese; softmax is a normalization function.

[0090] Step C: Input the encoded features into the attention mechanism layer to obtain the attention coefficient of each speech frame in the target speech.

[0091] In this implementation, after obtaining the encoding feature h corresponding to the target speech in step A, it can be further input into the attention mechanism layer connected to the back end of the encoder to obtain the attention coefficient (here defined as a) for each speech frame in the target speech, which is specifically expressed as follows:

[0092] a u =(a u1 ,a u2 ,...,a uT ) (4)

[0093] Among them, a u It represents the attention coefficient when decoding the u-th character. This coefficient can be obtained by, but not limited to, additive or multiplicative attention.

[0094] An optional implementation method is that the implementation process of this step C may include: first decoding the encoded features to obtain decoded features, and then inputting the encoded features and decoded features into the attention mechanism layer together to perform attention calculation to obtain the attention weighted coefficient of each speech frame in the target speech.

[0095] In this implementation, after decoding the encoded feature h and obtaining the decoded feature (defined as s here), the two can be used as the input of the attention mechanism layer to perform attention calculation and obtain the normalized attention weight coefficient a. The specific calculation formula is as follows:

[0096] a u,t =soft max(V T tanh(W s s u +W h h t )) (5)

[0097] Where u represents the decoded u-th character; t represents the t-th frame of the target speech; W s 、W h , V represent the weight parameters of the model, V T The superscript T in represents the transposition of the column vector V into a row vector; anh() represents the neural network activation function; and softmax is the normalization function.

[0098] Step D: Input the initial probability distribution of characters and the attention coefficient corresponding to each speech frame in the target speech into the pseudo-joint network layer to obtain the intermediate probability distribution of characters corresponding to each speech frame in the target speech; and concatenate the intermediate probability distribution of characters with the probability of a one-dimensional empty character to obtain an updated intermediate probability distribution.

[0099] In this implementation, after obtaining the initial probability distribution g of the characters corresponding to each speech frame in the target speech through step B and the attention coefficient a of each speech frame in the target speech through step C, the two can be further input into the pseudo-joint network layer to obtain the intermediate probability distribution of the characters corresponding to each speech frame in the target speech (defined as p here), which is specifically expressed as follows:

[0100] p u =g·a u =(g1,g2,...,g T )·(a u1 ,a u2 ,...,a uT ) (6)

[0101] Among them, a ut represents the probability of decoding a non-blank character in the current t-th frame when the u-th character is decoded, and t is any positive integer between 1 and T; p ut It means that when a non-blank character is decoded in the current t-th frame, the encoder layer outputs the corresponding probability of each character, and the value of t is still any positive integer between 1 and T.

[0102] On this basis, this application can be applied to 1-aut It means that when the uth character is decoded, the probability of predicting a blank character (blank) in the tth frame is obtained. Therefore, the intermediate probability distribution p of the character can be u and the probability of a one-dimensional empty character 1-a u After splicing, the updated intermediate probability distribution is obtained as in, It can be seen that the updated intermediate probability distribution is similar to the RNN-T joint network probability distribution, which is specifically expressed as follows:

[0103]

[0104] Among them, p(m|t,u) represents the updated intermediate probability distribution; softmax is the normalization function.

[0105] Step E: Use the RnnT loss constraint function to constrain the updated intermediate probability distribution to obtain the final probability distribution of the characters corresponding to each speech frame in the target speech.

[0106] In this implementation, after obtaining the updated intermediate probability distribution p(m|t,u) of the characters corresponding to each speech frame in the target speech through step D, the RnnT loss constraint function can be further used to constrain the updated intermediate probability distribution, thereby obtaining the final probability distribution of the characters corresponding to each speech frame in the target speech. The RnnT loss constraint function is specifically expressed as follows:

[0107] L=-ln(p(m|t,u)) (8)

[0108] Here, L represents the RnnT loss constraint function, which is used to explicitly constrain the relationship between the acoustic features of the target speech and the corresponding characters of the target speech, thereby improving the generalization ability and recognition performance of the speech recognition model. It can be understood that when the value of L is minimized, the constraint effect is the best.

[0109] Next, this embodiment will introduce the construction process of the speech recognition model. Among them, an optional implementation method is that the construction process of the speech recognition model can specifically include: first obtaining sample speech, and then using the sample speech and RnnT loss constraint function to train the initial speech recognition model to obtain the speech recognition model.

[0110] Specifically, in this implementation, in order to build a speech recognition model, a lot of preparatory work needs to be done in advance. First, a large amount of speech data needs to be collected as sample speech to constitute the model training data. For example, a large amount of recording data can be collected in advance, such as the voices of each contestant in a recitation competition, or conference recordings, etc., which can be used as sample speech to constitute the model training data, and the text recognition results corresponding to these sample speech can be manually marked. Then, the initial speech recognition model can be trained based on these sample speech and the text recognition results corresponding to the sample speech and the RnnT loss constraint function, and then the speech recognition model can be generated.

[0111] Among them, an optional implementation method is that the initial reception service process recognition model can be (but not limited to) a neural network model including an input layer, an encoder layer, a classification layer, an attention mechanism layer, a simulated joint network layer, and an output layer.

[0112] Specifically, when conducting model training, a sample speech can be extracted from the training data in turn as the model input, and the corresponding text recognition result can be used as the output. Multiple rounds of model training can be performed, and the text recognition results obtained in each round of training can be compared with the corresponding manual annotation results. The model parameters are updated according to the difference between the two until the preset conditions are met. For example, if the value of the RnnT loss constraint function is very small, the update of the model parameters is stopped, the training of the speech recognition model is completed, and a trained speech recognition model is generated.

[0113] On this basis, after generating a speech recognition model based on sample speech training, the generated speech recognition model can be further verified using verification speech. The specific verification process can include the following steps (1)-(3):

[0114] Step (1): Obtain verification voice.

[0115] In this embodiment, in order to verify the speech recognition model, it is first necessary to obtain verification speech. For example, with the user's permission, 1,000 speech sounds can be collected from the user's daily conversations as verification speech sounds. The verification speech refers to speech information that can be used to verify the speech recognition model. After obtaining these verification speech sounds and the character recognition labels corresponding to each verification speech sound, the subsequent step (2) can be continued.

[0116] Step (2): input the verification speech into the speech recognition model to obtain the character recognition result corresponding to the verification speech.

[0117] After obtaining the verification voice through step (1), the verification voice can be further input into the voice recognition model to obtain the character probability distribution corresponding to the verification voice, and then the character recognition result corresponding to the verification voice is obtained based on the character probability distribution to execute the subsequent step (3).

[0118] Step (3): When the character recognition result of the verification speech is inconsistent with the character label result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

[0119] After obtaining the character recognition result of the verification speech through step (2), if the character recognition result of the verification speech is inconsistent with the actual recognition result corresponding to the verification speech (such as the manually annotated character text marking result), the verification speech can be used as a sample speech again to update the parameters of the speech recognition model.

[0120] Through the above embodiments, the speech recognition model can be effectively verified using the verification voice. When the character recognition result of the verification voice is inconsistent with the actual recognition result corresponding to the verification voice (such as the manually marked character marking result), the speech recognition model can be adjusted and updated in time, which helps to improve the recognition precision and accuracy of the recognition model.

[0121] S203: Recognize the target speech according to the probability distribution of characters corresponding to the target speech to obtain a recognition result of the target speech.

[0122] In this embodiment, the target speech is input into the speech recognition model in step S202. After the character probability distribution corresponding to the target speech is obtained, the character probability distribution corresponding to the target speech can be further analyzed, and based on the analysis results, the target speech can be recognized to obtain the recognition result of the target speech. For example, the character with the highest probability can be selected from the character probability distribution corresponding to the target speech to form the recognition text corresponding to the target speech as the final recognition result. This can ensure a good recognition effect while reducing the delay of the recognition process and ensuring the real-time performance of the recognition process.

[0123] In summary, the present embodiment provides a speech recognition method, which first obtains the target speech to be recognized, and then inputs the target speech into a pre-built speech recognition model to identify the character probability distribution corresponding to the target speech; wherein the speech recognition model is a neural network model obtained by training the recursive neural network converter RnnT loss constraint function on the basis of a neural network model based on an attention mechanism; then, the target speech can be recognized according to the character probability distribution corresponding to the target speech to obtain the recognition result of the target speech. It can be seen that since the speech recognition model pre-built in the embodiment of the present application is obtained by training the recursive neural network converter RnnT loss constraint function on the basis of a neural network model based on an attention mechanism, the RnnT loss constraint function can be used to explicitly make alignment constraints between the input features and the output characters during model training, thereby realizing the optimization of the end-to-end speech recognition model based on the attention mechanism, improving the generalization ability and recognition effect of the model, and also improving the speech recognition effect and accuracy when performing speech recognition.

[0124] Second embodiment

[0125] This embodiment will introduce a speech recognition device. For related content, please refer to the above method embodiment.

[0126] See also Figure 4 , is a schematic diagram of the composition of a speech recognition device provided in this embodiment, the device 400 includes:

[0127] The first acquisition unit 401 is used to acquire the target speech to be recognized;

[0128] The first recognition unit 402 is configured to input the target speech into a pre-built speech recognition model to identify and obtain a probability distribution of characters corresponding to the target speech; the speech recognition model is a neural network model trained based on an attention mechanism and a recursive neural network transformer (RnnT) loss constraint function;

[0129] The second recognition unit 403 is configured to recognize the target speech according to the probability distribution of characters corresponding to the target speech, and obtain a recognition result of the target speech.

[0130] In one implementation of this embodiment, the apparatus further includes:

[0131] A second acquiring unit, configured to acquire a sample speech;

[0132] The training unit is used to train the initial speech recognition model using the sample speech and the RnnT loss constraint function to obtain the speech recognition model.

[0133] In one implementation of this embodiment, the initial speech recognition model includes an input layer, an encoder layer, a classification layer, an attention mechanism layer, a pseudo-joint network layer, and an output layer.

[0134] In one implementation of this embodiment, the first identification unit 402 includes:

[0135] A first input subunit, configured to input the acoustic features of the target speech into the encoder layer through the input layer to obtain encoded features;

[0136] A second input subunit is configured to input the coding feature into the classification layer to obtain an initial probability distribution of characters corresponding to each speech frame in the target speech;

[0137] A third input subunit is used to input the encoded features into the attention mechanism layer to obtain an attention coefficient for each speech frame in the target speech;

[0138] a fourth input subunit, configured to input the initial probability distribution of characters and the attention coefficient corresponding to each speech frame in the target speech into the pseudo-joint network layer to obtain an intermediate probability distribution of characters corresponding to each speech frame in the target speech; and concatenate the intermediate probability distribution of characters with the probability of a one-dimensional empty character to obtain an updated intermediate probability distribution;

[0139] The constraint subunit is used to constrain the updated intermediate probability distribution using the RnnT loss constraint function to obtain the final probability distribution of the characters corresponding to each speech frame in the target speech.

[0140] In one implementation of this embodiment, the third input subunit includes:

[0141] a decoding subunit, configured to decode the encoding feature to obtain a decoding feature;

[0142] The computing subunit is used to input the encoding features and the decoding features into the attention mechanism layer, perform attention calculation, and obtain the attention weighting coefficient of each speech frame in the target speech.

[0143] In one implementation of this embodiment, the apparatus further includes:

[0144] A third acquiring unit, configured to acquire a verification voice;

[0145] An input unit, configured to input the verification speech into the speech recognition model to obtain a character recognition result corresponding to the verification speech;

[0146] An updating unit is configured to, when a character recognition result of the verification speech is inconsistent with a character label result corresponding to the verification speech, use the verification speech as the sample speech again to update the speech recognition model.

[0147] In one implementation of this embodiment, the RnnT loss constraint function is used to explicitly constrain the relationship between the acoustic features of the target speech and the characters corresponding to the target speech, so as to improve the generalization ability and recognition effect of the speech recognition model.

[0148] Furthermore, an embodiment of the present application also provides a speech recognition device, comprising: a processor, a memory, and a system bus;

[0149] The processor and the memory are connected via the system bus;

[0150] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any one of the implementation methods of the above-mentioned speech recognition method.

[0151] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0152] Furthermore, an embodiment of the present application also provides a computer program product, which, when running on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech recognition method.

[0153] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0154] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0155] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0156] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: include: Obtain the target speech to be recognized; Inputting the target speech into a pre-built speech recognition model to identify and obtain the character probability distribution corresponding to the target speech; The speech recognition model is a neural network model obtained by training based on a neural network model based on an attention mechanism and a recursive neural network transformer RnnT loss constraint function; the speech recognition model includes an input layer, an encoder layer, a classification layer, an attention mechanism layer, a pseudo-joint network layer, and an output layer; Recognizing the target speech according to the probability distribution of characters corresponding to the target speech to obtain a recognition result of the target speech; Inputting the target speech into a pre-built speech recognition model to identify and obtain a probability distribution of characters corresponding to the target speech includes: Inputting the acoustic features of the target speech into the encoder layer through the input layer to obtain encoding features; Inputting the coding features into the classification layer to obtain an initial probability distribution of characters corresponding to each speech frame in the target speech; Inputting the encoded features into the attention mechanism layer to obtain an attention coefficient for each speech frame in the target speech; Inputting the initial probability distribution of characters and the attention coefficient corresponding to each speech frame in the target speech into the pseudo-joint network layer to obtain the intermediate probability distribution of characters corresponding to each speech frame in the target speech; and concatenating the intermediate probability distribution of characters with the probability of a one-dimensional empty character to obtain an updated intermediate probability distribution; The updated intermediate probability distribution is constrained by using the RnnT loss constraint function to obtain a final probability distribution of characters corresponding to each speech frame in the target speech.

2. The method according to claim 1, characterized in that The speech recognition model is constructed as follows: Get sample speech; The sample speech and the RnnT loss constraint function are used to train the initial speech recognition model to obtain the speech recognition model.

3. The method according to claim 1, characterized in that Inputting the encoded features into the attention mechanism layer to obtain the attention coefficient of each speech frame in the target speech includes: Decoding the encoded features to obtain decoded features; The encoding features and the decoding features are input into the attention mechanism layer together, and attention calculation is performed to obtain the attention weighted coefficient of each speech frame in the target speech.

4. The method according to claim 2, characterized in that The method further comprises: Get verification voice; Inputting the verification speech into the speech recognition model to obtain a character recognition result corresponding to the verification speech; When the character recognition result of the verification speech is inconsistent with the character label result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

5. The method according to any one of claims 1 to 4, characterized in that The RnnT loss constraint function is used to explicitly constrain the relationship between the acoustic features of the target speech and the characters corresponding to the target speech, so as to improve the generalization ability and recognition effect of the speech recognition model.

6. A speech recognition device, characterized in that: include: A first acquisition unit, configured to acquire a target speech to be recognized; A first recognition unit is configured to input the target speech into a pre-built speech recognition model to identify and obtain a probability distribution of characters corresponding to the target speech; The speech recognition model is a neural network model obtained by training based on a neural network model based on an attention mechanism and a recursive neural network transformer RnnT loss constraint function; the speech recognition model includes an input layer, an encoder layer, a classification layer, an attention mechanism layer, a pseudo-joint network layer, and an output layer; A second recognition unit is configured to recognize the target speech according to a probability distribution of characters corresponding to the target speech, and obtain a recognition result of the target speech; The first recognition unit includes: A first input subunit, configured to input the acoustic features of the target speech into the encoder layer through the input layer to obtain encoded features; A second input subunit is configured to input the coding feature into the classification layer to obtain an initial probability distribution of characters corresponding to each speech frame in the target speech; A third input subunit is used to input the encoded features into the attention mechanism layer to obtain an attention coefficient for each speech frame in the target speech; a fourth input subunit, configured to input the initial probability distribution of characters and the attention coefficient corresponding to each speech frame in the target speech into the pseudo-joint network layer to obtain an intermediate probability distribution of characters corresponding to each speech frame in the target speech; and concatenate the intermediate probability distribution of characters with the probability of a one-dimensional empty character to obtain an updated intermediate probability distribution; The constraint subunit is used to constrain the updated intermediate probability distribution using the RnnT loss constraint function to obtain the final probability distribution of the characters corresponding to each speech frame in the target speech.

7. A speech recognition device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is configured to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor is enabled to perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 5.

Images