A speech recognition method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a speech recognition method, device, equipment and storage medium. The method includes: receiving speech information to be recognized; obtaining language information corresponding to the speech information to be recognized; determining the corresponding language residual module in the encoder included in the speech recognition model according to the language information; using the language residual module to recognize the speech information to be recognized to obtain a preliminary recognition result; inputting the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result. It can be seen that in the embodiments of the present application, a specific language residual module can be determined according to the language information to recognize the speech information to be recognized, thereby making speech recognition more targeted, and can better improve the encoder's ability to distinguish and extract multi-language information, thereby improving the accuracy of speech recognition.

Description

Technical Field

[0001] The present application relates to the field of speech recognition, and in particular to a speech recognition method, apparatus, device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, speech recognition technology, as an important research direction in artificial intelligence technology, has been gradually applied to more and more application fields, such as finance, operators, government affairs, medical care, e-commerce, catering and hotels, education and training, real estate, and automobiles. In the application of speech recognition, users may not only use standard Mandarin for speech input, but also have the need to use dialects, foreign languages ​​and other personalized languages ​​for speech input.

[0003] At present, a speech recognition model is obtained by mixing data of multiple languages ​​such as Mandarin, dialects, and foreign languages ​​for training to meet the needs of users for voice input in different languages. However, the speech recognition model trained in this way is prone to overfitting to languages ​​with larger magnitudes due to the large difference in the magnitude of training data in different languages, resulting in large deviations in speech recognition results and low speech recognition accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method, apparatus, device and storage medium to improve the accuracy of speech recognition.

[0005] The technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a speech recognition method, the method comprising:

[0007] Receiving voice information to be recognized;

[0008] Obtaining language information corresponding to the voice information to be recognized;

[0009] Determine, according to the language information, a corresponding language residual module in an encoder included in a speech recognition model;

[0010] Using the language residual module to recognize the speech information to be recognized, and obtaining a preliminary recognition result;

[0011] The preliminary recognition result is input into the decoder included in the speech recognition model for recognition to obtain a speech recognition result.

[0012] In a possible implementation, the language residual module is arranged between an input end of a submodule hidden layer in the encoder and an output end of the submodule hidden layer.

[0013] In a possible implementation, the using the language residual module to recognize the speech information to be recognized to obtain a preliminary recognition result includes:

[0014] The language residual module is used to calculate a first attention allocation result corresponding to the speech information to be recognized; the first attention allocation result is applied to the speech information to be recognized to obtain a first output result;

[0015] Using the submodule hidden layer of the encoder to calculate and obtain a second attention allocation result corresponding to the speech information to be recognized; applying the second attention allocation result to the speech information to be recognized to obtain a second output result;

[0016] The first output result and the second output result are added to obtain a preliminary recognition result.

[0017] In a possible implementation, when the encoder includes multiple submodule hidden layers, the adding of the first output result and the second output result to obtain a preliminary recognition result includes:

[0018] Adding the first output result corresponding to the current language residual module and the second output result corresponding to the current submodule hidden layer to obtain the output result of the current submodule hidden layer;

[0019] Passing the output result of the current submodule hidden layer to the next submodule hidden layer;

[0020] Using the next language residual module to calculate the third attention allocation result corresponding to the output result of the hidden layer of the current submodule; applying the third attention allocation result to the output result of the hidden layer of the current submodule to obtain a third output result;

[0021] Using the next submodule hidden layer to calculate, a fourth attention allocation result corresponding to the output result of the current submodule hidden layer is obtained; applying the fourth attention allocation result to the output result of the current submodule hidden layer to obtain a fourth output result;

[0022] Adding the third output result and the fourth output result to obtain an output result of the hidden layer of the next submodule;

[0023] If the next submodule hidden layer is the last submodule hidden layer of the encoder, determining the output result of the next submodule hidden layer as the preliminary recognition result;

[0024] If the next submodule hidden layer is not the last submodule hidden layer of the encoder, the output result of the next submodule hidden layer is passed to the submodule hidden layer after the next submodule hidden layer until the output result of the last submodule hidden layer of the encoder is obtained, and the output result of the last submodule hidden layer is determined to be the preliminary recognition result.

[0025] In a possible implementation, inputting the preliminary recognition result into a decoder for recognition to obtain a speech recognition result includes:

[0026] Inputting the language information and the preliminary recognition result into the decoder;

[0027] Determine a corresponding language decoding space in the decoder according to the language information;

[0028] The preliminary recognition result is recognized using the language decoding space to obtain a speech recognition result.

[0029] In a possible implementation, inputting the preliminary recognition result into a decoder for recognition to obtain a speech recognition result includes:

[0030] Inputting the language information and the preliminary recognition result into the decoder;

[0031] Determine a corresponding language decoding space in the decoder according to the language information;

[0032] The preliminary recognition result is recognized using the language decoding space to obtain a speech recognition result.

[0033] In a possible implementation, the acquiring the language information corresponding to the to-be-recognized voice information includes:

[0034] Receiving language recognition information corresponding to the voice information to be recognized selected by the terminal;

[0035] Or, the language recognition information corresponding to the voice information to be recognized is determined according to the historical voice recognition results of the terminal.

[0036] In a possible implementation, the language residual module includes any one of the following: a deep neural network DNN module, a convolutional neural network CNN module, and an attention module.

[0037] In a second aspect, an embodiment of the present application provides a speech recognition device, the device comprising:

[0038] A receiving unit, used for receiving voice information to be recognized;

[0039] An acquisition unit, used to acquire language information corresponding to the voice information to be recognized;

[0040] A module determination unit, configured to determine a corresponding language residual module in an encoder included in a speech recognition model according to the language information;

[0041] A first recognition unit, configured to recognize the speech information to be recognized by using the language residual module to obtain a preliminary recognition result;

[0042] The second recognition unit is used to input the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result.

[0043] In a third aspect, an embodiment of the present application provides a speech recognition device, the device comprising: a processor, a memory, and a system bus;

[0044] The processor and the memory are connected via the system bus;

[0045] The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the speech recognition method described in any one of the first aspects above.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a terminal device, the terminal device executes the speech recognition method described in any one of the above first aspects.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the speech recognition method described in any one of the first aspects above.

[0048] Based on the above technical solution, this application has the following beneficial effects:

[0049] The embodiments of the present application disclose a speech recognition method, device, equipment and storage medium. The method includes: receiving speech information to be recognized; obtaining language information corresponding to the speech information to be recognized; determining the corresponding language residual module in the encoder included in the speech recognition model according to the language information; using the language residual module to recognize the speech information to be recognized to obtain a preliminary recognition result; inputting the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result. It can be seen that in the embodiments of the present application, a specific language residual module can be determined according to the language information to recognize the speech information to be recognized, thereby making speech recognition more targeted, and can better improve the encoder's ability to distinguish and extract multi-language information, thereby improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a speech recognition method disclosed in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of the structure of an encoder disclosed in an embodiment of the present application;

[0053] Figure 3 A flowchart of another speech recognition method disclosed in an embodiment of the present application;

[0054] Figure 4 A flowchart of another speech recognition method disclosed in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the structure of a speech recognition device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] The embodiments of the present application disclose a speech recognition method, device, equipment and storage medium. The method includes: receiving speech information to be recognized; obtaining language information corresponding to the speech information to be recognized; determining the corresponding language residual module in the encoder included in the speech recognition model according to the language information; using the language residual module to recognize the speech information to be recognized to obtain a preliminary recognition result; inputting the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result. It can be seen that in the embodiments of the present application, a specific language residual module can be determined according to the language information to recognize the speech information to be recognized, thereby making speech recognition more targeted, and can better improve the encoder's ability to distinguish and extract multi-language information, thereby improving the accuracy of speech recognition.

[0058] See also Figure 1 , a flow chart of a speech recognition method disclosed in an embodiment of the present application, the method comprising:

[0059] Step S101, receiving voice information to be recognized;

[0060] Step S102: obtaining language information corresponding to the voice information to be recognized;

[0061] The language information is used to indicate the language corresponding to the voice information to be recognized. The languages ​​in the embodiment of the present application include Minnan dialect, English, German, etc., which are not specifically limited and can be set according to actual needs.

[0062] In the embodiment of the present application, the method of obtaining the language information corresponding to the voice information to be recognized includes: receiving the language recognition information corresponding to the voice information to be recognized selected by the terminal; or determining the language recognition information corresponding to the voice information to be recognized based on the historical voice recognition results of the terminal. That is, in the embodiment of the present application, the user can select the specified language information, or the terminal can determine the language information based on the historical use of voice recognition.

[0063] Step S103, determining a language residual module corresponding to the language in the encoder included in the speech recognition model according to the language information;

[0064] Among them, different language information corresponds to different language residual modules. For example: if the language information indicates that the language corresponding to the speech information to be recognized is English, then the language residual module corresponding to English is activated; if the language information indicates that the language corresponding to the speech information to be recognized is German, then the language residual module corresponding to German is activated, and so on. It can be understood that the above is only an exemplary description and should not be understood as a limitation to this application.

[0065] It should be noted that the traditional encoder can be composed of multiple network structures such as the popular Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), TRANSFORMER, etc., which are stacked to extract high-order hidden features from speech features. Taking TRANSFORMER as an example, TRANSFORMER is stacked by several submodules, each of which contains a feedforward layer, a self-attention layer, etc., collectively referred to as the submodule hidden layer of the encoder.

[0066] In the embodiment of the present application, the internal structure of the encoder is optimized and a language residual module is introduced. Figure 2, is a structural diagram of an encoder disclosed in an embodiment of the present application, wherein the language residual module is arranged between the input end and the output end of the submodule hidden layer of the encoder, and each submodule hidden layer in the encoder has a group of language residual modules to be activated in parallel therewith, and when speech recognition is required, the language residual module corresponding to the language information in the group of language residual modules to be activated corresponding to each submodule hidden layer is activated according to the acquired language information.

[0067] The language residual module in the embodiment of the present application includes any one of the following: a deep neural network DNN module, a convolutional neural network CNN module, an attention module, etc., without specific limitation, and can be set according to actual needs.

[0068] Step S104: using the language residual module to recognize the speech information to be recognized, and obtaining a preliminary recognition result;

[0069] Step S105: input the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result.

[0070] It should be noted that the decoder can be a network structure such as RNN, LSTM, etc., and contains a set of attention modules inside, which converts the high-order latent features extracted by the encoder into a probability distribution of a set of character sequences, and then obtains the speech recognition result through this probability distribution. It can be understood that the high-order latent features are the preliminary recognition results in the embodiments of the present application.

[0071] In the embodiment of the present application, a speech recognition method is disclosed, which comprises receiving speech information to be recognized; obtaining language information corresponding to the speech information to be recognized; determining the corresponding language residual module in the encoder included in the speech recognition model according to the language information; using the language residual module to recognize the speech information to be recognized, and obtaining a preliminary recognition result; inputting the preliminary recognition result into the decoder included in the speech recognition model for recognition, and obtaining a speech recognition result. It can be seen that in the embodiment of the present application, a specific language residual module can be determined according to the language information to recognize the speech information to be recognized, thereby making speech recognition more targeted, and can better improve the encoder's ability to distinguish and extract multi-language information, thereby improving the accuracy of speech recognition.

[0072] In a possible implementation, step S104 in the speech recognition method provided in the embodiment of the present application specifically includes:

[0073] Step S1041: using the language residual module to calculate and obtain a first attention allocation result corresponding to the speech information to be recognized; applying the first attention allocation result to the speech information to be recognized to obtain a first output result;

[0074] In a possible implementation, during the training process of the speech recognition model, when training is performed using training data of different languages, the corresponding language residual module in the encoder is activated for training, so that the language residual module corresponding to the language will pre-learn a series of mapping relationship pairs of keys and corresponding values ​​corresponding to the language. During the speech recognition process, the language residual module will be used to calculate the query information corresponding to the speech information to be recognized in the specified language, and then the query information and the mapping relationship pairs pre-learned by the language residual module are used to calculate the attention allocation result corresponding to the speech information to be recognized.

[0075] It should be noted that query information refers to the representation information extracted from the input of the submodule hidden layer, which can be obtained by passing the input of the submodule hidden layer through a neural network, such as a layer of DNN. The input of the submodule hidden layer in the embodiment of the present application can be the speech information to be recognized or the output result of the previous submodule hidden layer.

[0076] Step S1042: using the submodule hidden layer of the encoder to calculate and obtain a second attention allocation result corresponding to the speech information to be recognized; applying the second attention allocation result to the speech information to be recognized to obtain a second output result;

[0077] Similarly, in a possible implementation, during the training of the speech recognition model, the submodule hidden layer will pre-learn a series of mapping pairs of keys and corresponding values ​​corresponding to all languages. It should be noted that the mapping pairs learned by the submodule hidden layer are different from the mapping pairs learned by the language residual module. During the speech recognition process, the submodule hidden layer will be used to calculate the query information corresponding to the speech information to be recognized, and then the query information and the mapping pairs pre-learned by the submodule hidden layer will be used to calculate the attention allocation result corresponding to the speech information to be recognized.

[0078] Step S1043: Add the first output result and the second output result to obtain a preliminary recognition result.

[0079] It should be noted that, assuming that the formula is abstracted as z = f(x) + g(x, y), where z is the output result of the submodule hidden layer, x is the input data of the submodule hidden layer, and y is the language information. The addition here means that the submodule hidden layer learns the common speech information f(x) that is not dependent on the language, and the language residual module learns the individual speech information g(x, y) of different languages. The latter is regarded as the residual of the former, and the two are complementary and added.

[0080] It can be seen that the output result of the language residual module in the embodiment of the present application is calculated through the pre-learned language individual speech information, and the output result of the sub-module hidden layer is obtained through the pre-learned common speech information. In this way, adding the output result of the language residual module and the output result of the sub-module hidden layer can obtain a more accurate preliminary recognition result and improve the accuracy of speech recognition.

[0081] In a possible implementation, in the speech recognition method provided in the embodiment of the present application, when the encoder includes multiple submodule hidden layers, step S1043 specifically includes:

[0082] Step S10431, adding the first output result corresponding to the current language residual module and the second output result corresponding to the current submodule hidden layer to obtain the output result of the current submodule hidden layer;

[0083] It should be noted that the encoder includes multiple submodule hidden layers, the input of the first submodule hidden layer of the encoder is the speech information to be recognized, and the subsequent submodule hidden layers use the output result of the previous submodule hidden layer as input. The current submodule hidden layer here is the first submodule hidden layer of the encoder, and its input is the speech information to be recognized.

[0084] Step S10432, passing the output result of the hidden layer of the current submodule to the hidden layer of the next submodule;

[0085] Step S10433: using the next language residual module to calculate and obtain a third attention allocation result corresponding to the output result of the hidden layer of the current submodule; applying the third attention allocation result to the output result of the hidden layer of the current submodule to obtain a third output result;

[0086] Step S10434, using the next submodule hidden layer to calculate and obtain a fourth attention allocation result corresponding to the output result of the current submodule hidden layer; applying the fourth attention allocation result to the output result of the current submodule hidden layer to obtain a fourth output result;

[0087] Step S10435: Add the third output result and the fourth output result to obtain the output result of the hidden layer of the next submodule;

[0088] Step S10436: If the next submodule hidden layer is the last submodule hidden layer of the encoder, determine the output result of the next submodule hidden layer as the preliminary recognition result;

[0089] Step S10437: If the next submodule hidden layer is not the last submodule hidden layer of the encoder, the output result of the next submodule hidden layer is passed to the submodule hidden layer after the next submodule hidden layer, until the output result of the last submodule hidden layer of the encoder is obtained, and the output result of the last submodule hidden layer is determined to be the preliminary recognition result.

[0090] It can be understood that in the embodiment of the present application, the output result of the next sub-module hidden layer is passed to the sub-module hidden layer after the next sub-module hidden layer, and the sub-module hidden layer after the next sub-module hidden layer and the language residual module connected in parallel will respectively calculate the attention allocation result corresponding to the output result of the next sub-module hidden layer, and then apply the attention allocation result to the output result of the next sub-module hidden layer until the last sub-module hidden layer of the encoder is calculated and the preliminary recognition result is output.

[0091] It can be seen that each sub-module hidden layer in the encoder in the embodiment of the present application and the language residual module connected in parallel will calculate the attention allocation result of the sub-module hidden layer input, and add them up to obtain the final output result of the sub-module hidden layer, and use the output result of the previous sub-module hidden layer as the input of the current sub-module hidden layer, so that a more accurate preliminary recognition result can be obtained, thereby improving the accuracy of speech recognition.

[0092] See also Figure 3 , a flow chart of another speech recognition method provided in an embodiment of the present application, step S105 in the speech recognition method includes:

[0093] Step S1051, inputting the language information and the preliminary recognition result into the decoder;

[0094] Figure 3 The encoder output in is the preliminary recognition result in the embodiment of the present application.

[0095] Step S1052, determining a corresponding language decoding space in the decoder according to the language information;

[0096] Step S1052: using the language decoding space to recognize the preliminary recognition result to obtain a speech recognition result.

[0097] It can be seen that the decoder is optimized in the embodiment of the present application, and language information is introduced to guide the encoder's attention, ensuring that the decoder performs speech recognition decoding in the decoding space of the specified language, reducing mutual crosstalk between recognition of different languages, and further improving speech recognition accuracy.

[0098] In a possible implementation, step S1051 in the speech recognition method provided in the embodiment of the present application specifically includes:

[0099] Step S10511, generating a corresponding language embedding representation according to the language information;

[0100] The generation method of language embedding representation includes but is not limited to known priors, model prediction, etc. For example: if the language information is English, the language embedding representation is

[0001] ; if the language information is German, the language embedding representation is

[0010] , etc. It is understood that the above is only an exemplary description and should not be construed as a limitation on the present application.

[0101] Step S10512: concatenate the language embedding representation onto the preliminary recognition result to obtain a preliminary recognition result carrying the language information;

[0102] Step S10513: input the preliminary recognition result carrying language information into the decoder.

[0103] It can be seen that in the embodiment of the present application, by splicing the language embedding representation on the preliminary recognition result, it is possible to determine the decoding space corresponding to the language of the speech information to be recognized through the language embedding representation, ensure that speech recognition decoding is performed in the decoding space of the specified language, reduce the mutual crosstalk between recognition of different languages, and further improve the accuracy of speech recognition.

[0104] See also Figure 4 , which is a flow chart of another speech recognition method disclosed in an embodiment of the present application. In the speech recognition method, the language information is specified by the user or output by the front module, the encoder receives the audio input, obtains the language information, determines the corresponding language residual module according to the language information, and uses the language residual module to recognize the audio feature input to obtain a preliminary recognition result; then the preliminary recognition result is input into the decoder, the decoder obtains the language information, determines the corresponding language decoding space according to the language information, and uses the language decoding space to recognize the preliminary recognition result to obtain the final text output.

[0105] See also Figure 5 , a schematic diagram of the structure of a speech recognition device disclosed in an embodiment of the present application, the device comprising:

[0106] The receiving unit 501 is used to receive the voice information to be recognized;

[0107] An acquisition unit 502 is used to acquire language information corresponding to the speech information to be recognized;

[0108] An activation unit 503, configured to determine a corresponding language residual module in an encoder included in a speech recognition model according to the language information;

[0109] The first recognition unit 504 is used to recognize the speech information to be recognized by using the language residual module to obtain a preliminary recognition result;

[0110] The second recognition unit 505 is used to input the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result.

[0111] In an embodiment of the present application, a speech recognition device is disclosed, which receives speech information to be recognized; obtains language information, the language information is used to indicate the language corresponding to the speech information to be recognized; activates the corresponding language residual module in the encoder included in the speech recognition model according to the language information; uses the language residual module to recognize the speech information to be recognized, and obtains a preliminary recognition result; inputs the preliminary recognition result into the decoder included in the speech recognition model for recognition, and obtains a speech recognition result. It can be seen that in the embodiment of the present application, a specific language residual module can be activated according to the language information to recognize the speech information to be recognized, thereby making speech recognition more targeted, and can better improve the encoder's ability to distinguish and extract multi-language information, and improve speech recognition accuracy.

[0112] In a possible implementation, the language residual module in the speech recognition device provided in the embodiment of the present application is arranged between the input end of the submodule hidden layer of the encoder and the output end of the submodule hidden layer.

[0113] In a possible implementation, the first recognition unit 504 in the speech recognition device provided in the embodiment of the present application includes:

[0114] A first calculation unit is used to calculate a first attention allocation result corresponding to the speech information to be recognized by using the language residual module; and apply the first attention allocation result to the speech information to be recognized to obtain a first output result;

[0115] A second calculation unit is used to calculate the second attention allocation result corresponding to the speech information to be recognized by using the submodule hidden layer of the encoder; and apply the second attention allocation result to the speech information to be recognized to obtain a second output result;

[0116] An adding unit is used to add the first output result and the second output result to obtain a preliminary recognition result.

[0117] In a possible implementation, in the speech recognition device provided in the embodiment of the present application, when the encoder includes multiple submodule hidden layers, the adding unit specifically includes:

[0118] A first adding subunit is used to add a first output result corresponding to the current language residual module and a second output result corresponding to the current submodule hidden layer to obtain an output result of the current submodule hidden layer;

[0119] A transmission unit, used to transmit the output result of the current submodule hidden layer to the next submodule hidden layer;

[0120] A third calculation unit is used to calculate a third attention allocation result corresponding to the output result of the hidden layer of the current submodule by using the next language residual module; and apply the third attention allocation result to the output result of the hidden layer of the current submodule to obtain a third output result;

[0121] A fourth calculation unit is used to calculate the fourth attention allocation result corresponding to the output result of the hidden layer of the current submodule by using the hidden layer of the next submodule; and apply the fourth attention allocation result to the output result of the hidden layer of the current submodule to obtain a fourth output result;

[0122] A second adding subunit is used to add the third output result and the fourth output result to obtain the output result of the hidden layer of the next submodule;

[0123] a result determination unit, configured to determine, if the next submodule hidden layer is the last submodule hidden layer of the encoder, an output result of the next submodule hidden layer as the preliminary recognition result;

[0124] If the next submodule hidden layer is not the last submodule hidden layer of the encoder, the output result of the next submodule hidden layer is passed to the submodule hidden layer after the next submodule hidden layer until the output result of the last submodule hidden layer of the encoder is obtained, and the output result of the last submodule hidden layer is determined to be the preliminary recognition result.

[0125] In a possible implementation manner, the second recognition unit 505 in the speech recognition device provided in the embodiment of the present application specifically includes:

[0126] An input unit, used for inputting the language information and the preliminary recognition result into the decoder;

[0127] A decoding space determination unit, configured to determine a corresponding language decoding space in the decoder according to the language information;

[0128] The recognition subunit is used to recognize the preliminary recognition result by using the language decoding space to obtain a speech recognition result.

[0129] In a possible implementation manner, the input unit in the speech recognition device provided in the embodiment of the present application specifically includes:

[0130] A generating unit, configured to generate a corresponding language embedding representation according to the language information;

[0131] A concatenation unit, configured to concatenate the language embedding representation onto the preliminary recognition result to obtain a preliminary recognition result carrying the language information;

[0132] The input subunit is used to input the preliminary recognition result carrying language information into the decoder.

[0133] In a possible implementation manner, the acquisition unit 502 in the speech recognition device provided in the embodiment of the present application is specifically used for:

[0134] Receiving language recognition information corresponding to the voice information to be recognized selected by the terminal;

[0135] Or, the language recognition information corresponding to the voice information to be recognized is determined according to the historical voice recognition results of the terminal.

[0136] In a possible implementation, the language residual module in the speech recognition device provided in the embodiment of the present application includes any one of the following: a deep neural network DNN module, a convolutional neural network CNN module, and an attention module.

[0137] Furthermore, an embodiment of the present application also provides a speech recognition device, including: a processor, a memory, and a system bus;

[0138] The processor and the memory are connected via the system bus;

[0139] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned speech recognition method.

[0140] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0141] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech recognition method.

[0142] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0144] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0145] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: The method comprises: Receiving voice information to be recognized; Obtaining language information corresponding to the voice information to be recognized; Determine, according to the language information, a corresponding language residual module in an encoder included in a speech recognition model; Using the language residual module to recognize the speech information to be recognized, and obtaining a preliminary recognition result; Inputting the preliminary recognition result into the decoder included in the speech recognition model for recognition to obtain a speech recognition result; The step of using the language residual module to recognize the speech information to be recognized to obtain a preliminary recognition result includes: The language residual module is used to calculate a first attention allocation result corresponding to the speech information to be recognized; the first attention allocation result is applied to the speech information to be recognized to obtain a first output result; Using the submodule hidden layer of the encoder to calculate and obtain a second attention allocation result corresponding to the speech information to be recognized; applying the second attention allocation result to the speech information to be recognized to obtain a second output result; The first output result and the second output result are added to obtain a preliminary recognition result.

2. The method according to claim 1, characterized in that The language residual module is arranged between the input end of the submodule hidden layer in the encoder and the output end of the submodule hidden layer.

3. The method according to claim 1, characterized in that When the encoder includes multiple submodule hidden layers, the adding of the first output result and the second output result to obtain a preliminary recognition result includes: Adding the first output result corresponding to the current language residual module and the second output result corresponding to the current submodule hidden layer to obtain the output result of the current submodule hidden layer; Passing the output result of the current submodule hidden layer to the next submodule hidden layer; Using the next language residual module to calculate the third attention allocation result corresponding to the output result of the hidden layer of the current submodule; applying the third attention allocation result to the output result of the hidden layer of the current submodule to obtain a third output result; Using the next submodule hidden layer to calculate, a fourth attention allocation result corresponding to the output result of the current submodule hidden layer is obtained; applying the fourth attention allocation result to the output result of the current submodule hidden layer to obtain a fourth output result; Adding the third output result and the fourth output result to obtain an output result of the hidden layer of the next submodule; If the next submodule hidden layer is the last submodule hidden layer of the encoder, determining the output result of the next submodule hidden layer as the preliminary recognition result; If the next submodule hidden layer is not the last submodule hidden layer of the encoder, the output result of the next submodule hidden layer is passed to the submodule hidden layer after the next submodule hidden layer until the output result of the last submodule hidden layer of the encoder is obtained, and the output result of the last submodule hidden layer is determined to be the preliminary recognition result.

4. The method according to claim 1, characterized in that The step of inputting the preliminary recognition result into a decoder for recognition to obtain a speech recognition result comprises: Inputting the language information and the preliminary recognition result into the decoder; Determine a corresponding language decoding space in the decoder according to the language information; The preliminary recognition result is recognized using the language decoding space to obtain a speech recognition result.

5. The method according to claim 4, wherein inputting the language information and the preliminary recognition result into the decoder comprises: Generate a corresponding language embedding representation according to the language information; splicing the language embedding representation onto the preliminary recognition result to obtain a preliminary recognition result carrying the language information; The preliminary recognition result carrying the language information is input into the decoder.

6. The method according to claim 1, characterized in that The acquiring of the language information corresponding to the voice information to be recognized includes: Receiving language recognition information corresponding to the voice information to be recognized selected by the terminal; Or, the language recognition information corresponding to the voice information to be recognized is determined according to the historical voice recognition results of the terminal.

7. The method according to claim 1, characterized in that The language residual module includes any one of the following: a deep neural network DNN module, a convolutional neural network CNN module, and an attention module.

8. A speech recognition device, characterized in that: The device comprises: A receiving unit, used for receiving voice information to be recognized; An acquisition unit, used to acquire language information corresponding to the voice information to be recognized; A module determination unit, configured to determine a corresponding language residual module in an encoder included in a speech recognition model according to the language information; A first recognition unit, configured to recognize the speech information to be recognized by using the language residual module to obtain a preliminary recognition result; A second recognition unit, used for inputting the preliminary recognition result into a decoder included in the speech recognition model for recognition to obtain a speech recognition result; The first identification unit comprises: A first calculation unit is used to calculate a first attention allocation result corresponding to the speech information to be recognized by using the language residual module; and apply the first attention allocation result to the speech information to be recognized to obtain a first output result; A second calculation unit is used to calculate the second attention allocation result corresponding to the speech information to be recognized by using the submodule hidden layer of the encoder; and apply the second attention allocation result to the speech information to be recognized to obtain a second output result; An adding unit is used to add the first output result and the second output result to obtain a preliminary recognition result.

9. A speech recognition device, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to perform the speech recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the speech recognition method according to any one of claims 1 to 7.

Images