Speech recognition method, device, computer-readable storage medium, and computer equipment

Abstract

The embodiments of the present application disclose a speech recognition method, apparatus, computer-readable storage medium and computer equipment, which can realize end-to-end multi-language switching-free speech recognition, and uniformly establish a preset speech recognition model for multiple languages. The preset speech recognition model includes independent parameter modules and shared parameter modules corresponding to preset interconnected languages. The independent parameter modules corresponding to a preset number of languages correspond to the languages respectively, and the model parameters are shared by the shared parameter module. In this way, no matter what language of the speech data to be recognized is input, the shared parameter module can be used to reduce the model parameters of the preset speech recognition model. At the same time, because the independent parameter modules are language-bound, that is, the independent parameter modules set independent parameters according to the language, the mutual influence between different languages is avoided, the language confusion is reduced, and the accuracy of multi-language speech recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a speech recognition method, apparatus, computer-readable storage medium, and computer equipment. Background Art

[0002] my country has over 100 dialects. While Mandarin is the most widely used, dialects still play an important role in daily life. For example, residents of regions A, B, and C primarily communicate in dialects A, B, and C. Consequently, the demand for speech recognition that supports a mix of Mandarin and dialects (referred to as dialect-based speech recognition) is increasing.

[0003] Current methods for Fangpu speech recognition include: using multiple single-language recognition systems in parallel for calculation, and finally judging the language through a language judgment module and throwing out the corresponding language results. This method will generate a large amount of invalid calculations, greatly increasing the deployment cost; or directly use Mandarin modeling, add dialect data and directly mix it into a unified Fangpu modeling system. However, due to the large differences in grammar and pronunciation of different dialects, the recognition effect of this modeling method often does not achieve the desired effect, and may also cause some loss of the effect of Mandarin.

[0004] It can be seen from this that the current Fangpu speech recognition method has low recognition efficiency and poor robustness. Summary of the Invention

[0005] The embodiments of the present application provide a speech recognition method, apparatus, computer-readable storage medium, and computer equipment, which can improve the recognition efficiency and accuracy of a preset speech recognition model.

[0006] The present invention provides a speech recognition method, including:

[0007] receiving a selected language and obtaining a preset speech recognition model, wherein the selected language is at least one of a preset number of languages in the preset speech recognition model;

[0008] Receiving input audio data to be recognized;

[0009] Performing language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and performing speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in a decoding module to obtain an audio recognition result corresponding to the target language for the audio data, wherein the target language is at least one of the selected languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages;

[0010] Returns the audio recognition result.

[0011] The present invention also provides a method for speech recognition, including:

[0012] Receiving input audio data to be recognized and obtaining a preset speech recognition model, wherein the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages;

[0013] Performing language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and performing speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in a decoding module to obtain an audio recognition result of the audio data corresponding to the target language, wherein the target language is at least one language from a preset number of languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages;

[0014] Returns the audio recognition result.

[0015] The present invention also provides a method for speech recognition, including:

[0016] Acquiring speech data to be recognized in a current frame, a speech recognition result in a frame previous to the current frame, and a preset speech recognition model, wherein the preset speech recognition model includes a prediction network, an encoding network, and a joint network, the prediction network includes a language embedding layer and a preset network layer, the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages;

[0017] Merging the encoded features obtained by encoding the speech data using the encoding network and the predicted features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer corresponding to each language in the language embedding layer and the prediction network layer to obtain a merged feature, wherein the language embedding layer includes the sub-embedding layers corresponding to the preset number of languages;

[0018] performing joint mapping processing on the merged features using a sub-joint network layer corresponding to each language in the joint network layer and a sub-softmax network layer corresponding to each language in the softmax network layer to obtain a mapping probability of the speech data corresponding to each language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages;

[0019] The mapping probability is decoded and recognized using a sub-decoding module corresponding to each language in the decoding module to obtain a speech recognition result corresponding to each language of the speech data.

[0020] The present invention also provides a method for speech recognition, including:

[0021] Obtaining training speech data of a current frame of a target training sample in a specific language and a training speech recognition result of a previous frame of the current frame, and obtaining an initial speech recognition model, wherein the initial speech recognition model includes an encoding network, a prediction network, and a joint network, the prediction network includes a language embedding layer and a prediction network layer, the joint network includes a joint network layer and a softmax network layer, the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages, wherein the specific language is at least one of the predetermined number of languages predetermined by the initial speech recognition model;

[0022] Merging the training encoding features obtained by encoding the training speech data using the encoding network and the training prediction features obtained by predicting the training speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the specific language to obtain a training merged feature, wherein the language embedding layer includes the sub-embedding layers corresponding to the preset number of languages;

[0023] Performing joint mapping processing on the training merged features using a sub-joint network layer corresponding to a specific language in the joint network layer and a sub-softmax network layer corresponding to a specific language in the softmax network layer to obtain a training mapping probability that the training speech data corresponds to the specific language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages;

[0024] Using a sub-decoding module corresponding to a specific language in a decoding module to perform decoding and recognition processing on the training mapping probability to obtain a training speech recognition result corresponding to the specific language of the training speech data;

[0025] Updating model parameters in the initial speech recognition model using the training speech recognition result of the current frame and the label data of the current frame of the target training sample, thereby training a preset number of languages to obtain a preset speech recognition model;

[0026] The preset speech recognition model is used to perform recognition processing on each frame of speech data corresponding to the audio data to be recognized, so as to obtain a speech recognition result of the speech data.

[0027] The present invention also provides a speech recognition device, including:

[0028] a receiving and obtaining module, configured to receive a selected language and obtain a preset speech recognition model, wherein the selected language is at least one of a preset number of languages in the preset speech recognition model;

[0029] The receiving and obtaining module is further configured to receive input audio data to be recognized;

[0030] a speech recognition module configured to perform language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and to perform speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in the decoding module, to obtain an audio recognition result corresponding to the target language for the audio data, wherein the target language is at least one of the selected languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages;

[0031] The return module is used to return the audio recognition result.

[0032] The present invention also provides a speech recognition device, including:

[0033] A receiving and obtaining module, configured to receive input audio data to be recognized and obtain a preset speech recognition model, wherein the preset speech recognition model is configured to perform speech recognition on speech data of a predetermined number of languages;

[0034] a speech recognition module configured to perform language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and to perform speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in the decoding module, to obtain an audio recognition result corresponding to the target language for the audio data, wherein the target language is at least one language from a preset number of languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages;

[0035] The return module is used to return the audio recognition result.

[0036] The present invention also provides a speech recognition device, including:

[0037] a first acquisition module, configured to acquire speech data to be recognized in a current frame, a speech recognition result in a frame previous to the current frame, and a preset speech recognition model, wherein the preset speech recognition model includes a prediction network, an encoding network, and a joint network, the prediction network includes a language embedding layer and a preset network layer, the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages;

[0038] a first merging module, configured to merge encoding features obtained by encoding the speech data using the encoding network and prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer corresponding to each language in the language embedding layer and the prediction network layer, to obtain a merged feature, wherein the language embedding layer includes the sub-embedding layers corresponding to the preset number of languages;

[0039] a first mapping module, configured to perform a joint mapping process on the merged features using a sub-joint network layer corresponding to each language in the joint network layer and a sub-softmax network layer corresponding to each language in the softmax network layer, so as to obtain a mapping probability of the speech data corresponding to each language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages;

[0040] The first decoding and recognition module is used to use the sub-decoding modules corresponding to each language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to each language of the speech data.

[0041] The present invention also provides a speech recognition device, including:

[0042] a second acquisition module, configured to acquire training speech data of a current frame of a target training sample in a specific language and a training speech recognition result of a previous frame of the current frame, and to acquire an initial speech recognition model, wherein the initial speech recognition model includes an encoding network, a prediction network, and a joint network, wherein the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is configured to perform speech recognition on speech data of a predetermined number of languages, wherein the specific language is at least one of the predetermined number of languages predetermined by the initial speech recognition model;

[0043] a second merging module, configured to merge the training coding features obtained by encoding the training speech data using the encoding network and the training prediction features obtained by predicting the training speech recognition result of the previous frame using the prediction network, to obtain a training merged feature;

[0044] a second mapping module, configured to perform joint mapping processing on the training merged features using a sub-joint network layer corresponding to a specific language in the joint network layer and a sub-softmax network layer corresponding to a specific language in the softmax network layer, so as to obtain a training mapping probability that the training speech data corresponds to a specific language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages;

[0045] A second decoding and recognition module is configured to perform decoding and recognition processing on the training mapping probabilities using a sub-decoding module corresponding to a specific language in the decoding module to obtain a training speech recognition result corresponding to each language of the training speech data;

[0046] An updating module, configured to update model parameters in the initial speech recognition model using the training speech recognition result of the current frame and the label data of the current frame of the target training sample, thereby training a preset number of languages to obtain a preset speech recognition model;

[0047] The recognition module is used to use the preset speech recognition model to perform recognition processing on each frame of speech data corresponding to the audio data to be recognized, so as to obtain a speech recognition result of the speech data.

[0048] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the speech recognition method described in any of the above embodiments.

[0049] An embodiment of the present application also provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps of the speech recognition method described in any of the above embodiments by calling the computer program stored in the memory.

[0050] The speech recognition method, apparatus, computer-readable storage medium, and computer device provided in the embodiments of the present application can achieve end-to-end multi-language switching-free speech recognition, and uniformly establish a preset speech recognition model for a preset number of languages. The preset speech recognition model includes independent parameter modules and shared parameter modules corresponding to the preset number of languages. The independent parameter modules corresponding to the preset number of languages correspond to the languages respectively, and the model parameters are shared by the shared parameter module. In this way, regardless of the language of the input speech data to be recognized, the shared parameter module can be used to reduce the model parameters of the preset speech recognition model. Since the recognition of a preset number of languages can be achieved in one model, the recognition efficiency of the preset speech recognition model is improved compared to the parallel calculation of multiple single-language recognition systems. At the same time, since the independent parameter modules are language-bound, that is, the independent parameter modules set independent parameters according to the languages, the mutual influence between different languages is avoided, the language confusion is reduced, the differences in grammar and pronunciation of different languages are taken into account, and the accuracy of multi-language speech recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram of the structure of the speech recognition model provided in the embodiment of the present application.

[0053] Figure 2 A schematic diagram of the structure of the preset speech recognition model provided in an embodiment of the present application.

[0054] Figure 3 A flowchart of the speech recognition method provided in an embodiment of the present application.

[0055] Figure 4 A flowchart of the speech recognition method provided in an embodiment of the present application.

[0056] Figure 5 A schematic diagram of the structure of the preset speech recognition model provided in an embodiment of the present application.

[0057] Figure 6 A flow chart of a speech recognition method provided in an embodiment of the present application.

[0058] Figure 7 Another structural diagram of the preset speech recognition model provided in an embodiment of the present application.

[0059] Figure 8 Another flowchart of a speech recognition method provided in an embodiment of the present application.

[0060] Figure 9 A schematic diagram of another structure of a preset speech recognition model provided in an embodiment of the present application.

[0061] Figure 10 Another flowchart of a speech recognition method provided in an embodiment of the present application.

[0062] Figure 11 Another flowchart of a speech recognition method provided in an embodiment of the present application.

[0063] Figure 12 Another flowchart of a speech recognition method provided in an embodiment of the present application.

[0064] Figure 13 Another flowchart of the speech recognition method provided in an embodiment of the present application.

[0065] Figure 14 A flowchart of another speech recognition method provided in an embodiment of the present application.

[0066] Figure 15 A schematic diagram of the structure of the speech recognition device provided in an embodiment of the present application.

[0067] Figure 16 A schematic diagram of the structure of the speech recognition device provided in an embodiment of the present application.

[0068] Figure 17 Another structural diagram of the speech recognition device provided in an embodiment of the present application.

[0069] Figure 18 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0071] The embodiments of the present application provide a speech recognition method, device, computer-readable storage medium and computer equipment. Specifically, the speech recognition method of the embodiment of the present application can be executed by a computer device, and the speech recognition device in the embodiment of the present application is integrated in the computer device, wherein the computer device can be a terminal or server or other equipment. The terminal can be a smart phone, tablet computer, laptop computer, touch screen, game console, personal computer (PC, Personal Computer), smart car terminal, robot or robot-like device. The server can be an independent physical server, or a service node in a blockchain system, or a server cluster composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0072] The preset speech recognition model in the embodiments of the present application is a speech recognition model based on a multilingual recurrent neural network transformer (RNN-T). That is, the preset speech recognition model in the embodiments of the present application is a speech recognition model obtained by improving the RNN-T model. Before providing a detailed description of the preset speech recognition model in the embodiments of the present application, we first briefly describe the structure of the RNN-T model to facilitate understanding of the preset speech recognition model in the embodiments of the present application.

[0073] Figure 1 It is a structural diagram of the speech recognition model. Figure 1 The speech recognition model described in the above is an RNN-T model, which includes an encoding network (also called a decoder), a prediction network, and a joint network. The speech recognition result ya-1 output by the RNN-T model at the t-1 frame of the audio data x or the t-1 moment of the audio data x is input into the prediction network for processing to obtain prediction features such as a prediction feature vector, and the speech data x at the t frame of the audio data x or the t moment of the audio data x is input into the prediction network for processing to obtain prediction features such as a prediction feature vector. t , input into the encoding network for processing to obtain encoding features such as encoding feature vectors, input the prediction features and encoding features together into the joint network, and finally generate the posterior probability (mapping probability) of the entire space after processing by the joint network layer and softmax network layer of the joint network, sort the posterior probabilities, select the top value, and obtain ya, and so on until the end.

[0074] The output of the joint network in the RNN-T model is a direct mixture of multiple languages, such as a direct mixture of Mandarin and dialects. It does not take into account the differences in grammar and pronunciation of different dialects. The effect is not as good as that of a monolingual speech recognition system and the accuracy is low.

[0075] Based on the RNN-T model, this application improves the RNN-T model. The structural diagram of the preset speech recognition model obtained after the improvement is as follows: Figure 2 As shown. Among them, the prediction network and the joint network are improved. Specifically, the prediction network is divided into a language embedding layer (Embedding) and a prediction network layer, wherein the language embedding layer is provided with sub-embedding layers corresponding to a preset number of languages, wherein the preset number of languages includes two or more different languages, and the functions implemented by the prediction network layer are the same as the current ones, which will not be repeated below; the joint network layer in the joint network is provided with sub-joint network layers corresponding to a preset number of languages; the softmax network layer in the joint network is provided with sub-softmax network layers corresponding to a preset number of languages. Among them, the sub-joint network layers corresponding to the preset number of languages in the joint network layer, the sub-softmax network layers corresponding to the preset number of languages in the softmax network layer, and the sub-embedding layers corresponding to the preset number of languages in the language embedding layer correspond one to one respectively.

[0076] For example, the preset number of languages includes language A, language B, and language C. Correspondingly, the language embedding layer includes a sub-embedding layer corresponding to language A, a sub-embedding layer corresponding to language B, and a sub-embedding layer corresponding to language C. The joint network layer includes a sub-joint network layer corresponding to language A, a sub-joint network layer corresponding to language B, and a sub-joint network layer corresponding to language C. The softmax network layer includes a sub-softmax network layer corresponding to language A, a sub-softmax network layer corresponding to language B, and a sub-softmax network layer corresponding to language C.

[0077] In an embodiment of the present application, the language embedding layer is part of the prediction network. In other embodiments, the language embedding layer can also be an independent module, that is, the prediction network does not include the language embedding layer. Regardless of whether the language embedding layer is part of the prediction network, the roles of the language embedding layer and the prediction network layer remain unchanged. The functions implemented by the sub-union network layer and the sub-softmax network layer of the same language are the same as those implemented by the current joint network. Specifically, the functions of the sub-union network layer and the sub-softmax network layer will not be introduced in the following text. Among them, the decoding module corresponding to the corresponding language may not be part of the preset speech recognition model, or it may be part of the preset speech recognition model. Regardless of whether the decoding module of the corresponding language is part of the preset speech recognition model, the function implemented by the decoding module of the corresponding language remains unchanged.

[0078] If the preset speech recognition model implements end-to-end, switch-free speech recognition for both a dialect and Mandarin, the preset number of languages includes Mandarin and at least one dialect. If the preset speech recognition model implements end-to-end, switch-free speech recognition for multiple dialects, the preset number of languages includes multiple dialects. The preset number of languages depends on the speech recognition functionality implemented in a specific scenario. For example, the preset number of languages may include Mandarin, English, French, etc.

[0079] Figure 3 This is a flow chart of the speech recognition method provided by the embodiment of the present application. Figure 2 The preset speech recognition model shown, the speech recognition method includes the following steps.

[0080] 101. Receive a selected language and obtain a preset speech recognition model, wherein the selected language is at least one language among a preset number of languages in the preset speech recognition model.

[0081] For example, a computer device may provide an interactive interface on which a target language may be selected, or a target language may be selected through other means. The computer device receives the selected target language, which is at least one of a preset number of languages, for example, Mandarin or a dialect. The preset speech recognition model may recognize speech data in the preset number of languages.

[0082] Obtain a preset speech recognition model, which performs speech recognition on speech data of a predetermined number of languages, the preset speech recognition model including a prediction network, an encoding network and a joint network, the prediction network including a language embedding layer and a prediction network layer, and the joint network including a joint network layer and a softmax network layer.

[0083] The structure of the preset speech recognition model in this embodiment is as follows Figure 2 As shown, the encoding network, which can also be called an encoder, is used to encode each frame of speech data in the audio data to be recognized to obtain encoding features and / or encoding vectors. The prediction network is used to predict the speech recognition results of the previous frame of speech data to obtain prediction features and / or prediction vectors. The joint network is used to perform joint mapping processing on the merged features after the prediction features and the encoding features are fused to obtain the mapping probability of the current frame of speech data corresponding to each language. The decoding module is used to decode and recognize the mapping probability of each language to obtain the mapping probability of the speech data corresponding to each language.

[0084] Among them, the language embedding layer includes sub-embedding layers corresponding to a preset number of languages, the joint network layer includes sub-joint network layers corresponding to a preset number of languages, the softmax network layer includes sub-softmax network layers corresponding to a preset number of languages, and the decoding module includes sub-decoding modules corresponding to a preset number of languages.

[0085] 102. Receive input audio data to be recognized.

[0086] The audio data to be recognized may be audio data input by a user. For example, a computer device may provide an interactive interface that allows the user to input the audio data to be recognized or select pre-set audio data to be recognized. The audio data to be recognized may be pre-set audio data to be recognized, etc. The computer device receives the input audio data to be recognized. The audio data includes multiple frames of speech data.

[0087] 103. Perform language recognition processing on the audio data using a preset speech recognition model to obtain a target language corresponding to the audio data, and perform speech recognition processing on the audio data using an independent parameter module and a shared parameter module of the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in the decoding module to obtain an audio recognition result corresponding to the target language of the audio data, wherein the target language is at least one of the selected languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules of a preset number of languages.

[0088] Among them, the independent parameter module refers to a module related to a specific language. The independent parameter module corresponding to each language includes a sub-embedding layer corresponding to each language, a sub-joint network layer corresponding to each language, and a sub-softmax network layer corresponding to each language. If the decoding module also belongs to a module in the preset speech recognition model, the independent parameter module corresponding to each language also includes a sub-decoding module corresponding to each language. In the embodiment of the present application, the decoding module does not belong to the preset speech recognition module as an example. For example, the independent parameter module corresponding to the target language includes a sub-embedding layer corresponding to the target language, a sub-joint network layer corresponding to the target language, a sub-softmax network layer corresponding to the target language, etc.

[0089] The independent parameter module sets independent parameters according to the language, so that the independent parameter module can reflect the information of the language to the greatest extent, improve the accuracy, and improve the accuracy of speech recognition.

[0090] Among them, the shared parameter module is a module that can be shared by a preset number of languages. Figure 2 In the preset speech recognition model shown, the shared parameter module includes a prediction network layer and an encoding network.

[0091] The shared parameter module uses common model parameters, that is, regardless of the language, the corresponding model parameters remain unchanged, or it can also be understood as shared parameter module reuse or sharing, thus reducing model parameters, improving speech recognition efficiency, and increasing training speed.

[0092] In one embodiment, if the selected language includes one, the target language is the selected language, that is, the target language is one language. Correspondingly, the language recognition processing is performed on the audio data using the preset speech recognition model, and the target language determined is the selected language. Figure 4 As shown, the above step 103 includes the following steps.

[0093] 201 , obtaining speech data to be recognized in a current frame of audio data and a speech recognition result in a frame previous to the current frame.

[0094] Since the audio data includes multiple frames of speech data, when the preset speech recognition model is used for processing, the data is processed frame by frame.

[0095] The current frame can be understood as the t-th frame of audio data x or the t-th moment of audio data x, and the previous frame of the current frame can be understood as the t-1-th frame of audio data x or the t-1-th moment of audio data x. The speech data to be recognized in the current frame is the speech information to be recognized in the current frame, such as Figure 2 In the middle, use x t The speech recognition result of the previous frame is represented by yu-1, and the speech recognition result of the previous frame includes information such as the characters output in the corresponding language and / or the positions of the characters output in the corresponding language.

[0096] 202. Merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the target language to obtain a merged feature.

[0097] Among them, the encoding network is used to encode the speech data to obtain the encoding features, the sub-embedding layer and prediction network layer corresponding to the target language are used to predict the speech recognition results of the previous frame of the current frame to obtain the prediction features, and the encoding features and the prediction features are merged to obtain the merged features.

[0098] In one embodiment, the preset speech recognition model further includes a feature extraction layer. Specifically, the feature extraction layer performs acoustic feature processing on speech data to obtain acoustic features, and the encoding network encodes the acoustic features to obtain coded features. This embodiment of the present application uses this case as an example for illustration. That is, the input of the feature extraction layer is speech data, and the output is acoustic features. The input of the encoding network is acoustic features, and the output is coded features. The coded features of the speech data can be 1*512 dimensions, or other dimensions.

[0099] In one embodiment, the feature extraction layer may also be part of the encoding network. Accordingly, the encoding network includes a feature extraction layer and an encoding network layer. The feature extraction layer processes the acoustic features of speech data to obtain acoustic features, and the encoding network layer encodes the acoustic features to obtain encoded features. Whether the feature extraction layer is an independent module or part of the encoding network, their functions remain unchanged.

[0100] In one embodiment, if the preset speech recognition model does not include a feature extraction layer, the speech data of the current frame is the data corresponding to the acoustic features obtained using the feature extraction layer.

[0101] Among them, the encoding network uses shared model parameters, that is, regardless of the language, the corresponding model parameters remain unchanged, or it can also be understood as encoding network reuse or sharing, thus reducing model parameters, improving speech recognition efficiency, and increasing training speed.

[0102] Because the prediction network includes a language embedding layer and a prediction network layer, and the language embedding layer includes a predetermined number of sub-embedding layers corresponding to different languages, the speech recognition result of the previous frame may include the speech recognition result corresponding to the target language. Accordingly, the prediction features are obtained by predicting the speech recognition result of the previous frame using the sub-embedding layers and the prediction network layer corresponding to the target language.

[0103] In which, the speech recognition result of the previous frame is input into the sub-embedding layer corresponding to the target language in the language embedding layer. For example, if the target language is language A, the speech recognition result corresponding to language A in the speech recognition result of the previous frame is input into the sub-embedding layer corresponding to language A in the language embedding layer. If the target language is language B, the speech recognition result corresponding to language B in the speech recognition result of the previous frame is input into the sub-embedding layer corresponding to language B in the language embedding layer. Then, the prediction network layer of the prediction network is used to perform prediction processing on the embedding layer features of the target language to obtain the prediction features of the target language, that is, the embedding layer features of the target language are input into the prediction network layer for prediction processing to obtain the prediction features of the target language. In which, the prediction processing of the prediction network layer is essentially to convert the embedding layer features into a high-order representation. For example, after the target language is processed by the prediction network layer, the features obtained can be 1*512 dimensions or other dimensions.

[0104] In the first frame of audio data x or at the first moment of audio data, since the speech recognition result of the previous frame is empty, the output result of the prediction network, i.e., the predicted feature, is a preset value, such as 0 or empty. Alternatively, an empty character is input to the prediction network, and the prediction network performs prediction processing to obtain the corresponding predicted feature. For other frames or other moments, please refer to the methods described in this application.

[0105] Among them, since the language embedding layer includes sub-embedding layers corresponding to a preset number of languages, each sub-embedding layer corresponds to one language, that is, each sub-embedding layer sets independent parameters according to the language, so that the features obtained after processing by the sub-embedding layer reflect the information of the language to the greatest extent, thereby improving the accuracy of the embedding layer features and the accuracy of speech recognition. The encoding network and the prediction network layer use a shared method to reduce model parameters, or it can also be understood as the reuse of the encoding network and the prediction network layer to improve the efficiency of speech recognition.

[0106] Among them, the above-mentioned step of using the encoding network to encode the speech data to obtain encoding features, and the step of using the sub-embedding layer and prediction network layer corresponding to the target language to predict the speech recognition results of the previous frame of the current frame to obtain prediction features, can be executed in parallel to improve the efficiency of speech recognition.

[0107] After obtaining the prediction features and the coding features, the prediction features and the coding features are merged to obtain a merged feature. The merging process includes superposition and splicing, and may also be other processing methods.

[0108] For example, to superimpose the predicted and encoded features, we can obtain a merged feature by adding the values of the corresponding rows and columns of the predicted and encoded features to the values of the corresponding rows and columns of the merged feature. If the predicted feature is 1*512 dimensional and the encoded feature is 1*512 dimensional, the merged feature obtained after superimposition is still 1*512 dimensional.

[0109] For example, if the prediction feature and the encoding feature are concatenated to obtain a merged feature, that is, if the prediction feature is 1*512 dimensional and the encoding feature is 1*512 dimensional, then the merged feature obtained after the concatenation process is 1*1024 dimensional.

[0110] The prediction features and the coding features are merged to obtain the merged features. In this embodiment, it can be understood that the prediction features and the coding features corresponding to the target language when the current frame is input are merged to obtain the merged features corresponding to the target language.

[0111] 203. Using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer, the merged features are jointly mapped to obtain a mapping probability that the speech data of the current frame corresponds to the target language.

[0112] In the joint network layer and softmax network layer, there are sub-joint network layers for each language and sub-softmax network layers for each language respectively. The model parameters in the sub-joint network layer for each language are only related to the corresponding language and are not related to other languages. The same is true for the sub-softmax network layer for each language.

[0113] When the language to be processed is the target language, the merged features are jointly mapped using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer to obtain the mapping probability that the speech data of the current frame corresponds to the target language. It can be understood that the merged features are input into the sub-joint network layer corresponding to the target language in the joint network layer, and after processing by the sub-joint network layer, they are input into the sub-softmax network layer corresponding to the target language in the softmax network layer for processing to obtain the mapping probability that the speech data of the current frame corresponds to the target language.

[0114] The joint mapping process includes joint processing and normalized mapping processing.

[0115] Correspondingly, step 203 includes: using the sub-joint network layer corresponding to the target language in the joint network layer to jointly process the merged features to obtain the joint features of the speech data corresponding to the target language; using the sub-softmax network layer corresponding to the target language in the softmax network layer to perform normalized mapping processing on the joint features of the target language to obtain the mapping probability that the speech data of the current frame corresponds to the target language.

[0116] For Mandarin (Chinese), the modeling unit of the joint network is generally at the character level or syllable level. For example, there are 3,000 commonly used Chinese characters. Common characters are used as modeling units, and blank characters, start characters ( <s> ) and the end character (< / s> ) has a total of 3003 output nodes. The joint network is used to map the joint features obtained by jointly processing the merged features to the output space of the modeling unit. For example, for Mandarin, after normalized mapping processing, the resulting mapping probability is a probability value with a dimension of 1*3003.

[0117] This step reduces language confusion through a language-bound joint network layer and a softmax network layer, takes into account the differences in grammar and pronunciation of different languages, achieves decoding decoupling between languages, and improves the accuracy of speech recognition.

[0118] 204 , using a sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to the target language of the speech data.

[0119] Among them, the decoding module includes sub-decoding modules corresponding to a preset number of languages, and different languages correspond to different sub-decoding modules. When the language to be processed is the target language, the sub-decoding module corresponding to the target language is used to perform decoding and recognition processing on the mapping probability of the target language to obtain the speech recognition result of the speech data corresponding to the target language. The decoding and recognition processing is used to map the mapping probability after normalized index processing to the corresponding character / Chinese according to the corresponding decoding method. The decoding method can use greedy decoding, beam search decoding, or other decoding methods. Among them, the speech recognition result of the speech data of the current frame corresponding to the target language can be used u To express.

[0120] Among them, the decoding module can be a module independent of the preset speech recognition model. The embodiment of the present application takes this case as an example to illustrate. The decoding module can also be part of the preset speech recognition model. Regardless of whether the decoding module is part of the preset speech recognition model, its function is to decode and recognize the mapping probability, which will not be repeated below.

[0121] After obtaining the speech recognition result of the target language corresponding to the speech data of the current frame, the speech recognition result of the current frame is used as the speech recognition result of the previous frame, and then the speech data of the next frame of audio data x is recognized until the end symbol is output or the audio data recognition is completed, then the recognition of audio data x is terminated. That is, the above steps 201 to 204 are a loop execution process until the current speech recognition is completed to obtain the speech recognition result corresponding to each frame of speech data of the audio data.

[0122] When only one language is selected, that is, the target language is the selected language, the shared encoding network and prediction network layers can be shared, and only the target language is jointly mapped. At this time, the preset speech recognition model can be used by the current monolingual model, and the effect is the same as the monolingual model.

[0123] 205 : Using the speech recognition result of the speech data in each frame corresponding to the target language as the audio recognition result of the audio data corresponding to the target language.

[0124] 104. Return the audio recognition result.

[0125] The audio recognition results corresponding to the voice data of all frames of the audio data may be returned at once, or the voice recognition results corresponding to the voice data of each frame may be returned frame by frame to return the audio recognition results. In one embodiment, the returned audio recognition results may also be displayed.

[0126] This embodiment enables end-to-end multi-language speech recognition without switching. A preset speech recognition model is uniformly established for a preset number of languages. The preset speech recognition model includes independent parameter modules and shared parameter modules corresponding to the preset number of languages. The independent parameter modules corresponding to the preset number of languages correspond to the languages respectively. Shared parameter modules, such as the encoding network and the prediction network layer, share model parameters. Thus, regardless of the language of the input speech data to be recognized, the shared parameter modules can be utilized to reduce the model parameters of the preset speech recognition model. Since recognition of a preset number of languages can be achieved within a single model, the recognition efficiency of the preset speech recognition model is improved compared to parallel calculations of multiple single-language recognition systems. Furthermore, since the independent parameter modules are language-bound, that is, independent parameter modules such as the species embedding layer, the joint network, and the softmax network are set independently according to the language, mutual influence between different languages is avoided, language confusion is reduced, and differences in grammar and phrasing between different languages are taken into account, thereby improving the accuracy of multi-language speech recognition.

[0127] In order to further improve the speech recognition efficiency of the preset speech recognition model, the present invention is to Figure 2 The preset speech recognition model shown in the figure is further improved. Figure 2 The preset speech recognition model shown in the figure adds a language classification network, which can also be called a language discrimination network, a language classification layer, a language discrimination layer, etc. Specifically, a language classification network is added after the encoding network, as shown in the following example. Figure 5 shown.

[0128] The language classification network is used to determine the posterior probability of each language in the speech data at the frame level, and based on the posterior probabilities, to determine confidence data for each language. Based on the confidence data, it is then determined whether there is a first language that does not require decoding and recognition processing. This first language is at least one of the selected languages. Simply put, the language classification network is used to determine whether decoding and recognition processing for a particular language should be discarded in each frame, thereby eliminating decoding and recognition processing for that language from the discarded frame onwards. Alternatively, it is used to determine whether a language exists in each frame that cannot be the final language.

[0129] In one embodiment, if the selected language includes multiple languages, such as Figure 6 As shown, the above step 103 includes the following steps.

[0130] 301 , obtaining speech data to be recognized in a current frame of audio data and a speech recognition result in a frame previous to the current frame.

[0131] 302. Merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the selected language to obtain a merged feature.

[0132] Among them, the encoding network is used to encode the speech data to obtain the encoding features, the sub-embedding layer and prediction network layer corresponding to the selected language are used to predict the speech recognition result of the previous frame to obtain the prediction features, and the encoding features and the prediction features are merged to obtain the merged features.

[0133] Among them, the speech recognition result of the previous frame includes the speech recognition result corresponding to the selected language, and the step of using the sub-embedding layer and prediction network layer corresponding to the selected language to predict the speech recognition result of the previous frame to obtain prediction features includes: using the sub-embedding layer corresponding to the selected language in the language embedding layer to perform feature processing on the speech recognition result corresponding to the target language in the previous frame to obtain the embedding layer features corresponding to the selected language; using the prediction network layer to predict the embedding layer features of the selected language to obtain the prediction features of the selected language.

[0134] Among them, the prediction features and the coding features are merged to obtain the merged features, which can also be understood as merging the prediction features and the coding features corresponding to the selected language, or merging the prediction features and the coding features corresponding to each language when the current frame is input to obtain the merged features corresponding to each language.

[0135] For example, when the languages corresponding to the current frame input include language A and language B, the prediction features and encoding features corresponding to language A are merged to obtain the merged features corresponding to language A, and the prediction features and encoding features corresponding to language B are merged to obtain the merged features corresponding to language B.

[0136] Among them, all the languages corresponding to the input of the current frame vary from frame to frame. For example, the preset number of languages is 5, and the corresponding languages are 5 when the 10th frame is input. In the 10th frame, 1 first language is determined, that is, 1 language is eliminated, so the candidate languages in the 10th frame are 4. Then, in the 11th frame, the corresponding languages are 4 when the 11th frame is input. If a new first language is determined in the 11th frame, that is, 1 language is eliminated, then the candidate languages in the 11th frame are 3. This shows that in different frames, the corresponding languages may be different, the determined first language may be different, and the candidate languages may also be different.

[0137] 303 , performing language recognition processing on the speech data using a language classification network to obtain the target language corresponding to the audio data.

[0138] The language classification network is used to classify the coding features to determine whether there is a first language that does not require decoding and recognition processing, where the first language is at least one of the selected languages. If so, decoding and recognition processing is not performed on the first language, and the remaining languages in the current frame after removing the first language are used as target languages. If not, all languages in the current frame are used as target languages, that is, the language used when the current frame was input is used as the target language. The input to the language classification network is the coding features of the selected language or the coding features of the language corresponding to the current frame input. The language of the current frame may vary depending on the number of frames.

[0139] Among them, the above-mentioned step of using a language classification network to perform language classification processing on the coding features to determine whether there is a first language that does not require decoding and recognition processing includes: using the language classification network to perform confidence processing on the coding features to obtain confidence data of a preset number of languages in the current frame; and determining whether there is a first language that does not require decoding and recognition processing based on the confidence data of the preset number of languages in the current frame.

[0140] In one embodiment, the step of performing confidence processing on the encoding features using a language classification network to obtain confidence data for a preset number of languages in the current frame includes: performing posterior probability processing on the encoding features using the language classification network to obtain posterior probabilities for the preset number of languages in the current frame, and determining confidence data for the preset number of languages in the current frame based on the posterior probabilities for the preset number of languages in the current frame. The confidence data for languages not selected from the preset number of languages may be 0 or very low.

[0141] The coding features are input into a language classification network, which can determine the posterior probabilities of a preset number of languages in the current frame based on the coding features. The posterior probabilities can be calculated using any of the existing methods, which will not be described in detail here.

[0142] After obtaining the posterior probabilities for a preset number of languages in the current frame, confidence data corresponding to the preset number of languages is determined based on the posterior probabilities for the preset number of languages. For example, the average of the posterior probabilities for each of the preset number of languages in the current frame and all previous frames is determined, and the average is used as the confidence data for the corresponding language in the current frame. In this manner, confidence data for each of the preset number of languages is obtained.

[0143] The confidence data of each of the preset number of languages can be determined according to formula (1).

[0144]

[0145] Among them, l represents different languages, t represents the number of audio frames, T represents the number of all audio frames corresponding to the current frame, and p t (l) represents the posterior probability of language l at time t / frame t, P l (t) represents the confidence data of language l at time t / frame t.

[0146] After obtaining the confidence data, a determination is made based on the confidence data as to whether a first language that does not require decoding and recognition processing exists. Specifically, a determination is made as to whether confidence data for a language in the current frame satisfies a first preset confidence condition, or whether confidence data for a language in the current frame within a preset frame satisfies a second preset confidence condition. If so, the existing language is treated as the first language, and the presence of a first language that does not require decoding and recognition processing is determined. Otherwise, a determination is made as to whether each language in the current frame belongs to the language corresponding to the speech data.

[0147] In one embodiment, the first preset confidence condition includes being less than a first confidence threshold, that is, if the confidence data of a language in the current frame is less than the first confidence threshold, it is considered that the existing language cannot be the language of the final speech recognition, and the existing language is used as the first language. Otherwise, if the confidence data of no language in the current frame is less than the first confidence threshold, it is determined that each language in the current frame belongs to the language corresponding to the speech data, that is, each language in the current frame may be the language of the final speech recognition.

[0148] In one embodiment, the first preset confidence condition may also be that the average value of the confidence data of the language minus the confidence data of a preset number of languages in the current frame is less than a second preset threshold value, that is, if the average value of the confidence data of the language in the current frame minus the confidence data of the preset number of languages in the current frame is less than the second preset threshold value, it is considered that the corresponding language cannot be the language of the final speech recognition, and the corresponding language is used as the first language. Otherwise, if the average value of the confidence data of the language that does not exist in the current frame minus the confidence data of the preset number of languages in the current frame is less than the second preset threshold value, it is determined that each language in the current frame belongs to the language corresponding to the speech data, that is, each language in the current frame may be the language of the final speech recognition.

[0149] In one embodiment, the maximum posterior probability value among the posterior probabilities corresponding to the preset number of languages in each frame of the current frame and all previous frames can be determined first, and then the average value of the maximum posterior probability values in the current frame and all previous frames can be determined. The average value is calculated using P lmax (t) is used to represent the first preset confidence condition, which can also be: the confidence data P of a preset number of languages in the current frame l (t) and the average value P lmax (t) The difference is greater than the third confidence threshold. If the language confidence data P exists in the previous frame l (t) and the average value P lmax (t) If the difference is greater than the third confidence threshold, it means that the corresponding language cannot be the language of the final speech recognition, and the corresponding language is used as the first language. Otherwise, if there is no confidence data P for the language in the current frame, l (t) and the average value P lmax (t) If the difference is greater than the third preset threshold, it is determined that each language in the current frame belongs to the language corresponding to the speech data, that is, each language in the current frame may be the language of the final speech recognition.

[0150] Among them, p lmax (t) can be calculated as shown in formula (2).

[0151]

[0152] Among them, p max (t) represents the maximum value of the confidence data of a preset number of languages in time t / frame t, P lmax (t) represents the average value of the maximum values of the confidence data of a preset number of languages at time t / frame t.

[0153] Among them, it is assumed that at time t / frame t, the confidence data corresponding to language A, language B, and language C are p A 、p B 、p C , then we have the data in Table 1 below.

[0154] Table 1 P l (t)-P max (t) Example

[0155] t 1 2 3 4 <![CDATA[p t (l)]]> <![CDATA[p t (l)-p lmax (t)]]> <![CDATA[p A ]]> 0.3 0.8 1 1 0.775 0.05 <![CDATA[p B ]]> 0.5 0.1 0 0 0.15 0.675 pC 0.3 0.1 0 0 0.1 0.725 <![CDATA[p lmax (t)]]> 0.5 0.8 1 1 <![CDATA[P max (t)=0.825]]>

[0156] In one embodiment, the second preset confidence condition includes being less than a fourth confidence threshold, and the preset frame may be one frame, five frames, 30 frames, or other data. That is, if there is a language in the current frame whose confidence data in the preset frame is less than the fourth confidence threshold, it is considered that the existing language cannot be the language of the final speech recognition, and the existing language is used as the first language. Otherwise, if there is no language in the current frame whose confidence data in the preset frame is less than the fourth confidence data threshold, it is determined that each language in the current frame belongs to the language corresponding to the speech data, that is, each language in the current frame may be the language of the final speech recognition.

[0157] In one embodiment, the second preset confidence condition may also be that the average value of the confidence data of the language in the preset frame minus the confidence data of a preset number of languages in the current frame is less than the fifth confidence threshold, that is, if there is a language in the current frame whose confidence data in the preset frame minus the average value of the confidence data of a preset number of languages in the current frame is less than the fifth confidence threshold, it is considered that the corresponding language cannot be the language of the final speech recognition, and the corresponding language is used as the first language; otherwise, it is determined that each language in the current frame belongs to the language corresponding to the speech data, that is, each language in the current frame may be the language of the final speech recognition.

[0158] In one embodiment, the second preset confidence condition may also be that P l (t)-P lmax The values of (t) are all greater than the sixth confidence threshold, that is, if there is a language in the current frame P in the preset frame l (t)-P lmax(t) are all greater than the sixth confidence threshold, then it is considered that the corresponding language cannot be the language of the final speech recognition, and the corresponding language is used as the first language; otherwise, it is determined that each language in the current frame belongs to the language corresponding to the speech data, that is, each language in the current frame may be the language of the final speech recognition.

[0159] Among them, the first confidence threshold, the second confidence threshold, the third confidence threshold, the fourth confidence threshold, the fifth confidence threshold, and the sixth confidence threshold can be pre-set, and the first confidence threshold and the second confidence threshold can be the same or different, for example, the first confidence threshold is greater than the second confidence threshold, the fourth confidence threshold and the fifth confidence threshold can be the same or different, for example, the fourth confidence threshold is greater than the fifth confidence threshold, the third confidence threshold and the sixth confidence threshold can be the same or different, for example, the sixth confidence threshold is greater than the third confidence threshold.

[0160] The first preset reliability condition and the second preset reliability condition may also be other conditions.

[0161] In the embodiment of the present application, determining whether there is a first language that does not require decoding and recognition processing can also be understood as whether there is a new first language that does not require decoding and recognition processing.

[0162] 304 , using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer, jointly map the merged features to obtain a mapping probability of the speech data corresponding to the target language.

[0163] Specifically, the merged features are jointly processed using the sub-joint network layer corresponding to the target language in the joint network layer to obtain the joint features corresponding to the target language; and the sub-softmax network layer corresponding to the target language in the softmax network layer is used to perform normalized mapping processing on the joint features of the corresponding language to obtain the mapping probability of the speech data corresponding to the target language.

[0164] That is, the merged features are used as the input of the sub-joint network layer corresponding to the target language in the joint network layer, and after joint processing using the sub-joint network layer corresponding to the target language, the joint features of the target language are output, and the joint features of the target language are used as the input of the sub-softmax network layer corresponding to the target language in the softmax network layer, and after normalized mapping processing using the sub-softmax network layer corresponding to the target language, the mapping probability of the current frame of speech data corresponding to the target language is output.

[0165] For example, if the languages of the current frame input include Language A, Language B, and Language C, the first language determined is Language A, and joint mapping processing is not performed on the merged features corresponding to Language A. If the target languages are Language B and Language C, joint mapping processing is performed on the merged features corresponding to Language B and Language C. Language A will not be processed in subsequent frames.

[0166] This step reduces language confusion through a language-bound joint network layer and a softmax network layer, takes into account the differences in grammar and pronunciation of different languages, achieves decoding decoupling between languages, and improves the accuracy of speech recognition.

[0167] 305 , using the sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to the target language of the speech data.

[0168] For example, if the target languages are Language B and Language C, the sub-decoding modules corresponding to Language B and Language C in the decoding module are used to perform decoding and recognition processing on the mapping probabilities of Language B and Language C respectively, so as to obtain speech recognition results corresponding to Language B and Language C respectively.

[0169] 306 : Using the speech recognition result of the speech data in each frame corresponding to the target language as the audio recognition result of the audio data corresponding to the target language.

[0170] Among them, by adding a language classification network to the preset speech recognition model, a part of the joint mapping processing and decoding recognition processing calculations that may not be the final language can be eliminated, reducing the amount of invalid calculations in the joint network layer, softmax network layer, and decoding module, thereby improving the efficiency of speech recognition.

[0171] For example, if the languages input in the current frame include Language A, Language B, and Language C, the first language determined is Language A. Decoding and recognition processing is not performed on the mapping probabilities corresponding to Language A. The target languages are Language B and Language C. The sub-decoding modules corresponding to Language B and Language C in the decoding module are used to perform decoding and recognition processing on the mapping probabilities of Language B and Language C, respectively, to obtain speech recognition results corresponding to Language B and Language C, respectively. Language A will not be processed in subsequent frames.

[0172] In one embodiment, after performing language recognition processing on speech data using a language classification network to obtain the target language corresponding to the audio data, a joint mapping process is performed on the merged features of the language of the current frame using a sub-joint network layer corresponding to the language of the current frame in the joint network layer and a sub-softmax network layer corresponding to the language of the current frame in the softmax network layer to obtain a mapping probability of the speech data corresponding to the language of the current frame. The sub-decoding module corresponding to the target language in the decoding module then performs decoding and recognition processing on the mapping probability to obtain a speech recognition result indicating that the speech data corresponds to the target language. In this embodiment, the language recognition process and the joint mapping process can be performed in parallel. This embodiment can also be understood as not performing decoding and recognition processing on the first language after the preset classification network determines that there is a first language that does not require decoding and recognition processing. In this way, decoding and recognition processing is not performed on the first language, which is unlikely to become the final language. This reduces the amount of inefficient computation in the decoding and recognition process and improves the efficiency of speech recognition.

[0173] like Figure 7 FIG. 1 is another structural diagram of a preset speech recognition model provided in an embodiment of the present application. The preset speech recognition model in this embodiment is relative to Figure 2 For the preset speech recognition model shown (with improved language embedding layer and joint network), the preset speech recognition model in this embodiment further adds valid frame nodes in the joint network.

[0174] The valid frame node is obtained through pre-training. Among them, the valid frame node is trained to be activated when the output is a non-empty character, indicating that the merged features calculated for the frame need to be input into the corresponding sub-union network layer and sub-softmax network layer for processing. When the output is an empty character, the valid frame node is not activated. The calculation of the merged features of the input by the valid frame node is performed in parallel with the calculation of the merged features of the input by the sub-union network layer and sub-softmax network layer. It should be noted that the valid frame node in the sub-union network layer and sub-softmax network layer is one valid frame node.

[0175] The reason for adding valid frame nodes is that the RNN-T model outputs empty characters in most frames, but only a few frames output valid characters. Sending the mapping probability of each frame to the corresponding decoding module for decoding and recognition processing would incur a large amount of computational overhead. Therefore, adding valid frame nodes further improves speech recognition efficiency and avoids the large computational overhead of decoding and recognition processing for mapping probabilities when empty characters are present.

[0176] In one embodiment, if Figure 8 As shown, the above step 103 may also include the following steps. Figure 7 The preset speech recognition model shown specifically includes the following steps.

[0177] 401 , obtaining speech data to be recognized in a current frame of audio data and a speech recognition result in a frame previous to the current frame.

[0178] 402. Merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the selected language to obtain a merged feature.

[0179] 403. Using the sub-joint network layer corresponding to the selected language in the joint network layer and the sub-softmax network layer corresponding to the selected language in the softmax network layer, the merged features are jointly mapped to obtain a mapping probability that the speech data of the current frame corresponds to the selected language.

[0180] 404 , performing language recognition processing on the speech data using the valid frame nodes to obtain the target language corresponding to the audio data.

[0181] Wherein, step 403 and step 404 are executed in parallel, and the inputs of step 403 and step 404 are both the combined features of the selected language or the combined features of the language when the current frame is input.

[0182] In one embodiment, step 404 includes: using a valid frame node to perform valid frame recognition on the merged features of each language of the current frame to obtain a valid frame recognition result for each language of the current frame; when there is a valid frame recognition result of a second language indicating that the current frame is an invalid frame corresponding to the second language, the remaining languages excluding the second language in the languages of the current frame are used as the target language; otherwise, all languages of the current frame are used as the target language, wherein the current frame being an invalid frame means that the speech data corresponding to the current frame is an empty character, and wherein the second language is at least one of the selected languages.

[0183] Among them, the above-mentioned step of using the valid frame node to perform valid frame recognition on the combined features of each language of the current frame to obtain the valid frame recognition result of each language of the current frame includes: using the valid frame node to perform valid frame recognition processing on the combined features of each language of the current frame to obtain the valid frame recognition probability of each language; and determining the valid frame recognition result of each language based on the valid frame recognition probability of each language.

[0184] For example, if the languages of the current frame are Language A, Language B, and Language C, after obtaining the merged features corresponding to Language A, Language B, and Language C, the merged features corresponding to Language A are input into the valid frame node for valid frame recognition processing, resulting in a valid frame recognition probability for Language A, such as 0.7. The merged features corresponding to Language B are input into the valid frame node for valid frame recognition processing, resulting in a valid frame recognition probability for Language B, such as 0.9. The merged features corresponding to Language C are input into the valid frame node for valid frame recognition processing, resulting in a valid frame recognition probability for Language C, such as 0.95. Based on the valid frame recognition probability for each language, the valid frame recognition result for each language is determined.

[0185] The detection of whether there is a valid frame of the second language and the recognition result indicates that the current frame is an invalid frame corresponding to the second language, wherein the current frame being an invalid frame means that the voice data corresponding to the current frame is a null character.

[0186] Among them, it is determined whether the valid frame recognition probability of each language is less than the preset valid frame recognition probability; if so, the current frame is determined to be an invalid frame of the corresponding language, that is, the voice data corresponding to the current frame is equivalent to an empty character for the corresponding language; otherwise, the current frame is determined to be a valid frame of the corresponding language, that is, the voice data corresponding to the current frame is a non-empty character relative to the corresponding language.

[0187] Continuing with the example above, assuming the preset valid frame recognition probability is 0.8, the current frame is determined to be a valid frame for language B (valid frame recognition probability is 0.9) and language C (valid frame recognition probability is 0.95), and is an invalid frame for language A (valid frame recognition probability is 0.7).

[0188] Since the current frame is an invalid frame for language A, there is a second language, which is language A. The corresponding target languages are language B and language C excluding language A.

[0189] If there is no valid frame recognition result of the second language, indicating that the current frame is an invalid frame corresponding to the second language, the mapping probabilities of all languages when the current frame is input are decoded and recognized to obtain speech recognition results corresponding to all languages when the speech data is input.

[0190] 405 , using the sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability of the target language to obtain a speech recognition result corresponding to the target language of the speech data.

[0191] The second language is language A, and the mapping probability corresponding to language A is not decoded and recognized. The target languages are language B and language C. The sub-decoding modules corresponding to language B and language C in the decoding module are used to decode and recognize the mapping probabilities of language B and language C respectively, so as to obtain speech recognition results corresponding to language B and language C respectively.

[0192] In this embodiment, the above steps 401 to 405 are executed in a loop until the audio data processing is completed.

[0193] 406 : Using the speech recognition result of the speech data in each frame corresponding to the target language as the audio recognition result of the audio data corresponding to the target language.

[0194] In this embodiment, valid frame nodes are used to perform valid frame recognition processing on the merged features to obtain the valid frame recognition probability of each language. Based on the valid frame recognition probability, the valid frame recognition result of each language is determined. In this way, for languages with valid frame recognition results as valid frames, decoding recognition processing is performed, while for languages with invalid frame recognition results as invalid frames, decoding recognition processing is not performed. In this way, the computational complexity of decoding recognition processing is reduced and the efficiency of speech recognition is improved.

[0195] like Figure 9 FIG. 1 is another structural diagram of a preset speech recognition model provided in an embodiment of the present application. The preset speech recognition model in this embodiment is relative to Figure 2 For the preset speech recognition model (with an improved language embedding layer and joint network), this embodiment also adds a language classification network and a valid frame node to the joint network. The functions of the language classification network and the valid frame node are described separately above and will not be repeated here.

[0196] In one embodiment, if Figure 10 As shown, the above step 103 may also include the following steps. Figure 9 The preset speech recognition model shown is used to perform speech recognition, which specifically includes the following steps.

[0197] 501. Acquire speech data to be recognized in a current frame of audio data and a speech recognition result in a frame previous to the current frame.

[0198] 502. Merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the selected language to obtain a merged feature.

[0199] 503. Using the sub-joint network layer corresponding to the selected language in the joint network layer and the sub-softmax network layer corresponding to the selected language in the softmax network layer, the merged features are jointly mapped to obtain a mapping probability that the speech data corresponds to the selected language.

[0200] 504 , performing language recognition processing on the speech data using the language classification network and the valid frame nodes to obtain the target language corresponding to the audio data.

[0201] In one embodiment, step 504 includes: performing language classification processing on the encoded features using a language classification network to obtain confidence data for a preset number of languages in the current frame; performing effective frame identification on the merged features of each language in the current frame using an effective frame node to obtain an effective frame identification result for each language in the current frame; determining whether there is a third language that does not require decoding and identification based on the confidence data and the effective frame identification result for each language; if there is a third language, then decoding and identification processing is not performed on the third language in the current frame, and the remaining languages in the current frame after excluding the third language are used as target languages, wherein the third language is at least one of the selected languages; if there is no third language, then all languages in the current frame are used as target languages.

[0202] The step of determining whether there is a third language that does not require decoding and recognition based on the confidence data and the valid frame recognition results of each language includes: whether there is confidence data for the corresponding language that meets a first preset confidence condition, and at the same time, the valid frame recognition probability of the corresponding language is less than a preset valid frame recognition probability; if so, then the existing language is regarded as a third language that does not require decoding and recognition.

[0203] In this embodiment, the language classification network is used to obtain confidence data of a preset number of languages. Correspondingly, during training, the speech classification network is also trained to obtain training confidence data of only a preset number of languages.

[0204] In one embodiment, step 504 includes: performing language classification processing on the encoded features using a language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one of the selected languages; performing valid frame recognition on the merged features of each language in the current frame using a valid frame node to obtain a valid frame recognition result for each language; and determining whether there is a valid frame recognition result for a second language based on the valid frame recognition result for each language, indicating that the current frame is an invalid frame corresponding to the second language; if the first language and / or the second language exists, then not performing decoding and recognition processing on the mapping probabilities corresponding to the first language and / or the second language, and using the remaining languages in the selected languages excluding the first language and / or the second language as target languages; otherwise, using all languages in the current frame as target languages; alternatively, not performing decoding and recognition processing on the mapping probabilities belonging to both the first language and the second language, and using the remaining languages in the selected languages excluding the first language and the second language as target languages; otherwise, using all languages in the current frame as target languages.

[0205] 505 , using the sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to the target language of the speech data.

[0206] The above steps 501 to 505 are executed in a loop until the audio data processing is completed.

[0207] 506 : Using the speech recognition result of the speech data in each frame corresponding to the target language as the audio recognition result of the audio data corresponding to the target language.

[0208] In this embodiment, a language classification network is used to first discard languages that do not require decoding and recognition processing, and then the languages of invalid frames are discarded again based on the valid frame nodes, further reducing the amount of calculation and improving efficiency. In this case, it can be comparable to a single language model, and the efficiency is almost the same as that of a single language. At the same time, a language classification network and valid frame nodes are added to the preset speech recognition model to achieve the purpose of switching-free multi-language recognition.

[0209] like Figure 11 As shown, the above step 103 may also include the following steps. Figure 9 The preset speech recognition model shown is used to perform speech recognition, which specifically includes the following steps.

[0210] 601 , obtaining speech data to be recognized in a current frame of audio data and a speech recognition result in a frame previous to the current frame.

[0211] 602. Merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the selected language to obtain a merged feature.

[0212] 603 , performing language classification processing on the coded features using a language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one language from a predetermined number of languages.

[0213] If so, execute step 604; otherwise, execute step 605.

[0214] 604. Jointly mapping the merged features using the sub-joint network layer corresponding to the candidate language in the joint network layer and the sub-softmax network layer corresponding to the candidate language in the softmax network layer to obtain a mapping probability that the speech data of the current frame corresponds to the candidate language, and performing effective frame recognition on the merged features corresponding to the candidate language using the effective frame node to obtain an effective frame recognition result of the candidate language, wherein the candidate language is the remaining language after excluding the first language from all languages corresponding to the current frame input.

[0215] 605. Using the sub-joint network layer corresponding to each language of the current frame in the joint network layer and the sub-softmax network layer corresponding to each language in the softmax network layer, the merged features are jointly mapped to obtain the mapping probability of the speech data of the current frame corresponding to each language, and using the valid frame node to perform valid frame recognition on the merged features corresponding to each language to obtain the valid frame recognition result for each language in the current frame.

[0216] 606 , detecting whether there is a valid frame of the second language. The recognition result indicates that the current frame is an invalid frame corresponding to the second language, wherein the current frame being an invalid frame means that the voice data corresponding to the current frame is a null character.

[0217] If so, execute step 607; otherwise, execute step 608.

[0218] 607 : The mapping probabilities corresponding to the second language are not decoded and recognized. Sub-decoding modules corresponding to the remaining languages are used to decode and recognize the mapping probabilities of the remaining languages to obtain speech recognition results corresponding to the speech data in the remaining languages.

[0219] 608 , using the sub-decoding module corresponding to each language to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to each language of the speech data.

[0220] In this embodiment, a language classification network is used to first discard languages that do not require decoding and recognition processing, and then the languages of invalid frames are discarded again based on the valid frame nodes, further reducing the amount of calculation and improving efficiency. In this case, it can be comparable to a single language model, and the efficiency is almost the same as that of a single language. At the same time, a language classification network and valid frame nodes are added to the preset speech recognition model to achieve the purpose of switching-free multi-language recognition.

[0221] In the above embodiment, an end-to-end preset speech recognition model is used to achieve multi-language switching-free speech recognition. Regardless of the language of the speech data to be recognized, the preset speech recognition model can share the encoding network and prediction network layer, thereby reducing the model parameters of the preset speech recognition model. Compared with the parallel calculation of multiple single-language recognition systems, the recognition efficiency of the preset speech recognition model is improved. At the same time, since the species embedding layer, joint network and softmax network are language-bound, that is, the species embedding layer, joint network and softmax network are set with independent parameters according to the language, the mutual influence between different languages is avoided, the language confusion is reduced, the differences in grammar and pronunciation of different languages are taken into account, and the accuracy of multi-language speech recognition is improved.

[0222] Figure 12 is a flow chart of a speech recognition method provided in an embodiment of the present application, which includes the following steps. This embodiment differs from the above embodiments in that, whereas the above embodiments select a language and then determine a target language from the selected languages, this embodiment does not select a language. Therefore, the target language must be determined from a preset number of languages in a preset speech recognition model.

[0223] 701 , receiving input audio data to be recognized, and obtaining a preset speech recognition model, where the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages.

[0224] The speech recognition module can be found in Figure 2 The speech recognition model shown. The preset speech recognition model includes a prediction network, an encoding network, and a joint network. The prediction network includes a language embedding layer and a prediction network layer. The joint network includes a joint network layer and a softmax network layer. The language embedding layer includes sub-embedding layers corresponding to a preset number of languages, the joint network layer includes sub-joint network layers corresponding to a preset number of languages, the softmax network layer includes sub-softmax network layers corresponding to a preset number of languages, and the decoding module includes sub-decoding modules corresponding to a preset number of languages.

[0225] The audio data to be recognized may be audio data input by a user, such as a segment of audio data input by a user on an interactive interface, or may be preset audio data to be recognized, which is received by a computer device.

[0226] 702. Perform language recognition processing on the audio data using a preset speech recognition model to obtain a target language corresponding to the audio data, and perform speech recognition processing on the audio data using an independent parameter module and a shared parameter module of the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in the decoding module to obtain an audio recognition result corresponding to the target language of the audio data, wherein the target language is at least one language from a preset number of languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages.

[0227] Among them, please refer to the description above for the independent parameter module and the shared parameter module, which will not be repeated here.

[0228] For example, each frame of speech data of the audio data is input into the prediction network for prediction processing to obtain prediction features, each frame of speech data is input into the encoding network for encoding processing to obtain encoding features, the encoding features and the prediction features are merged to obtain merged features, and the merged features are input into the sub-joint network layer corresponding to each language in the joint network layer. After processing by the sub-joint network layer, the processed results are input into the sub-softmax network layer corresponding to each language in the softmax network layer to obtain the mapping probability of the speech data corresponding to each language, and the decoding module corresponding to each language is used to perform decoding and recognition processing on the mapping probability to obtain the speech recognition results corresponding to each language of the speech data.

[0229] In one embodiment, if Figure 13 As shown, the above step 702 includes the following steps.

[0230] 801 , obtaining speech data to be recognized in a current frame and speech recognition results for each language in a frame before the current frame.

[0231] For the detailed description of this step, please refer to the above description and will not be repeated here.

[0232] 802. Merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the target language to obtain a merged feature.

[0233] Among them, the encoding network is used to encode the speech data to obtain the encoding features, the sub-embedding layer and prediction network layer corresponding to the target language are used to predict the speech recognition results of the previous frame of the current frame to obtain the prediction features, and the encoding features and the prediction features are merged to obtain the merged features.

[0234] Among them, since the prediction network includes a language embedding layer and a prediction network layer, the language embedding layer includes a predetermined number of sub-embedding layers corresponding to the preset number of languages. The speech recognition result of the previous frame may include speech recognition results corresponding to the preset number of languages. Correspondingly, the above-mentioned step of using the sub-embedding layer and the prediction network layer corresponding to the target language to perform prediction processing on the speech recognition result of the previous frame to obtain prediction features includes: using the sub-embedding layer corresponding to each language in the language embedding layer to perform feature processing on the speech recognition result of the previous frame for each language to obtain the embedding layer features corresponding to each language, and using the prediction network layer to perform prediction processing on the embedding layer features of each language to obtain the prediction features of each language.

[0235] Assume that each language includes language A, language B and language C. The speech recognition result of language A in the previous frame is input into the sub-embedding layer corresponding to language A in the species embedding layer. After the sub-embedding layer performs feature processing on the speech recognition result, the embedding layer features corresponding to language A are obtained. The embedding layer features corresponding to language A are input into the prediction network layer for prediction processing to obtain the prediction features of language A, and so on.

[0236] The speech data is encoded using a coding network to obtain coding features, and the predicted features and the coding features are merged to obtain merged features.

[0237] 803. Using the sub-joint network layer corresponding to each language in the joint network and the sub-softmax network layer corresponding to each language in the softmax network, the merged features are jointly mapped to obtain the speech recognition results corresponding to each language of the speech data of the current frame.

[0238] The merged features can be jointly processed using the sub-joint network layer corresponding to each language in the joint network layer to obtain the joint features of the speech data corresponding to each language; the sub-softmax network layer corresponding to each language in the softmax network layer can be used to perform normalized mapping processing on the joint features of each language to obtain the mapping probability of the speech data of the current frame corresponding to each language.

[0239] For example, the merged features of language A are input into the sub-joint network layer corresponding to language A in the joint network layer, and the sub-joint network layer corresponding to language A is used to jointly process the merged features to obtain the joint features of the speech data corresponding to language A; the joint features of speech A are input into the sub-softmax network layer corresponding to language A in the softmax network layer, and the sub-softmax network layer corresponding to language A is used to perform normalized mapping processing, and the mapping probability of the speech data of the current frame corresponding to language A is output, and so on.

[0240] In one embodiment, the speech recognition method further includes the following step 804 .

[0241] 804 , performing language recognition processing on the speech data using a language classification network and / or a valid frame node to obtain a target language corresponding to the audio data, where the target language is at least one of a preset number of languages.

[0242] Among them, the execution order of step 803 and step 804 is not limited. In some embodiments, step 804 can be executed first and then step 803. In one embodiment, 803 can be executed first and then 804. In some embodiments, step 803 and step 804 can be executed in parallel, etc.

[0243] In one embodiment, the step of performing language recognition processing on speech data using a language classification network to obtain a target language corresponding to the audio data includes: performing language classification processing on the encoded features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, the first language being at least one language among a preset number of languages; if so, using the remaining languages in the current frame input after excluding the first language as the target language; if not, using all the languages input in the current frame as the target language.

[0244] The step of performing language classification processing on the coding features using a language classification network to determine whether there is a first language that does not require decoding and recognition processing includes: performing confidence processing on the coding features using a language classification network to obtain confidence data for a preset number of languages in the current frame; and determining whether there is a first language that does not require decoding and recognition processing based on the confidence data for the preset number of languages in the current frame.

[0245] Among them, the step of using the language classification network to perform confidence processing on the coding features to obtain confidence data for a preset number of languages in the current frame includes: using the language classification network to perform posterior probability processing on the coding features to obtain the posterior probabilities of the preset number of languages in the current frame; and determining the confidence data of the preset number of languages in the current frame based on the posterior probabilities of the preset number of languages in the current frame.

[0246] In one embodiment, the step of performing language recognition processing on speech data using a language classification network to obtain a target language corresponding to the audio data includes: performing language classification processing on the encoded features using the language classification network to determine whether a first language that does not require decoding and recognition processing exists, wherein the first language is at least one language from a preset number of languages; if so, using the remaining languages in the current frame after excluding the first language as the target language; if not, using all languages in the current frame as the target language;

[0247] The step of using the sub-joint network layer corresponding to each language in the joint network and the sub-softmax network layer corresponding to each language in the softmax network to perform joint mapping processing on the merged features to obtain the speech recognition result of each language corresponding to the speech data of the current frame includes: using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer to perform joint mapping processing on the merged features to obtain the mapping probability of the speech data of the current frame corresponding to the target language.

[0248] In one embodiment, the step of performing language recognition processing on speech data using a valid frame node to obtain a target language corresponding to the audio data includes: performing valid frame recognition on the merged features of each language in the current frame using the valid frame node to obtain a valid frame recognition result corresponding to each language; when there is a valid frame recognition result of a second language indicating that the current frame is an invalid frame corresponding to the second language, the remaining languages of the current frame after excluding the second language are used as target languages, wherein the current frame being an invalid frame means that the speech data corresponding to the current frame is an empty character, and the second language is at least one language among a preset number of languages; when there is no valid frame recognition result of the second language indicating that the current frame is an invalid frame corresponding to the second language, all languages of the current frame are used as target languages.

[0249] In one embodiment, language recognition processing is performed on speech data using a language classification network and valid frame nodes to obtain a target language corresponding to the audio data, including: performing language classification processing on the encoded features using the language classification network to obtain confidence data for a preset number of languages in the current frame; performing valid frame recognition on the merged features of each language in the current frame using all valid frame nodes to obtain a valid frame recognition result for each language in the current frame; determining whether there is a third language that does not require decoding and recognition based on the confidence data and the valid frame recognition result for each language in the current frame; if there is a third language, decoding and recognition processing is not performed on the third language, and the remaining languages in the current frame after excluding the third language are used as target languages, wherein the third language is at least one language from the preset number of languages, and if not, all languages in the current frame are used as target languages.

[0250] In one embodiment, a language classification network and a valid frame node are used to perform language recognition processing on speech data to obtain a target language corresponding to the audio data, including: using the language classification network to perform language classification processing on the encoded features to determine whether there is a first language that does not need to be decoded and recognized, wherein the first language is at least one language among a preset number of languages; using the valid frame node to perform valid frame recognition on the combined features of each language in the current frame to obtain a valid frame recognition result for each language, and determining whether there is a valid frame recognition result of a second language representing the current frame based on the valid frame recognition result of each language. The frame is an invalid frame corresponding to the second language; if the first language and / or the second language exists, the mapping probabilities corresponding to the first language and / or the second language are not decoded and recognized, and the languages of the current frame excluding the first language and / or the second language are used as the target language; otherwise, all languages in the current frame are used as the target language; alternatively, the mapping probabilities belonging to both the first language and the second language are not decoded and recognized, that is, the languages of the current frame excluding the languages belonging to both the first language and the second language are used as the target language; otherwise, all languages in the current frame are used as the target language.

[0251] In one embodiment, a language classification network and a valid frame node are used to perform language recognition processing on speech data to obtain a target language corresponding to the audio data, including: performing language classification processing on the encoded features using the language classification network to determine whether there is a first language that does not need to be decoded and recognized, wherein the first language is at least one language among a preset number of languages; if so, using a sub-joint network layer corresponding to each language in the joint network and a sub-softmax network layer corresponding to each language in the softmax network to perform joint mapping processing on the merged features to obtain a speech recognition result corresponding to each language of the speech data of the current frame, including: using a sub-joint network layer corresponding to the candidate language in the joint network layer and a softmax network layer The sub-softmax network layer corresponding to the candidate language in the is used to perform joint mapping processing on the merged features to obtain the mapping probability of the speech data of the current frame corresponding to the candidate language; wherein the candidate language is the remaining language after excluding the first language from all the languages corresponding to the input of the current frame; valid frame recognition is performed on the merged features of the candidate language using the valid frame node to obtain the valid frame recognition result of the candidate language, and the valid frame recognition result of the candidate language is determined based on the valid frame recognition result of the candidate language to indicate that the current frame is an invalid frame corresponding to the second language; if the second language exists, the language after excluding the second language from the candidate language is used as the target language, and the mapping probability corresponding to the second language is not decoded and recognized; if the second language does not exist, the candidate language is used as the target language.

[0252] 805 , using the sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to the target language of the speech data.

[0253] For example, the sub-decoding module corresponding to language A is used to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to language A of the speech data.

[0254] After obtaining the speech recognition result of the target language corresponding to the speech data of the current frame, the speech recognition result of the current frame is used as the speech recognition result of the previous frame, and then the speech data of the current frame of the audio data x is recognized until the end symbol is output or the recognition of the audio data is ended, then the recognition of the audio data x is ended, that is, the above steps 801 to 805 are a cyclic execution process until the current speech recognition is completed. At the same time, please refer to the corresponding description above for the parts not described in detail in this embodiment, which will not be repeated here.

[0255] 806 : Using the speech recognition result of the speech data in each frame corresponding to the target language as the audio recognition result of the audio data corresponding to the target language.

[0256] For steps not described in detail in this embodiment, please refer to the description of the corresponding steps above and will not be repeated here.

[0257] 703. Return the audio recognition result.

[0258] The speech recognition results corresponding to the speech data of all frames may be returned at once, or the semantic recognition results corresponding to the speech data of each frame may be returned frame by frame. In one embodiment, the returned speech recognition results may also be displayed.

[0259] Figure 14 This is a flow chart of a speech recognition method provided by an embodiment of the present application. The speech recognition method is Figure 2 The training method of the preset speech recognition model shown includes the following steps.

[0260] 901. Obtain training speech data of a current frame of a target training sample in a specific language and a training speech recognition result of a previous frame of the current frame, and obtain an initial speech recognition model, wherein the initial speech recognition model includes an encoding network, a prediction network, and a joint network, the prediction network includes a language embedding layer and a preset network layer, the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages, wherein the specific language is at least one of the preset number of languages predetermined by the initial speech recognition model.

[0261] The preset number of languages are the languages in the target training sample set used during the initial speech recognition model training process. Audio datasets corresponding to the preset number of languages are obtained, and audio data is randomly sampled from the audio datasets corresponding to the preset number of languages to obtain a randomly sampled audio dataset of a specific language. During the training phase, each batch consists of a randomly sampled audio dataset of a specific language. The specific language is at least one language from the preset number of languages predetermined by the initial speech recognition model, and each of the specific languages is known in advance.

[0262] For each target audio data item in a specific language within each batch of the audio data set, the training speech data corresponding to each frame or each moment of the target audio data is obtained and input into the initial speech recognition model for training. Specifically, the training speech data for the current frame of the target training sample in the specific language and the training speech recognition result for the previous frame are obtained, where the training speech recognition result for the previous frame includes the training speech recognition result for the specific language.

[0263] Each target audio data in a specific language has label data, and the label data includes text corresponding to the voice data or other reasonable data.

[0264] 902. Merge the training coding features obtained by encoding the training speech data using the encoding network and the training prediction features obtained by predicting the training speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to each language in the language embedding layer to obtain a training merged feature.

[0265] Among them, the encoding network is used to encode the training speech data to obtain training encoding features, the sub-embedding layer and prediction network layer corresponding to each language in the language embedding layer are used to predict the training speech recognition result of the previous frame to obtain training prediction features, and the training encoding features and training prediction features are merged to obtain training merged features.

[0266] Correspondingly, the preset speech recognition model also includes a feature encoding layer, which inputs the training speech data into the feature extraction layer, performs acoustic feature processing on the training speech data using the feature extraction layer to obtain training acoustic features, and then inputs the training acoustic features into the encoding network, which encodes the training acoustic features using the encoding network to obtain training coded features. In one embodiment, the encoding network includes a feature extraction layer and an encoding network layer, which inputs the training speech data into the feature extraction layer, performs acoustic feature processing on the training speech data using the feature extraction layer to obtain training acoustic features, and then inputs the training acoustic features into the encoding network layer, which encodes the training acoustic features using the encoding network layer to obtain training coded features.

[0267] The prediction network includes a language embedding layer and a prediction network layer, wherein the language embedding layer includes a preset number of sub-embedding layers for each language. The training speech recognition results are predicted using the sub-embedding layers corresponding to specific languages in the language embedding layer and the prediction network layer to obtain training prediction features for the specific language. The language embedding layer includes a preset number of sub-embedding layers corresponding to each language.

[0268] The training speech recognition result of the specific language of the previous frame is input into the sub-embedding layer corresponding to the specific language in the language embedding layer, and the sub-embedding layer corresponding to the specific language is used to perform feature processing to obtain the embedding layer features corresponding to the specific language; the embedding layer features corresponding to the specific language are input into the prediction network layer, and the prediction network layer is used to perform prediction processing on the embedding layer features corresponding to the specific language to obtain the training prediction features of the specific language.

[0269] After obtaining the training prediction features and the training coding features, the training prediction features and the training coding features are merged to obtain the training merged features.

[0270] The merging process includes superposition processing or splicing processing, and the training prediction features and the training coding features are superimposed or spliced to obtain the training merged features.

[0271] 903. Perform joint mapping processing on the training merged features using the sub-joint network layer corresponding to each language in the joint network layer and the sub-softmax network layer corresponding to each language in the softmax network layer to obtain a training mapping probability of the training speech data corresponding to each language, wherein the joint network layer includes sub-joint network layers corresponding to a preset number of languages, and the softmax network layer includes sub-softmax network layers corresponding to a preset number of languages.

[0272] The training merged features are input into the sub-joint network layer corresponding to the specific language in the joint network layer, and the sub-joint network layer corresponding to the specific language is used to jointly process the training merged features, and the training joint features corresponding to the specific language of the training speech data are output. The training joint features of the specific language are input into the sub-softmax network layer corresponding to the specific language in the softmax network layer, and the sub-softmax network layer corresponding to the specific language is used to perform normalized mapping processing on the training joint features to obtain the training mapping probability that the training speech data of the current frame corresponds to the specific language.

[0273] 904 , using the sub-decoding module corresponding to the specific language in the decoding module to perform decoding and recognition processing on the training mapping probability to obtain a training speech recognition result corresponding to the specific language of the training speech data.

[0274] 905 , using the training speech recognition result of the current frame and the label data of the current frame of the target training sample to update the parameters in the initial speech recognition model, so as to train a preset number of languages to obtain a preset speech recognition model.

[0275] After obtaining the training speech recognition result of the specific language of the current frame, the parameters in the initial speech recognition model are updated according to the training speech recognition result of the specific language of the current frame and the label data of the specific language of the current frame of the target training sample.

[0276] Among them, the model parameters of the shared encoding network, shared prediction network layer, and the sub-embedding layer, sub-joint network layer, and sub-softmax network layer of a specific language will be updated. This avoids the mutual influence between different languages, realizes the decoding decoupling of different languages, and reduces language confusion.

[0277] Then, training is continued to complete the training of the preset number of languages, and finally a preset speech recognition model is obtained.

[0278] 906 , using a preset speech recognition model to perform recognition processing on each frame of speech data corresponding to the audio data to be recognized, to obtain a recognition result of the speech data.

[0279] In this embodiment, the Figure 2 The training of the preset speech recognition model shown in the figure is performed, and the preset speech recognition model obtained by training is used to perform recognition processing on the speech data to be recognized to obtain the recognition result of the speech data. The training in the embodiment of the present application is an end-to-end joint training without switching of multiple languages, that is, multi-language mixed training, which performs unified modeling for multiple languages, sets different sub-embedding layers, sub-joint network layers, and sub-softmax network layers according to the languages, and shares the prediction network layer and the encoding network, so that the parameters of different sub-embedding layers, sub-joint network layers, and sub-softmax network layers are not shared, thereby avoiding the mutual influence between different languages and reducing the language confusion. However, the parameters of the prediction network layer and the encoding network are shared, which reduces the model parameters of the preset speech recognition model.

[0280] In one embodiment, when the initial speech recognition model also includes a language classification network, the speech recognition method further includes training the language classification network. Specifically, the speech recognition method further includes: utilizing the language classification network to perform language classification processing on the trained coded features to determine whether a first language that does not require decoding and recognition exists; and updating parameters of the language classification network based on the determination of whether a first language that does not require decoding and recognition exists and the specific language. It will be understood that the language classification network is independent of the other current modules in the initial speech recognition model, i.e., training of the language classification network is independent of the other modules, and the training of the language classification network is language-independent, i.e., all languages share the same parameters of the language classification network.

[0281] In one embodiment, when the initial speech recognition model also includes a valid frame node, the speech recognition method further includes training the valid frame node. Specifically, the speech recognition method further includes: using the valid frame node to perform valid frame recognition on the merged features of a specific language to obtain a training valid frame recognition result for the specific language; and updating the parameters of the valid frame node based on the training valid frame recognition result and the label data of the current frame of the target training sample. The valid frame node is independent of the other current modules in the initial speech recognition model, that is, the training of the valid frame node is completed independently, and the training of the valid frame node is independent of the language, that is, all languages share the parameters of the language classification network.

[0282] In one embodiment, when the initial speech recognition model also includes a valid frame node and a language classification network, the speech recognition method further includes training the valid frame node and the language classification network. Specifically, the speech recognition method further includes: using the language classification network to perform language classification processing on the training coding features to determine whether there is a first language that does not require decoding and recognition processing; updating the parameters of the language classification network based on the determination result of whether there is a first language that does not require decoding and recognition processing and the specific language. Using the valid frame node to perform valid frame recognition on the merged features of the specific language to obtain a training valid frame recognition result for the specific language; and updating the parameters of the valid frame node based on the training valid frame recognition result and the label data of the current frame of the target training sample.

[0283] Although the valid frame nodes and language classification network participate in the training of the initial speech recognition model at the same time, their training is independent of the training of other modules of the initial speech recognition model, and both the valid frame nodes and the language classification network are language-independent.

[0284] In the embodiment of the present application, end-to-end joint training of preset speech recognition models for multiple languages is implemented, and language classification networks and valid frame nodes are added to achieve the purpose of multi-language switching-free. The language confusion is reduced through the language-bound decoding network, the model parameters are reduced through the shared encoding network and prediction network layer, the language classification network and valid frame nodes are added, and the amount of calculation is reduced in parallel, so that the Fangpu switching-free recognition effect can reach the effect of a single language.

[0285] It should be noted that when training the initial speech recognition model, the corresponding steps are the same as those for using the preset speech recognition model. However, when training the initial speech recognition model, the word "training" is added before all terms. For specific training steps, please refer to the steps for using the preset speech recognition model above, which will not be repeated here.

[0286] All of the above technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0287] In order to better implement the speech recognition method of the embodiment of the present application, the embodiment of the present application also provides a speech recognition device. Figure 15 , Figure 15 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application. The speech recognition device may include a receiving module 1001, a speech recognition module 1002, and a returning module 1003.

[0288] A receiving and obtaining module 1001 is configured to receive input audio data to be recognized and obtain a preset speech recognition model, wherein the preset speech recognition model includes a prediction network, an encoding network, and a joint network, wherein the joint network includes a joint network layer and a softmax network layer;

[0289] The speech recognition module 1002 is used to use the prediction network, encoding network, sub-joint network layer corresponding to each language in the joint network layer, sub-softmax network layer corresponding to each language in the softmax network layer, and decoding module corresponding to each language in the preset speech recognition model to perform speech recognition processing on each frame of speech data of the audio data to obtain a speech recognition result of the speech data, wherein the joint network layer includes sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes sub-softmax network layers corresponding to the preset number of languages.

[0290] The return module 1003 is used to return the speech recognition result.

[0291] In one embodiment, the receiving and acquiring module 1001 is further configured to receive a selected target language and acquire a preset speech recognition model, wherein the target language is at least one of a preset number of languages in the preset speech recognition model, the preset speech recognition model comprising a prediction network, a coding network, and a joint network, wherein the joint network comprises a joint network layer and a softmax network layer; and receive input audio data to be recognized. Correspondingly, the speech recognition module 1002 is configured to perform speech recognition processing on each frame of speech data in the audio data using the prediction network, coding network, a sub-joint network layer corresponding to the target language in the joint network layer, a sub-softmax network layer corresponding to the target language in the softmax network layer, and a decoding module corresponding to the target language in the preset speech recognition model, wherein the joint network layer comprises sub-joint network layers corresponding to the preset number of languages, and the softmax network layer comprises sub-softmax network layers corresponding to the preset number of languages. The returning module 1003 is configured to return the speech recognition result.

[0292] In one embodiment, the speech recognition module 1002 may include: Figure 16 For all modules included in the relevant embodiments, please refer to the following description and will not be described in detail here.

[0293] like Figure 16As shown, an embodiment of the present application further provides a speech recognition device, which may include a first acquisition module 1101, a first merging module 1102, a first mapping module 1103 and a first decoding and recognition module 1104.

[0294] The first acquisition module 1101 is used to obtain the speech data to be recognized in the current frame, the speech recognition result in the previous frame of the current frame and a preset speech recognition model, the preset speech recognition model includes a prediction network, an encoding network and a joint network, the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages.

[0295] The first merging module 1102 is used to merge the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the prediction network to obtain a merged feature.

[0296] Among them, the first merging module 1102 includes a first encoding module, a first prediction module and a first merging module. Among them, the first encoding module is used to use the encoding network to encode the speech data to obtain encoding features. The first prediction module is used to use the prediction network to predict the speech recognition result of the previous frame to obtain prediction features, such as using the sub-embedding layer corresponding to each language in the language embedding layer and the prediction network layer to predict the speech recognition result corresponding to each language in the previous frame to obtain prediction features for each language. The first merging module is used to merge the prediction features and the encoding features to obtain merged features.

[0297] Among them, the first prediction module includes a first embedding unit and a first prediction unit, wherein the first embedding unit is used to use the sub-embedding layer of each language in the language embedding layer to perform feature processing on the speech recognition result corresponding to each language in the previous frame to obtain the embedding layer features corresponding to each language; the first prediction unit is used to use the prediction network layer to perform prediction processing on the embedding layer features of each language to obtain the prediction features of each language.

[0298] The first mapping module 1103 is used to perform joint mapping processing on the merged features using the sub-joint network layer corresponding to each language in the joint network layer and the sub-softmax network layer corresponding to each language in the softmax network layer to obtain a mapping probability of the speech data corresponding to each language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages.

[0299] The first decoding and recognition module 1104 is configured to perform decoding and recognition processing on the mapping probability using a decoding module corresponding to each language, so as to obtain a speech recognition result corresponding to each language of the speech data.

[0300] In one embodiment, if Figure 15 As shown, the speech recognition device may further include a first language recognition module. The first language recognition module is used to perform language recognition processing on the speech data using a language classification network and / or valid frame nodes to obtain a target language corresponding to the audio data.

[0301] The first language recognition module includes a first classification module 1105, wherein the first classification module 1105 is used to perform language classification processing on the coding features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one language among a predetermined number of languages. If so, the remaining languages in the current frame after excluding the first language are used as target languages; otherwise, all languages in the current frame are used as target languages.

[0302] In one embodiment, if Figure 16 As shown, the first language identification module may further include a first frame detection module 1106, which is configured to perform valid frame identification on the merged features of each language using the valid frame node to obtain a valid frame identification result for each language; when there is a valid frame identification result of a second language indicating that the current frame is an invalid frame corresponding to the second language, the remaining languages of the current frame after excluding the second language are used as target languages; otherwise, all languages of the current frame are used as target languages.

[0303] The present application also provides a speech recognition device. Figure 17 , Figure 17 This is a structural diagram of a speech recognition device provided in an embodiment of the present application. The speech recognition device may include a second acquisition module 1201, a second merging module 1202, a second mapping module 1203, a second decoding and recognition module 1204, an updating module 1205, and a recognition module 1206.

[0304] The second acquisition module 1201 is used to obtain the training speech data of the current frame of the target training sample of a specific language and the training speech recognition result of the previous frame of the current frame, and to obtain an initial speech recognition model, wherein the initial speech recognition model includes an encoding network, a prediction network and a joint network, wherein the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages, wherein the specific language is at least one of the preset number of languages predetermined by the initial speech recognition model.

[0305] The second merging module 1202 is used to merge the training coding features obtained by encoding the training speech data using the encoding network and the training prediction features obtained by predicting the training speech recognition result of the previous frame using the prediction network to obtain a training merged feature.

[0306] Among them, the second merging module 1202 includes a second encoding module, a second prediction module and a second merging module. Among them, the second encoding module is used to use the encoding network to encode the training speech data to obtain training encoding features. The second prediction module is used to use the prediction network to predict the training speech recognition result of the previous frame to obtain training prediction features, such as using the sub-embedding layer corresponding to the specific language in the language embedding layer and the prediction network layer to predict the training speech recognition result corresponding to the specific language in the previous frame to obtain training prediction features of the specific language. The second merging module is used to merge the training prediction features and the training encoding features to obtain training merged features.

[0307] Among them, the second prediction module includes a second embedding unit and a second prediction unit, wherein the second embedding unit is used to use the sub-embedding layer of the specific language in the language embedding layer to perform feature processing on the training speech recognition result corresponding to the specific language in the previous frame to obtain the training embedding layer features corresponding to the specific language; the second prediction unit is used to use the prediction network layer to perform prediction processing on the training embedding layer features of the specific language to obtain the training prediction features of the specific language.

[0308] The second mapping module 1203 is used to perform joint mapping processing on the training merged features using the sub-joint network layer corresponding to the specific language in the joint network layer and the sub-softmax network layer corresponding to the specific language in the softmax network layer to obtain a training mapping probability that the training speech data corresponds to the specific language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages.

[0309] The second decoding and recognition module 1204 is configured to perform decoding and recognition processing on the training mapping probability using a decoding module corresponding to a specific language, so as to obtain a training speech recognition result corresponding to the specific language of the training speech data.

[0310] The updating module 1205 is used to update the model parameters in the initial speech recognition model using the training speech recognition result of the current frame and the label data of the current frame of the target training sample, so as to train a preset number of languages to obtain a preset speech recognition model.

[0311] The recognition module 1206 is configured to perform recognition processing on each frame of speech data in the audio data to be recognized using the preset speech recognition model to obtain a speech recognition result of the speech data.

[0312] In one embodiment, the speech recognition device may further include a second language recognition module.

[0313] The second language identification module includes a second classification module 1207, wherein the second classification module 1207 is configured to perform language classification processing on the training coding features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one language from a predetermined number of languages, and update parameters of the language classification network based on the determination result of whether there is a first language that does not require decoding and recognition processing and the specific language.

[0314] In one embodiment, the second language recognition module may also include a second frame detection module 1208, which is used to use the valid frame node to perform valid frame recognition on the merged features of the specific language to obtain a training valid frame recognition result of the specific language; and update the parameters of the valid frame node according to the training valid frame recognition result and the label data of the current frame of the target training sample.

[0315] All of the above technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0316] Accordingly, an embodiment of the present application further provides a computer device, which may be a terminal or a server. The computer device where the speech recognition method for training a preset speech recognition model is implemented and the computer device where the speech recognition method using the preset speech recognition model is implemented may be the same computer device or different computer devices.

[0317] like Figure 18 As shown, Figure 18 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 1300 includes a processor 1301 having one or more processing cores, a memory 1302 having one or more computer-readable storage media, and a computer program stored in the memory 1302 and executable on the processor. The processor 1301 is electrically connected to the memory 1302. Those skilled in the art will appreciate that the computer device structure shown in the figure does not constitute a limitation of the computer device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0318] The processor 1301 is the control center of the computer device 1300. It uses various interfaces and lines to connect the various parts of the entire computer device 1300. By running or loading software programs (computer programs) and / or modules stored in the memory 1302 and calling data stored in the memory 1302, it executes various functions of the computer device 1300 and processes data, thereby monitoring the computer device 1300 as a whole.

[0319] In the embodiment of the present application, the processor 1301 in the computer device 1300 will load the instructions corresponding to the processes of one or more application programs into the memory 1302 according to the following steps, and the processor 1301 will run the application programs stored in the memory 1302, thereby realizing various functions, such as the functions corresponding to the speech recognition method in any embodiment of the present application. The specific implementation of each of the above operations can be found in the previous embodiments and will not be repeated here. At the same time, the beneficial effects that can be achieved by any speech recognition method provided in the embodiment of the present application can be achieved, as detailed in the previous embodiments and will not be repeated here.

[0320] Optional, such as Figure 18 As shown, the computer device 1300 further includes: a touch screen 1303, a radio frequency circuit 1304, an audio circuit 1305, an input unit 1306, and a power supply 1307. Among them, the processor 1301 is electrically connected to the touch screen 1303, the radio frequency circuit 1304, the audio circuit 1305, the input unit 1306, and the power supply 1307 respectively. It can be understood by those skilled in the art that Figure 17The computer device structure shown in the figure does not constitute a limitation to the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0321] The touch screen 1303 can be used to display a graphical user interface and receive user actions on the operation instructions generated by the graphical user interface. The touch screen 1303 can include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the computer device, and these graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, a liquid crystal display (LCD), an organic light emitting diode (OLED, Organic Light-Emitting Diode) and the like can be used to configure the display panel. The touch panel can be used to collect user touch operations on or near it (such as the user uses any suitable object or accessory such as a finger, stylus on or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. The touch panel can cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor 1301 to determine the type of touch event, and then the processor 1301 provides corresponding visual output on the display panel according to the type of touch event. In the embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 1303 to implement input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to implement input and output functions. In other words, the touch display screen 1303 can also be used as part of the input unit 1306 to implement input functions.

[0322] In the embodiment of the present application, the touch screen 1303 is used to present a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface.

[0323] The radio frequency circuit 1304 may be used to transmit and receive radio frequency signals, thereby establishing wireless communication with a network device or other computer device through wireless communication, and transmitting and receiving signals between the network device or other computer device.

[0324] Audio circuit 1305 can be used to provide an audio interface between the user and the computer device through a speaker and microphone. Audio circuit 1305 can convert received audio data into electrical signals and transmit them to the speaker, which then converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 1305 and converted into audio data. The audio data is then output to processor 1301 for processing, then transmitted via RF circuit 1304 to, for example, another computer device, or to memory 1302 for further processing. Audio circuit 1305 may also include an earphone jack to allow communication between external headphones and the computer device.

[0325] The input unit 1306 may be configured to receive input digital, character information, or user feature information (such as fingerprint, iris, or facial information), and to generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control.

[0326] Power supply 1307 is used to supply power to various components of computer device 1300. Optionally, power supply 1307 can be logically connected to processor 1301 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Power supply 1307 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0327] although Figure 17 Not shown in the figure, the computer device 1300 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be repeated here.

[0328] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0329] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0330] To this end, the present invention provides a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute the steps of any of the speech recognition methods provided in the present invention. The specific implementation of each of the above operations can be found in the previous embodiments and will not be repeated here.

[0331] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0332] Since the computer program stored in the storage medium can execute the steps of any speech recognition method provided in the embodiments of the present application, the beneficial effects that can be achieved by any speech recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0333] The above is a detailed introduction to a speech recognition method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech recognition method, characterized in that: include: receiving a selected language and obtaining a preset speech recognition model, wherein the selected language is at least one of a preset number of languages in the preset speech recognition model; Receiving input audio data to be recognized; Performing language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and performing speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in a decoding module to obtain an audio recognition result corresponding to the target language for the audio data, wherein the target language is at least one of the selected languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages; Returns the audio recognition result.

2. The speech recognition method according to claim 1, wherein: The preset speech recognition model includes a prediction network, an encoding network and a joint network. The prediction network includes a language embedding layer and a prediction network layer. The joint network includes a joint network layer and a softmax network layer. The language embedding layer includes sub-embedding layers corresponding to a preset number of languages. The joint network layer includes sub-joint network layers corresponding to the preset number of languages. The softmax network layer includes sub-softmax network layers corresponding to the preset number of languages. The independent parameter module of the target language includes a sub-embedding layer corresponding to the target language, a sub-joint network layer corresponding to the target language, and a sub-softmax network layer corresponding to the target language. The shared parameter module includes a prediction network layer and an encoding network.

3. The speech recognition method according to claim 2, wherein: The audio data includes multiple frames of speech data. If the selected language is one, the target language is the selected language. The step of performing speech recognition processing on the audio data using the independent parameter module and the shared parameter module of the target language in the preset speech recognition model, and the sub-decoding module corresponding to the target language in the decoding module to obtain an audio recognition result corresponding to the target language of the audio data includes: Acquire speech data to be recognized in a current frame of the audio data, and a speech recognition result in a frame previous to the current frame; Merging the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the target language to obtain a merged feature; Performing joint mapping processing on the merged features using a sub-joint network layer corresponding to the target language in the joint network layer and a sub-softmax network layer corresponding to the target language in the softmax network layer to obtain a mapping probability that the speech data corresponds to the target language; Using a sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to the target language of the speech data; The speech recognition result of the speech data in each frame corresponding to the target language is used as the audio recognition result of the audio data corresponding to the target language.

4. The speech recognition method according to claim 3, wherein: The step of performing joint mapping processing on the merged features using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer to obtain a mapping probability that the speech data corresponds to the target language includes: Using the sub-joint network layer corresponding to the target language in the joint network layer to jointly process the merged features to obtain joint features corresponding to the target language; The joint features are normalized and mapped using a sub-softmax network layer corresponding to the target language in the softmax network layer to obtain a mapping probability that the speech data corresponds to the target language.

5. The speech recognition method according to claim 3, wherein: The speech recognition result of the previous frame is predicted using the sub-embedding layer and the prediction network layer corresponding to the target language to obtain prediction features, including: Performing feature processing on the speech recognition result corresponding to the target language in the previous frame using the sub-embedding layer of the target language in the language embedding layer to obtain embedding layer features corresponding to the target language; The prediction network layer is used to perform prediction processing on the embedding layer features of the target language to obtain the prediction features of the target language.

6. The speech recognition method according to claim 2, wherein: The audio data includes multiple frames of speech data. If multiple languages are selected, the joint network further includes valid frame nodes and / or the preset speech recognition model further includes a language classification network. The steps of using the preset speech recognition model to perform language recognition processing on the audio data to obtain a target language corresponding to the audio data, and using an independent parameter module and a shared parameter module of the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in a decoding module to perform speech recognition processing on the audio data to obtain an audio recognition result corresponding to the target language for the audio data include: Acquire the speech data to be recognized in the current frame of the audio data and the speech recognition result in the frame before the current frame; Merging the encoding features obtained by encoding the speech data using the encoding network and the prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the selected language to obtain a merged feature; Performing joint mapping processing on the merged features using a sub-joint network layer corresponding to the selected language in the joint network layer and a sub-softmax network layer corresponding to the selected language in the softmax network layer to obtain a mapping probability that the speech data corresponds to the selected language; Performing language recognition processing on the speech data using the language classification network and / or the valid frame node to obtain a target language corresponding to the audio data; Using a sub-decoding module corresponding to the target language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to the target language of the speech data; The speech recognition result of the speech data in each frame corresponding to the target language is used as the audio recognition result of the audio data corresponding to the target language.

7. The speech recognition method according to claim 6, characterized in that The step of performing language recognition processing on the speech data using a language classification network to obtain the target language corresponding to the audio data includes: performing language classification processing on the coded features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, the first language being at least one of the selected languages; If so, the remaining languages in the current frame excluding the first language are used as target languages. If not, all languages in the current frame are used as target languages.

8. The speech recognition method according to claim 7, characterized in that: The step of performing language classification processing on the coding features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing includes: Performing confidence processing on the coding features using the language classification network to obtain confidence data for a preset number of languages in the current frame; It is determined whether there is a first language that does not require decoding and recognition processing based on the confidence data of a preset number of languages in the current frame.

9. The speech recognition method according to claim 8, characterized in that The step of performing confidence processing on the coding features using the language classification network to obtain confidence data of a preset number of languages in the current frame includes: Performing posterior probability processing on the coding features using the language classification network to obtain posterior probabilities of a preset number of languages in the current frame; Confidence data of a preset number of languages in the current frame are determined based on the posterior probabilities of the preset number of languages in the current frame.

10. The speech recognition method according to claim 8, characterized in that: The step of determining whether there is a first language that does not require decoding and recognition processing based on the confidence data of a preset number of languages in the current frame includes: Determining whether confidence data of a language in the current frame satisfies a first preset confidence condition, or whether confidence data of a language in the current frame within a preset frame satisfies a second preset confidence condition; If it exists, the existing language is used as the first language that does not need to be decoded and recognized.

11. The speech recognition method according to claim 6, wherein: The step of performing language recognition processing on the speech data using a language classification network to obtain the target language corresponding to the audio data includes: performing language classification processing on the coded features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one of the selected languages; If it exists, the remaining languages of the current frame excluding the first language are used as the target language; if it does not exist, all the languages of the current frame are used as the target language; The step of using the sub-joint network layer corresponding to the selected language in the joint network layer and the sub-softmax network layer corresponding to the selected language in the softmax network layer to jointly map the merged features to obtain the mapping probability that the speech data corresponds to the selected language includes: using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer to jointly map the merged features to obtain the mapping probability that the speech data of the current frame corresponds to the target language.

12. The speech recognition method according to claim 6, wherein: Performing language recognition processing on the voice data using the valid frame node to obtain the target language corresponding to the audio data includes: Using the valid frame node to perform valid frame recognition on the combined features of each language in the current frame to obtain a valid frame recognition result corresponding to each language; When there is a valid frame recognition result of the second language indicating that the current frame is an invalid frame corresponding to the second language, the remaining language after excluding the second language from the language of the current frame is used as the target language, wherein the current frame being the invalid frame means that the speech data corresponding to the current frame is an empty character, and the second language is at least one of the selected languages.

13. The speech recognition method according to claim 12, wherein: The step of performing valid frame recognition on the combined features of each language in the current frame using the valid frame node to obtain a valid frame recognition result corresponding to each language includes: Using the valid frame node, performing valid frame recognition processing on the combined features of each language of the current frame to obtain a valid frame recognition probability corresponding to each language; A valid frame recognition result for each language is determined according to the valid frame recognition probability for each language.

14. The speech recognition method according to claim 13, characterized in that: The step of determining the valid frame recognition result for each language based on the valid frame recognition probability for each language includes: determining whether the valid frame recognition probability of each language is less than a preset valid frame recognition probability; If yes, determining that the current frame is an invalid frame of the corresponding language; Otherwise, it is determined that the current frame is a valid frame of the corresponding language.

15. The speech recognition method according to claim 6, characterized in that: Performing language recognition processing on the speech data using the language classification network and the valid frame node to obtain the target language corresponding to the audio data includes: Performing language classification processing on the coding features using the language classification network to obtain confidence data for a preset number of languages in the current frame; performing valid frame recognition on the combined features of each language in the current frame using the valid frame node to obtain a valid frame recognition result for each language in the current frame; determining whether there is a third language that does not require decoding and recognition based on the confidence data and the valid frame recognition results of each language in the current frame; If so, the remaining languages of the current frame excluding the third language are used as target languages. If not so, all languages of the current frame are used as target languages, wherein the third language is at least one of the selected languages.

16. The speech recognition method according to claim 15, characterized in that: The step of determining whether there is a third language that does not require decoding and recognition based on the confidence data and the valid frame recognition results of each language in the current frame includes: Whether the confidence data of the corresponding language satisfies a first preset confidence condition, and at the same time, the valid frame recognition probability of the corresponding language is less than a preset valid frame recognition probability; If it exists, the existing language will be regarded as the third language that does not need to be decoded and recognized.

17. The speech recognition method according to claim 6, characterized in that: Performing language recognition processing on the speech data using the language classification network and the valid frame node to obtain the target language corresponding to the audio data includes: performing language classification processing on the coded features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one of the selected languages; performing valid frame recognition on the combined features of each language in the current frame using the valid frame node to obtain a valid frame recognition result for each language, and determining, based on the valid frame recognition result for each language, whether there is a valid frame recognition result for a second language indicating that the current frame is an invalid frame corresponding to the second language; If the first language and / or the second language exists, the remaining languages excluding the first language and / or the second language in the current frame are used as the target language; otherwise, all languages in the current frame are used as the target language; or The remaining languages in the current frame except those belonging to both the first language and the second language are used as target languages. Otherwise, all languages in the current frame are used as target languages.

18. The speech recognition method according to claim 6, characterized in that: Performing language recognition processing on the speech data using the language classification network and the valid frame node to obtain the target language corresponding to the audio data includes: performing language classification processing on the coded features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing, wherein the first language is at least one of the selected languages; If so, the step of using the sub-joint network layer corresponding to the selected language in the joint network layer and the sub-softmax network layer corresponding to the selected language in the softmax network layer to jointly map the merged features to obtain a mapping probability that the speech data corresponds to the selected language includes: using the sub-joint network layer corresponding to the candidate language in the joint network layer and the sub-softmax network layer corresponding to the candidate language in the softmax network layer to jointly map the merged features to obtain a mapping probability that the speech data of the current frame corresponds to the candidate language; wherein the candidate language is the remaining language after excluding the first language from all languages corresponding to the current frame; performing valid frame recognition on the merged features of the candidate language using the valid frame node to obtain a valid frame recognition result for the candidate language, and determining, based on the valid frame recognition result for the candidate language, whether there is a valid frame recognition result for a second language indicating that the current frame is an invalid frame corresponding to the second language; If the second language exists, the language after excluding the second language from the candidate languages is used as the target language.

19. A speech recognition method, characterized in that: include: Receiving input audio data to be recognized and obtaining a preset speech recognition model, wherein the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages; Performing language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and performing speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in a decoding module to obtain an audio recognition result of the audio data corresponding to the target language, wherein the target language is at least one language from a preset number of languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages; Returns the audio recognition result.

20. A speech recognition method, characterized in that: include: Acquiring speech data to be recognized in a current frame, a speech recognition result in a frame previous to the current frame, and a preset speech recognition model, wherein the preset speech recognition model includes a prediction network, an encoding network, and a joint network, the prediction network includes a language embedding layer and a prediction network layer, the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages; Merging the encoded features obtained by encoding the speech data using the encoding network and the predicted features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layers corresponding to each language in the language embedding layer and the prediction network layer to obtain a merged feature, wherein the language embedding layer includes the sub-embedding layers corresponding to the preset number of languages; performing joint mapping processing on the merged features using a sub-joint network layer corresponding to each language in the joint network layer and a sub-softmax network layer corresponding to each language in the softmax network layer to obtain a mapping probability of the speech data corresponding to each language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages; The mapping probability is decoded and recognized using a sub-decoding module corresponding to each language in the decoding module to obtain a speech recognition result corresponding to each language of the speech data.

21. The speech recognition method according to claim 20, characterized in that: The language of the speech data is the target language; The step of using the sub-joint network layer corresponding to each language in the joint network layer and the sub-softmax network layer corresponding to each language in the softmax network layer to jointly map the merged features to obtain the mapping probability of the speech data corresponding to each language includes: using the sub-joint network layer corresponding to the target language in the joint network layer and the sub-softmax network layer corresponding to the target language in the softmax network layer to jointly map the merged features to obtain the mapping probability of the speech data corresponding to the target language; The step of using the sub-decoding module corresponding to each language in the decoding module to decode and identify the mapping probability to obtain the speech recognition result of the speech data corresponding to each language includes: using the sub-decoding module corresponding to the target language in the decoding module to decode and identify the mapping probability to obtain the speech recognition result of the speech data corresponding to the target language.

22. The speech recognition method according to any one of claims 20 to 21, characterized in that: The preset speech recognition model further includes a language classification network, and / or the joint network further includes a valid frame node; The speech recognition method further includes: performing language recognition processing on the speech data using the language classification network and / or the valid frame node to obtain a target language corresponding to the audio data.

23. A speech recognition method, characterized in that: include: Obtaining training speech data of a current frame of a target training sample in a specific language and a training speech recognition result of a previous frame of the current frame, and obtaining an initial speech recognition model, wherein the initial speech recognition model includes an encoding network, a prediction network, and a joint network, the prediction network includes a language embedding layer and a prediction network layer, the joint network includes a joint network layer and a softmax network layer, the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages, wherein the specific language is at least one of the predetermined number of languages predetermined by the initial speech recognition model; Merging the training encoding features obtained by encoding the training speech data using the encoding network and the training prediction features obtained by predicting the training speech recognition result of the previous frame using the sub-embedding layer and the prediction network layer corresponding to the specific language to obtain a training merged feature, wherein the language embedding layer includes the sub-embedding layers corresponding to the preset number of languages; Performing joint mapping processing on the training merged features using a sub-joint network layer corresponding to a specific language in the joint network layer and a sub-softmax network layer corresponding to a specific language in the softmax network layer to obtain a training mapping probability that the training speech data corresponds to the specific language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages; Using a sub-decoding module corresponding to a specific language in the decoding module to perform decoding and recognition processing on the training mapping probability to obtain a training speech recognition result corresponding to the specific language of the training speech data; Updating model parameters in the initial speech recognition model using the training speech recognition result of the current frame and the label data of the current frame of the target training sample, thereby training a preset number of languages to obtain the preset speech recognition model; The preset speech recognition model is used to perform recognition processing on each frame of speech data corresponding to the audio data to be recognized, so as to obtain a speech recognition result of the speech data.

24. The speech recognition method according to claim 23, wherein: The initial speech recognition model further includes a language classification network, and the speech recognition method further includes: Performing language classification processing on the training coding features using the language classification network to determine whether there is a first language that does not require decoding and recognition processing; Parameters of the language classification network are updated according to the result of determining whether there is a first language that does not require decoding and recognition processing and the specific language.

25. The speech recognition method according to claim 23, wherein: The joint network also includes valid frame nodes, and the training merged features are training merged features corresponding to a specific language; The speech recognition method further includes: Using the valid frame node to perform valid frame recognition on the combined features of the specific language to obtain a training valid frame recognition result of the specific language; The parameters of the valid frame node are updated according to the valid training frame recognition result and the label data of the current frame of the target training sample.

26. A speech recognition device, characterized in that: include: a receiving and obtaining module, configured to receive a selected language and obtain a preset speech recognition model, wherein the selected language is at least one of a preset number of languages in the preset speech recognition model; The receiving and obtaining module is further configured to receive input audio data to be recognized; a speech recognition module configured to perform language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and to perform speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in the decoding module, to obtain an audio recognition result corresponding to the target language for the audio data, wherein the target language is at least one of the selected languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages; The return module is used to return the audio recognition result.

27. A speech recognition device, characterized in that: include: A receiving and obtaining module, configured to receive input audio data to be recognized and obtain a preset speech recognition model, wherein the preset speech recognition model is configured to perform speech recognition on speech data of a predetermined number of languages; a speech recognition module configured to perform language recognition processing on the audio data using the preset speech recognition model to obtain a target language corresponding to the audio data, and to perform speech recognition processing on the audio data using an independent parameter module and a shared parameter module for the target language in the preset speech recognition model, and a sub-decoding module corresponding to the target language in the decoding module, to obtain an audio recognition result corresponding to the target language for the audio data, wherein the target language is at least one language from a preset number of languages, and the preset speech recognition model includes independent parameter modules and shared parameter modules for a preset number of languages; The return module is used to return the audio recognition result.

28. A speech recognition device, characterized in that: include: a first acquisition module, configured to acquire speech data to be recognized in a current frame, a speech recognition result in a frame previous to the current frame, and a preset speech recognition model, wherein the preset speech recognition model includes a prediction network, an encoding network, and a joint network, the prediction network includes a language embedding layer and a prediction network layer, the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is used to perform speech recognition on speech data of a predetermined number of languages; a first merging module, configured to merge encoding features obtained by encoding the speech data using the encoding network and prediction features obtained by predicting the speech recognition result of the previous frame using the sub-embedding layer corresponding to each language in the language embedding layer and the prediction network layer, to obtain a merged feature, wherein the language embedding layer includes the sub-embedding layers corresponding to the preset number of languages; a first mapping module, configured to perform a joint mapping process on the merged features using a sub-joint network layer corresponding to each language in the joint network layer and a sub-softmax network layer corresponding to each language in the softmax network layer, so as to obtain a mapping probability of the speech data corresponding to each language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages; The first decoding and recognition module is used to use the sub-decoding modules corresponding to each language in the decoding module to perform decoding and recognition processing on the mapping probability to obtain a speech recognition result corresponding to each language of the speech data.

29. A speech recognition device, characterized in that: include: a second acquisition module, configured to acquire training speech data of a current frame of a target training sample in a specific language and a training speech recognition result of a previous frame of the current frame, and to acquire an initial speech recognition model, wherein the initial speech recognition model includes an encoding network, a prediction network, and a joint network, wherein the joint network includes a joint network layer and a softmax network layer, and the preset speech recognition model is configured to perform speech recognition on speech data of a predetermined number of languages, wherein the specific language is at least one of the predetermined number of languages predetermined by the initial speech recognition model; a second merging module, configured to merge the training coding features obtained by encoding the training speech data using the encoding network and the training prediction features obtained by predicting the training speech recognition result of the previous frame using the prediction network, to obtain a training merged feature; a second mapping module, configured to perform joint mapping processing on the training merged features using a sub-joint network layer corresponding to a specific language in the joint network layer and a sub-softmax network layer corresponding to a specific language in the softmax network layer, so as to obtain a training mapping probability that the training speech data corresponds to a specific language, wherein the joint network layer includes the sub-joint network layers corresponding to the preset number of languages, and the softmax network layer includes the sub-softmax network layers corresponding to the preset number of languages; A second decoding and recognition module is configured to perform decoding and recognition processing on the training mapping probabilities using a sub-decoding module corresponding to a specific language in the decoding module to obtain a training speech recognition result corresponding to each language of the training speech data; An updating module, configured to update model parameters in the initial speech recognition model using the training speech recognition result of the current frame and the label data of the current frame of the target training sample, thereby training a preset number of languages to obtain the preset speech recognition model; The recognition module is used to use the preset speech recognition model to perform recognition processing on each frame of speech data corresponding to the audio data to be recognized, so as to obtain a speech recognition result of the speech data.

30. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the speech recognition method according to any one of claims 1 to 25.

31. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor executes the steps of the speech recognition method according to any one of claims 1 to 25 by calling the computer program stored in the memory.

Images