Speech recognition method, apparatus and device, and storage medium and program product

Abstract

Provided in the embodiments of the present application are a speech recognition method, apparatus and device, and a storage medium and a program product. The method comprises: acquiring speech to be processed that includes a language code, and a corresponding first speech scenario; when the first speech scenario is a non-streaming scenario, inputting said speech into a trained first speech recognition model, so as to obtain first recognition text, wherein the first speech recognition model comprises a first encoder network and a first decoder network, and the first encoder network comprises a plurality of first linear layers that are respectively configured corresponding to different languages, and first selection layers connected to the first linear layers; and when the first speech scenario is a streaming scenario, inputting said speech into a trained second speech recognition model that matches the language code, so as to obtain second recognition text, wherein the second speech recognition model comprises a second encoder network and a second decoder network, and the second decoder network comprises a plurality of decoders that are respectively configured corresponding to different languages.

Description

Speech recognition method, device, equipment, storage medium and program product

[0001] The present application claims priority to the Chinese patent application No. 202410477168.5, filed on April 19, 2024, with the Chinese Patent Office, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of computer technology, in particular to a speech recognition method, device, equipment, storage medium and program product. BACKGROUND

[0003] With the rapid development of computer technology, human-computer voice interaction / speech recognition has become very common in modern life. In people's daily applications, there are often needs for speech recognition or voice translation of different languages. When the related business of speech recognition or voice translation involves multiple languages or multiple dialects, it is necessary to separately optimize the speech recognition model for each language or dialect, which will lead to repeated and tedious performance optimization process of the business, increasing the training time cost and model maintenance cost. In addition, for small languages such as Bengali, Arabic and Hindi, the existing labeled data in the terminal is less and the labeling period is longer, which is easy to cause the model training data of a small language to be too small, resulting in poor model recognition ability. In related technologies, by directly mixing training data corresponding to multiple different languages, a single speech recognition model is trained using the training data, and a speech recognition model suitable for multiple languages can be obtained.

[0004] However, since the speech data characteristics of different speech businesses are different, the recognition ability requirements of the speech recognition model are also different, and directly using mixed multi-language training data to train a single speech recognition model cannot adapt to the data characteristics of different speech businesses for speech recognition, which needs to be improved. SUMMARY

[0005] Embodiments of the present application provide a speech recognition method, device, equipment, storage medium and program product, which solve the problem that a speech recognition model obtained by directly using mixed multi-language training data to train a single speech recognition model cannot adapt to the data characteristics of different speech businesses for speech recognition, can effectively optimize the recognition performance of the speech recognition model for each language in different speech scenarios, improve the word-level accuracy of small languages, and provide strong protection for related businesses of voice interaction or voice translation.

[0006] In a first aspect, embodiments of the present application provide a speech recognition method, which comprises:

[0007] obtaining a to-be-processed voice containing a language code and a corresponding first voice scene;

[0008] in a case where the first voice scene is a non-streaming scene, inputting the to-be-processed voice into a trained first voice recognition model to obtain first recognition text, the first voice recognition model comprising a first encoder network and a first decoder network, the first encoder network comprising a plurality of first linear layers respectively corresponding to different language settings and a first selection layer connected with the first linear layers, each first selection layer being configured to output a processing result of a corresponding connected first linear layer when matching the language code of the to-be-processed voice;

[0009] in a case where the first voice scene is a streaming scene, inputting the to-be-processed voice into a trained second voice recognition model matched with the language code to obtain second recognition text, the second voice recognition model comprising a second encoder network and a second decoder network, the second decoder network comprising a plurality of decoders respectively corresponding to different language settings, each decoder being configured to output a corresponding decoding result when matching the language code of the to-be-processed voice.

[0010] In a second aspect, the embodiments of the present application further provide a voice recognition device, which comprises:

[0011] a to-be-processed data obtaining module configured to obtain a to-be-processed voice containing a language code and a corresponding first voice scene;

[0012] a first voice recognition module configured to, in a case where the first voice scene is a non-streaming scene, input the to-be-processed voice into a trained first voice recognition model to obtain first recognition text, the first voice recognition model comprising a first encoder network and a first decoder network, the first encoder network comprising a plurality of first linear layers respectively corresponding to different language settings and a first selection layer connected with the first linear layers, each first selection layer being configured to output a processing result of a corresponding connected first linear layer when matching the language code of the to-be-processed voice;

[0013] a second voice recognition module configured to, in a case where the first voice scene is a streaming scene, input the to-be-processed voice into a trained second voice recognition model matched with the language code to obtain second recognition text, the second voice recognition model comprising a second encoder network and a second decoder network, the second decoder network comprising a plurality of decoders respectively corresponding to different language settings, each decoder being configured to output a corresponding decoding result when matching the language code of the to-be-processed voice.

[0014] In a third aspect, the embodiments of the present application further provide a voice recognition device, which comprises:

[0015] one or more processors;

[0016] a storage configured to store one or more programs,

[0017] when the one or more programs are executed by the one or more processors, the one or more processors implement the speech recognition method according to the embodiments of the present application.

[0018] In a fourth aspect, the embodiments of the present application further provide a non-transitory storage medium storing computer-executable instructions that, when executed by a computer processor, configure to perform the speech recognition method according to the embodiments of the present application.

[0019] In a fifth aspect, the embodiments of the present application further provide a computer program product, which includes a computer program stored in a computer readable storage medium, and at least one processor of a device reads and executes the computer program from the computer readable storage medium, so that the device performs the speech recognition method according to the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1 is a flowchart of a speech recognition method according to an embodiment of the present application;

[0021] FIG. 2 is a schematic diagram of a network structure of an example existing speech recognition model;

[0022] FIG. 3 is a flowchart of a speech recognition method according to an embodiment of the present application, which includes a process of applying a third speech recognition model for speech recognition in a streaming scenario;

[0023] FIG. 4 is a flowchart of a speech recognition method according to an embodiment of the present application, which includes a training process of a first speech recognition model and a second speech recognition model;

[0024] FIG. 5 is a flowchart of a process of training the first speech recognition model according to an embodiment of the present application;

[0025] FIG. 6 is a flowchart of a method of obtaining a first encoding vector through a first encoder network according to an embodiment of the present application;

[0026] FIG. 7 is a process schematic diagram of a method of obtaining a first encoding vector through a first decoder network according to an embodiment of the present application;

[0027] FIG. 8 is a flowchart of another method of training the first speech recognition model according to an embodiment of the present application;

[0028] FIG. 9 is a flowchart of a method of training the second speech recognition model according to an embodiment of the present application;

[0029] FIG. 10 is a flowchart of a method for obtaining a second probability score sequence through a second decoder network according to an embodiment of the present application;

[0030] FIG. 11 is a process diagram of a method for obtaining a predicted vector through a second decoder network according to an embodiment of the present application;

[0031] FIG. 12 is a flowchart of a speech recognition method including a training process of a first speech recognition model and a third speech recognition model according to an embodiment of the present application;

[0032] FIG. 13 is a flowchart of a speech recognition method including a process of training based on training speech in multiple dialects according to an embodiment of the present application;

[0033] FIG. 14 is a structural block diagram of a speech recognition device according to an embodiment of the present application;

[0034] FIG. 15 is a structural diagram of a speech recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] The embodiments of the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, and not to limit the embodiments of the present application. In addition, it should be noted that, for the sake of description, only the parts related to the embodiments of the present application are shown in the drawings, rather than all the structures.

[0036] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally represents a "or" relationship between the front and rear associated objects.

[0037] The speech recognition method provided by the embodiments of the present application can be used for multi-lingual speech recognition tasks in different speech scenarios. The method can be applied to speech live broadcast, speech social interaction, speech teaching, etc. The related speech services include speech real-time text conversion, speech real-time assistant, speech real-time translation, speech review, etc. The foregoing several application scenarios are only exemplary and explanatory, and in actual application, the speech recognition method can also be used in data transmission in other scenarios, which is not limited by the embodiments of the present application.

[0038] The execution subject of each step of the speech recognition method provided in the embodiments of the present application can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as a mobile phone, a PC (Personal Computer), a tablet computer and the like, or a server or the like, and the embodiments of the present application do not limit this.

[0039] FIG. 1 is a flowchart of a speech recognition method provided in the embodiments of the present application. As shown in FIG. 1, the method comprises the following steps:

[0040] In step S101, a to-be-processed speech containing a language code and a corresponding first speech scenario are obtained.

[0041] The language code is an encoding used to indicate the language of the to-be-processed speech. The to-be-processed speech can be audio data generated by a related speech service and requiring speech recognition. The first speech scenario can be divided into a streaming scenario and a non-streaming scenario. The streaming scenario corresponds to a high real-time speech service, for example, speech real-time conversion, speech real-time assistant, speech real-time translation and the like. In the streaming scenario, the to-be-processed speech obtained is non-sentence data with a short continuous time length. The non-streaming scenario corresponds to a low real-time speech service, for example, speech review and the like. In the non-streaming scenario, the to-be-processed speech obtained is sentence data with a long continuous time length.

[0042] In step S102, in the case where the first speech scenario is a non-streaming scenario, the to-be-processed speech is input into a trained first speech recognition model to obtain first recognition text, wherein the first speech recognition model comprises a first encoder network and a first decoder network. The first encoder network comprises a plurality of first linear layers respectively corresponding to different language settings and a first selection layer connected with the first linear layers. Each first selection layer is used to output the processing result of the corresponding connected first linear layer when the language code of the to-be-processed speech matches.

[0043] The first speech recognition model can be applied to a speech recognition task in a non-streaming scenario. The first speech recognition model can first perform acoustic feature extraction on the input speech to be processed, extract key information from the original speech information, and convert the key information into a sequence form recognizable by a computer. Then, the first speech recognition model can perform feature extraction and conversion on the input sequence through the first encoder network, and finally generate a corresponding text sequence through the first decoder network according to the output of the first encoder. FIG. 2 is a schematic diagram of a network structure of an example existing speech recognition model. As shown in FIG. 2, the existing speech recognition model includes an encoder network 101 and a decoder network 102. The encoder network can include a plurality of encoders 1011 connected in series. The decoder network can include a plurality of decoders 1021 connected in series. Since the speech to be processed in a non-streaming scenario is a long sentence data with a long duration, the first encoder network can provide rich context information for the encoding process. If the first encoder network is only provided with a plurality of encoders connected in series, the same encoding process is used for different languages of the speech to be processed, which can not adapt to the unique characteristics of different languages for encoding, affecting the multi-language speech recognition effect. In order to better capture the feature information of different languages, in the first encoder network, a corresponding first linear layer and a first selection layer connected to the first linear layer can be provided for each language. The plurality of first linear layers can be used to receive the output of the encoder. For example, in the case where the first encoder network includes a plurality of encoders connected in series, the plurality of first linear layers and the corresponding first selection layers can be determined according to the computing resource configuration and the model recognition effect of the actual application scenario. The first linear layer corresponding to each language can be used to better capture the key features unique to the language, and input the key features to the connected first selection layer. The first selection layer corresponding to each language can be used to output the processing result of the connected first linear layer when the language of the speech to be processed matches the language code. For example, the first decoder network corresponds to language A, language B, and language C, and is respectively provided with linear layer A and selection layer A connected thereto, linear layer B and selection layer B connected thereto, and linear layer C and selection layer C connected thereto. If the language code of the input speech to be processed corresponds to language A, the first speech recognition model sets selection layer A to an output state, for example, sets the vector multiplication factor of selection layer A to 1, and sets selection layer B and selection layer C to a non-output state, for example, sets the vector multiplication factor of selection layer B and C to 0. After linear layer A, linear layer B, and linear layer C receive the output result of the encoder and complete the processing, the results are output to selection layer A, selection layer B, and selection layer C, respectively. Only selection layer A can output the processing result of linear layer A, so as to strengthen the feature extraction of language A corresponding to the speech to be processed.Therefore, by introducing the first linear layer and the first selection layer, the first encoder network can better adapt to feature coding of different languages, which is beneficial to optimizing the multi-language speech recognition effect. In addition, the encoder structure in the first encoder network and the first decoder network can be configured with reference to the open-source Transformer network structure, which is not limited herein.

[0044] In step S103, in the case of the first speech scenario being a streaming scenario, the to-be-processed speech is input into the trained second speech recognition model matched with the language code to obtain second recognition text, wherein the second speech recognition model includes a second encoder network and a second decoder network, the second decoder network includes a plurality of decoders respectively corresponding to different languages, and each decoder is configured to output a corresponding decoding result when the language code of the to-be-processed speech matches.

[0045] The second speech recognition model can be applied to a speech recognition task in a streaming scenario. The second speech recognition model can first extract acoustic features from the input to-be-processed speech, extract key information from the original speech information, and convert it into a computer-recognizable sequence form. Then, the second encoder network is used to extract and convert the features of the input sequence. Finally, the second decoder network generates a corresponding text sequence according to the output of the second encoder. Since the to-be-processed speech provided by the streaming scenario is non-sentence data with a short duration in the data stream, the information amount of the to-be-processed speech is relatively small. In order to better optimize the output of the recognition text, in the second encoder network, a corresponding decoder can be set for each language. Each language corresponding decoder can be used to better decode and output based on the unique characteristics of different languages. Each language corresponding decoder can be used to output a corresponding decoding result when the language code of the to-be-processed speech matches the language. For example, the second decoder network corresponds to language A, language B, and language C, and is provided with decoder A, decoder B, and decoder C, respectively. If the language code of the input to-be-processed speech corresponds to language A, the decoder A can be selected for decoding and output, achieving the purpose of strengthening the text output of the to-be-processed speech corresponding to language A. Therefore, by setting decoders corresponding to different languages, the second encoder network can better adapt to text output of different languages, which is beneficial to optimizing the multi-language speech recognition effect. In addition, the decoder structure in the second encoder network and the second decoder network can be configured with reference to the open-source Transformer network structure, which is not limited herein.

[0046] The above scheme, corresponding to different speech scenes, the first speech recognition model and the second speech recognition model are respectively set, the first speech recognition model can effectively strengthen the feature extraction ability of the model corresponding to different languages through the multiple first linear layers and the first selection layers of the first encoder network thereof, and is suitable for the sentence recognition application in the non-streaming scene, the second speech recognition model corresponding to each language can effectively strengthen the text recognition ability of the model corresponding to each language through the multiple decoders of the second decoder network thereof, and is suitable for the real-time recognition application in the streaming scene, therefore, the recognition performance of the speech recognition model for each language in different speech scenes can be effectively optimized, the word-level accuracy of the small language is improved, and strong guarantee is provided for the related business of voice interaction or voice translation.

[0047] FIG. 3 is a flowchart of a speech recognition method provided by an embodiment of the present application, which includes a process of applying a third speech recognition model for speech recognition in a streaming scene. As shown in FIG. 3, the process includes the following steps:

[0048] In step S201, a to-be-processed speech containing a language code and a corresponding first speech scene are obtained.

[0049] In step S202, in a case where the first speech scene is a non-streaming scene, the to-be-processed speech is input into a trained first speech recognition model to obtain first recognition text, wherein the first speech recognition model includes a first encoder network and a first decoder network, the first encoder network includes multiple first linear layers respectively corresponding to different languages and first selection layers connected with the first linear layers, and each first selection layer is configured to output a processing result of a corresponding connected first linear layer when the language code of the to-be-processed speech matches.

[0050] In step S203, in the case where the first voice scene is a streaming scene, the voice to be processed is input into a trained third voice recognition model matched with the language code to obtain third recognition text, wherein the third voice recognition model comprises a second encoder network and a third decoder network, the third decoder network comprises a plurality of second linear layers respectively corresponding to different language settings and a second selection layer connected with the second linear layer, and each second selection layer is configured to output the processing result of the corresponding connected second linear layer when matched with the language code of the voice to be processed.

[0051] The second speech recognition model and the third speech recognition model can both be applied to a speech recognition task in a streaming scenario, and both use the same second encoder network. In an actual streaming scenario, one of the two can be selected for speech recognition. In this embodiment, the third speech recognition model can be used to recognize the to-be-processed speech in a streaming scenario. The third speech recognition model can first extract acoustic features from the input to-be-processed speech, extract key information from the original speech information, and convert it into a computer-recognizable sequence form, then perform feature extraction and conversion on the input sequence through the second encoder network, and finally generate a corresponding text sequence through the third decoder network according to the output of the second encoder. Referring to the network structure shown in FIG. 2, if the third decoder network is only provided with a plurality of decoders connected in series, the same decoding process is used for to-be-processed speech of different languages, which may not be able to adapt to the decoding of text characteristics of different languages, affecting the text recognition effect of multi-language speech. In order to better correspond to different languages for text output, in this embodiment of the application, a corresponding second linear layer and a second selection layer connected to the second linear layer can be provided for each language in the third decoder network. The plurality of second linear layers can be used to receive the output of the decoder. For example, in the case where the third decoder network includes a plurality of decoders connected in series, the plurality of second linear layers and the corresponding second selection layers can be determined to be provided after which decoders according to the computing resource configuration and model recognition effect of the actual application scenario. The second linear layer corresponding to each language can be used to better match the text recognition characteristics of the language for output, and input the output result into the connected second selection layer. The second selection layer corresponding to each language can be used to output the processing result of the connected second linear layer when the language of the to-be-processed speech matches the language code. For example, the second decoder network corresponds to language A, language B, and language C, and is respectively provided with linear layer A and selection layer A connected thereto, linear layer B and selection layer B connected thereto, and linear layer C and selection layer C connected thereto. If the language code of the input to-be-processed speech corresponds to language B, the third speech recognition model will set selection layer B to an output state, for example, set the vector multiplication factor of the selection layer B to 1, and set selection layer A and selection layer C to non-output states, for example, set the vector multiplication factor of the selection layer A and C to 0. Then, after the linear layer A, the linear layer B, and the linear layer C receive the output result of the decoder and complete the processing, the results will be output to the selection layer A, the selection layer B, and the selection layer C, respectively. Only the selection layer B can output the processing result of the linear layer B, achieving the purpose of strengthening the text recognition of the to-be-processed speech corresponding to language B. In addition, the decoder structure in the second encoder network and the second decoder network can be configured by referring to the open-source Transformer network structure, which is not limited herein.

[0052] By introducing the second linear layer and the second selection layer in the third encoder network, the decoding capability of the third encoder network for sequences in different languages is improved, so that the third speech recognition model can better match the language characteristics of different languages for text recognition, and the text recognition effect of multi-language speech is optimized.

[0053] FIG. 4 is a flowchart of a speech recognition method provided by an embodiment of the application, which includes a training process of a first speech recognition model and a second speech recognition model.

[0054] As shown in FIG. 4, the method includes the following steps:

[0055] In step S301, a plurality of training speeches in a plurality of languages, training texts corresponding to each training speech, and a second speech scenario are obtained.

[0056] The training speeches can be audio data generated in the actual development process of related speech services, and the training texts can be text annotations of the plurality of training speeches to provide correct text recognition results. The second speech scenario can be divided into a streaming scenario and a non-streaming scenario.

[0057] In step S302, in a case where the second speech scenario is a non-streaming scenario, each training speech and the corresponding training text are input to a pre-constructed first speech recognition model for training.

[0058] In step S303, in a case where the second speech scenario is a streaming scenario, each training speech and the corresponding training text are input to a pre-constructed second speech recognition model corresponding to each language for training.

[0059] In step S304, a to-be-processed speech containing a language code and a corresponding first speech scenario are obtained.

[0060] In step S305, in a case where the first speech scenario is a non-streaming scenario, the to-be-processed speech is input to the trained first speech recognition model to obtain a first recognition text, wherein the first speech recognition model includes a first encoder network and a first decoder network, the first encoder network includes a plurality of first linear layers respectively corresponding to different languages and a first selection layer connected to the first linear layers, and each first selection layer is configured to output a processing result of the corresponding connected first linear layer when the language code of the to-be-processed speech matches.

[0061] In a case where the first voice scene is a streaming scene, the voice to be processed is input into a trained second voice recognition model matched with the language code to obtain second recognition text, where the second voice recognition model comprises a second encoder network and a second decoder network, the second decoder network comprises a plurality of decoders respectively corresponding to different language settings, and each decoder is configured to output a corresponding decoding result when matched with the language code of the voice to be processed.

[0062] In the above, the corresponding voice recognition models are respectively constructed for the streaming scene and the non-streaming scene, the first voice recognition model is trained by inputting the training data corresponding to the non-streaming scene, so that the first voice recognition model has better voice recognition capability for the non-streaming scene, and the second voice recognition model is trained by inputting the training data corresponding to the streaming scene, so that the second voice recognition model has better voice recognition capability for the streaming scene, which is beneficial to establish the model recognition performance applied to different voice scenes.

[0063] FIG. 5 is a flowchart of a process of training the first voice recognition model according to an embodiment of the present application. As shown in FIG. 5, the process comprises the following steps:

[0064] In step S3021, an acoustic feature vector corresponding to each training voice is extracted, and the acoustic feature vector and a corresponding one-hot vector are spliced according to the language code of each training voice to obtain a first input vector corresponding to each training voice, where the one-hot vector is used to configure the gating function of a plurality of first selection layers in the first encoder network.

[0065] The acoustic feature vector can be used to represent the frequency domain characteristics of the training voice, and the acoustic feature vector can be extracted based on linear predictive coding, mel frequency cepstral coefficient, etc., which is not limited herein. The one-hot vector can be used to represent the language corresponding to the training voice, and each language code is configured with a corresponding one-hot vector, and each training voice can carry a language code used to indicate the language. For example, the training data of Bengali, Arabic and Hindi is used for multi-language joint training of the first voice recognition model, where the one-hot vector corresponding to Bengali is [1, 0, 0], the one-hot vector corresponding to Arabic is [0, 1, 0], and the one-hot vector corresponding to Hindi is [0, 0, 1]. If the language code of the input training voice corresponds to Bengali, the acoustic feature vector of the training voice can be spliced with [1, 0, 0] to obtain the first input vector corresponding to the training voice. In addition, the gating function of a plurality of selection layers in the first encoder network can be configured according to the one-hot vector. For example, if the language code of the input training voice corresponds to Bengali, the selection layer corresponding to Bengali in the first encoder network can be set to an output-enabled state, and the selection layers corresponding to other languages can be set to an output-disabled state.

[0066] Step S3022, mapping the training text to obtain a first text vector corresponding to each training voice.

[0067] The first text vector can be obtained by mapping the training text from high-dimensional data to a lower-dimensional vector representation through the embedding representation layer set in the first language model.

[0068] Step S3023, inputting the first input vector into the first encoder network to obtain a first encoding vector, and inputting the first encoding vector and the first text vector into the first decoder network to obtain a first probability score sequence.

[0069] The first encoder network can effectively capture the semantic and contextual information of the training voice, encode the first input vector to obtain the first encoding vector, and the first decoder network can output the result of each step prediction based on the first encoding vector. At the same time, by combining the first text vector, each step prediction can rely on the true result vector as input, rather than relying on the prediction result generated by the model itself in the previous step, which can reduce the accumulation of errors in the training process of the model, thereby speeding up the training process and improving the performance of the model. In addition, by combining the multi-lingual training dictionary corresponding to the first speech recognition model, the output of the decoder network can be converted into the output of the first probability score sequence through the linear layer and the normalized exponential function layer. The first probability score sequence can represent the prediction probability of each time step in the training voice. The multi-lingual training dictionary can be directly obtained by splicing the dictionary files of each language, or the multi-lingual mixed training text can be normalized to a finer modeling unit through byte pair encoding.

[0070] Step S3024, performing parameter iterative optimization to obtain an optimal first speech recognition model according to the loss value results of the loss function calculation based on the first encoding vector and the first probability score sequence, respectively.

[0071] By introducing the one-hot vector corresponding to the language code of the training voice, the first encoder network can be configured with a first selection layer adapted to the language code of the training voice, which is beneficial to the first encoder network to optimize and train the parameters corresponding to the language of the training voice. The first speech recognition model trained can effectively recognize the voice accurately in a non-streaming scenario according to the characteristics of each language.

[0072] FIG. 6 is a flowchart of a method for obtaining a first encoding vector by a first encoder network according to an embodiment of the present application. The first encoder network includes a first encoder block, a plurality of third selection layers, a second encoder block, and a plurality of fourth selection layers. The first encoder block includes a head-end encoder and a plurality of third linear layers connected to the head-end encoder. The second encoder block includes a plurality of intermediate encoders connected in series, a tail-end encoder, and a plurality of fourth linear layers connected to the tail-end encoder. As shown in FIG. 6, the method includes the following steps:

[0073] In step S30231, a first input vector is input to the first encoder block to obtain a plurality of first feature vectors, and a first target feature vector matching a language of a current training speech is extracted from the plurality of first feature vectors by the plurality of third selection layers.

[0074] After the first input vector is input to the first encoder block, the first input vector is first processed by the head-end encoder to obtain a first intermediate encoding result. The first intermediate encoding result is input to each third linear layer, respectively, to obtain a plurality of first feature vectors output by the plurality of third linear layers. Each third linear layer is connected to a third selection layer. Since the third selection layer is configured to correspond to a one-hot vector in the first input vector, only the third selection layer in an output state can output the input first feature vector. The output first feature vector is the first target feature vector matching the language of the current training speech.

[0075] In step S30232, a first result vector obtained by adding the first target feature vector and a first intermediate encoding vector generated by the head-end encoder is input to the second encoder block to obtain a plurality of second feature vectors, and a second target feature vector matching the language of the current training speech is extracted from the plurality of second feature vectors by the plurality of fourth selection layers.

[0076] The first result vector can be an intermediate result of feature enhancement processing corresponding to the language of the current training speech on the first intermediate encoding vector. After the first result vector is input to the second encoder block, the first result vector is first processed by the plurality of intermediate encoders connected in series, and then a final second intermediate encoding result is input to each fourth linear layer by the tail-end encoder, respectively, to obtain a plurality of second feature vectors output by the plurality of fourth linear layers. Each fourth linear layer is connected to a fourth selection layer. Since the fourth selection layer is configured to correspond to the one-hot vector in the first input vector, only the fourth selection layer in an output state can output the input second feature vector. The output second feature vector is the second target feature vector matching the language of the current training speech.

[0077] Step S30233, add the second target feature vector and the second intermediate encoding vector generated by the end encoder to obtain a first encoding vector.

[0078] In this embodiment, the first encoding vector can be the feature enhancement processing result corresponding to the output of the head-end encoder and the end encoder respectively for the language of the current training speech.

[0079] Through the multiple third linear layers and third selection layers, and multiple fourth linear layers and fourth selection layers provided in the first encoder network, the feature extraction of the language characteristics corresponding to the current training speech can be effectively enhanced during the encoding process, so that the output first encoding vector can be more conducive to subsequent decoding processing to obtain accurate text recognition results.

[0080] In one embodiment, FIG. 7 is a process diagram of a method for obtaining a first encoding vector by a first decoder network according to an embodiment of the present application. As shown in FIG. 7, the first input vector obtained by splicing the acoustic feature vector of the training speech and the corresponding one-hot vector is input into the first encoder block 201. In the first encoder block 201, the first input vector is first input into the first encoder 2011. The first encoder 2011 can be an encoding structure with an output dimension of 512 dimensions of the attention layer and a head number of 8 of the multi-head attention. The vector x1 processed by the first encoder 2011 is input into N linear layers 2012 with a size of 512 dimensions respectively, and the output results x11-x1N can be obtained. The output results x11-x1N are input into corresponding selection layers 202 for vector multiplication respectively. The vector values of each selection layer are determined by the one-hot vector corresponding to the training speech. For example, if the one-hot vector corresponding to the training speech is [1, 0, 0], the vector value of the first selection layer 202 is [[0]], the vector value of the second selection layer 202 is [[0]], and the vector value of the third selection layer 202 is [[0]]. The N output vectors processed by the plurality of selection layers 202 are added to the vector x1 output by the first encoder 2011 to obtain the vector x2. The vector x2 is input into the second encoder block 203. In the second encoder block 203, the vector x2 is sequentially input into a plurality of encoders 2031. The number of the plurality of encoders 2031 can be 12. Each encoder 2031 can be an encoding structure with an output dimension of 512 dimensions of the attention layer and a head number of 8 of the multi-head attention. The vector x3 processed by the last encoder 2031 is input into N linear layers 2032 with a size of 512 dimensions respectively, and the output results x31-x3N can be obtained. The output results x31-x3N are input into corresponding selection layers 204 for vector multiplication respectively. The N output vectors processed by the plurality of selection layers 204 are added to the vector x3 output by the last encoder 2031 to obtain the vector y1. The vector y1 is used for subsequent loss function calculation and second decoder network calculation.

[0081] FIG. 8 is a flowchart of another method for training a first speech recognition model according to an embodiment of the present application. As shown in FIG. 8, the method comprises the following steps:

[0082] In step S3021, the acoustic feature vector corresponding to each training speech is extracted. The acoustic feature vector and the corresponding one-hot vector are spliced according to the language code of each training speech to obtain the first input vector corresponding to each training speech. The one-hot vector is used to configure the gating function of the plurality of first selection layers in the first encoder network.

[0083] In step S3022, the training text is mapped to obtain the first text vector corresponding to each training speech.

[0084] Step S3023, input the first input vector into the first encoder network to obtain a first encoding vector, and input the first encoding vector and the first text vector into the first decoder network to obtain a first probability score sequence.

[0085] Step S3024, calculate a first loss value based on the first encoding vector and a corresponding real label vector by using a first loss function, and calculate a second loss value based on the first probability score sequence and a corresponding real text sequence by using a second loss function.

[0086] The first loss function can be a CTC (Connectionist Temporal Classification) loss function or other sequence-to-sequence loss function, which is not limited herein. The second loss function can be a cross-entropy loss function, a negative log-likelihood loss function, etc., which is not limited herein.

[0087] Step S3025, according to a weighted calculation result of the first loss value and the second loss value, perform parameter iterative optimization on the first speech recognition model through back propagation until the first loss function and the second loss function converge to obtain an optimal first speech recognition model.

[0088] The first loss function and the second loss function can be combined by weighting, which is used to represent the overall loss of the recognition result of the first speech recognition model. Through multiple rounds of training of parameter iterative optimization, the first loss function and the second loss function can converge to obtain an optimal first speech recognition model.

[0089] Through the weighted fusion calculation of the first loss function and the second loss function, the parameters of the first encoder network and the second decoder network can be effectively optimized in parallel, and the feature extraction capability of the first encoder network and the sequence conversion capability of the second decoder network can be simultaneously improved, so as to achieve the purpose of optimizing the speech recognition performance of the first speech recognition model.

[0090] FIG. 9 is a flowchart of a method for training a second speech recognition model according to an embodiment of the present application. As shown in FIG. 9, the method comprises the following steps:

[0091] Step S3031, extract an acoustic feature vector corresponding to each training speech.

[0092] Step S3032, mapping the training text to obtain a first text vector corresponding to each training voice, mapping a language index corresponding to each training voice to obtain a language index vector, and splicing the first text vector and the language index vector to obtain a second input vector corresponding to each training voice, wherein the language index vector is used to determine a decoder corresponding to each training voice.

[0093] The language index can be a serial number set for each language, starting from serial number 0. For example, if the total number of languages is n, the value range of the language index corresponding to the training audio is 0-n-1. The language index vector can be obtained by mapping the language index to a lower-dimensional vector representation through an embedding representation layer set in the second language model. Optionally, since the dimension of the vector obtained by splicing the first text vector and the language index vector can be different from the output dimension of the attention layer in the decoder, the second input vector consistent with the output dimension of the attention layer can be obtained by remapping through the set linear layer after splicing. In addition, the language index vector can be used to determine the decoder corresponding to each training voice. For example, if the language code of the input training voice corresponds to Bengali, the decoder corresponding to Bengali in the second decoder network is selected to decode the second input vector and output.

[0094] Step S3033, inputting the acoustic feature vector into the second encoder network to obtain a second encoding vector, and inputting the second encoding vector and the second input vector into the second decoder network to obtain a second probability score sequence.

[0095] Step S3034, performing parameter iterative optimization to obtain an optimal second speech recognition model corresponding to each language according to the loss value results of the loss function calculation based on the second encoding vector and the second probability score sequence, respectively.

[0096] By introducing the language index corresponding to the training voice, the second decoder network can select the decoder corresponding to the language index when processing the second input vector, which is beneficial to the second decoder network to perform parameter optimization training based on the language to which the training voice belongs, so that the trained second speech recognition model can effectively recognize the voice accurately in the streaming scenario according to the characteristics of each language.

[0097] FIG. 10 is a flowchart of a method for obtaining a second probability score sequence through a second decoder network according to an embodiment of the present application. The second decoder network further includes a fifth linear layer, a sixth linear layer, and a normalized exponential function layer, as shown in FIG. 10, including the following steps:

[0098] Step S30331, inputting the second input vector into the fifth linear layer to obtain an intermediate vector.

[0099] In the second input vector is the splicing result of the first text vector and the language index vector, the dimensions may not be consistent with the dimensions of the attention layer in the decoder, and the fifth linear layer can be used to remap the intermediate vector to the dimensions consistent with the attention layer in the decoder. For example, the first text vector obtained by mapping the training text is consistent with the 512 dimensions of the attention layer, and the language index vector obtained by mapping the language index is 128 dimensions, and the second input vector obtained by splicing is remapped to 512 dimensions by the fifth linear layer, which is consistent with the output dimension of the attention layer.

[0100] Step S30332, input the intermediate vector and the second encoding vector into the decoder matched with the language index of the current training speech to obtain a predicted sequence.

[0101] The predicted sequence can be a prediction result obtained by decoding and recognizing in combination with the language characteristics of the current training speech.

[0102] Step S30333, sequentially pass the predicted sequence through the sixth linear layer and the normalization exponential function layer to obtain a second probability score sequence.

[0103] The second probability score sequence can represent the prediction probability of each time step in the training speech that may correspond to a text unit, and is used for calculation of a loss function to update the model parameters of the second speech recognition model.

[0104] The above, by selecting the decoder corresponding to the language of the current training speech in the second decoder network to process the intermediate vector and the second encoding vector, the second decoder network can effectively adapt to different language characteristics to output text recognition results, and the accuracy of speech recognition is improved.

[0105] In one embodiment, FIG. 11 is a process diagram of a method for obtaining a predicted vector by a second decoder network according to an embodiment of the present application. As shown in FIG. 11, the training text is input into the embedding representation layer 301 to obtain a first text vector of 512 dimensions, and the language index is input into the embedding representation layer 302 to obtain a language index vector of 128 dimensions. Splicing the first text vector and the language index vector can obtain a vector k1, and remapping the vector k1 through the linear layer 303 can obtain a vector x2 of 512 dimensions. The vector x2 is input into the decoder 304 corresponding to the language index of the current training speech for processing, and the output results of other decoders 304 are 0. The decoder 304 can be a 6-layer decoding structure with 8 heads of multi-head attention, and the processing results of the multiple decoders 304 are added to obtain a vector y2, which can be used for further processing through the linear layer and the normalization exponential function layer to obtain a second probability score sequence.

[0106] FIG. 12 is a flowchart of a speech recognition method provided by an embodiment of the present application, which includes a training process of a first speech recognition model and a third speech recognition model. As shown in FIG. 12, the method includes the following steps:

[0107] In step S401, a plurality of training speeches in a plurality of languages, training texts corresponding to each training speech, and a second speech scenario are obtained.

[0108] In step S402, in a case where the second speech scenario is a non-streaming scenario, each training speech and the corresponding training text are input to a pre-constructed first speech recognition model for training.

[0109] In step S403, in a case where the second speech scenario is a streaming scenario, each training speech and the corresponding training text are input to a pre-constructed third speech recognition model corresponding to each language for training.

[0110] In step S404, a to-be-processed speech containing a language code and a corresponding first speech scenario are obtained.

[0111] In step S405, in a case where the first speech scenario is a non-streaming scenario, the to-be-processed speech is input to the trained first speech recognition model to obtain a first recognition text, wherein the first speech recognition model includes a first encoder network and a first decoder network, the first encoder network includes a plurality of first linear layers respectively arranged for different languages and a first selection layer connected to the first linear layers, and each first selection layer is configured to output a processing result of a corresponding connected first linear layer when the language code of the to-be-processed speech matches.

[0112] In step S406, in a case where the first speech scenario is a streaming scenario, the to-be-processed speech is input to the trained third speech recognition model matched with the language code to obtain a third recognition text, wherein the third speech recognition model includes a second encoder network and a third decoder network, the third decoder network includes a plurality of second linear layers respectively arranged for different languages and a second selection layer connected to the second linear layers, and each second selection layer is configured to output a processing result of a corresponding connected second linear layer when the language code of the to-be-processed speech matches.

[0113] In the above, the corresponding speech recognition models are constructed for the streaming scenario and the non-streaming scenario respectively, the first speech recognition model is trained by inputting the training data corresponding to the non-streaming scenario, so that the first speech recognition model has better speech recognition capability for the non-streaming scenario, the third speech recognition model is trained by inputting the training data corresponding to the streaming scenario, so that the third speech recognition model has better speech recognition capability for the streaming scenario, which is conducive to establishing the model recognition performance applied to different speech scenarios.

[0114] FIG. 13 is a flowchart of a speech recognition method provided by an embodiment of the present application, which includes a process of training based on multiple dialects. The multiple training speeches include training speeches corresponding to multiple dialects in the same language. As shown in FIG. 13, the process includes the following steps:

[0115] In step S501, multiple training speeches of multiple languages, training texts corresponding to each training speech, and a second speech scenario are obtained.

[0116] In step S502, in a case where the second speech scenario is a non-streaming scenario, each training speech and the corresponding training text are input to a pre-constructed first speech recognition model for training.

[0117] In step S503, based on the training speech corresponding to each dialect, fine-tuning training is performed on the trained first speech recognition model to enable the first speech recognition model to perform parameter adjustment suitable for a speech recognition task corresponding to each dialect.

[0118] In step S504, in a case where the second speech scenario is a streaming scenario, each training speech and the corresponding training text are input to a pre-constructed second speech recognition model corresponding to each language for training.

[0119] In step S505, based on the training speech corresponding to each dialect, fine-tuning training is performed on the trained second speech recognition model to enable the second speech recognition model to perform parameter adjustment suitable for a speech recognition task corresponding to each dialect.

[0120] In step S506, a to-be-processed speech including a language code and a corresponding first speech scenario are obtained.

[0121] In step S507, in a case where the first speech scenario is a non-streaming scenario, the to-be-processed speech is input to the trained first speech recognition model to obtain a first recognition text. The first speech recognition model includes a first encoder network and a first decoder network. The first encoder network includes multiple first linear layers respectively corresponding to different languages and a first selection layer connected to the first linear layers. Each first selection layer is configured to output a processing result of a corresponding connected first linear layer when the language code of the to-be-processed speech matches.

[0122] In step S508, in a case where the first speech scenario is a streaming scenario, the to-be-processed speech is input to the trained second speech recognition model matched with the language code to obtain a second recognition text. The second speech recognition model includes a second encoder network and a second decoder network. The second decoder network includes multiple decoders respectively corresponding to different languages. Each decoder is configured to output a corresponding decoding result when the language code of the to-be-processed speech matches.

[0123] By determining that multiple dialects of a certain language exist in the multiple languages corresponding to the training data, the optimization of the dialect recognition performance of the model trained based on the mixed training data can be continued. By re-inputting the training data of each dialect into the trained first speech recognition model and the second speech recognition model for fine-tuning training, the first speech recognition model and the second speech recognition model corresponding to each dialect can be obtained, and the accuracy of the model in recognizing dialect speech can be improved.

[0124] FIG. 14 is a structural block diagram of a speech recognition device provided by an embodiment of the present application, which is configured to execute the speech recognition method provided by the above-described embodiments and has the corresponding function modules and beneficial effects of the execution method. As shown in FIG. 14, the device includes:

[0125] The obtaining module 401 is configured to obtain the to-be-processed speech containing the language code and the corresponding first speech scenario.

[0126] The first speech recognition module 402 is configured to, in the case where the first speech scenario is a non-streaming scenario, input the to-be-processed speech into the trained first speech recognition model to obtain the first recognition text. The first speech recognition model includes a first encoder network and a first decoder network. The first encoder network includes multiple first linear layers respectively corresponding to different languages and a first selection layer connected to the first linear layers. Each first selection layer is used to output the processing result of the corresponding connected first linear layer when the language code of the to-be-processed speech matches.

[0127] The second speech recognition module 403 is configured to, in the case where the first speech scenario is a streaming scenario, input the to-be-processed speech into the trained second speech recognition model matched with the language code to obtain the second recognition text. The second speech recognition model includes a second encoder network and a second decoder network. The second decoder network includes multiple decoders respectively corresponding to different languages. Each decoder is used to output the corresponding decoding result when the language code of the to-be-processed speech matches.

[0128] The above scheme includes the following steps: obtaining a to-be-processed voice containing a language code and a corresponding first voice scene; in a case where the first voice scene is a non-streaming scene, inputting the to-be-processed voice into a trained first voice recognition model to obtain first recognition text, the first voice recognition model including a first encoder network and a first decoder network, the first encoder network including a plurality of first linear layers respectively corresponding to different languages and a first selection layer connected to the first linear layers, each first selection layer being configured to output a processing result of a corresponding connected first linear layer when the language code of the to-be-processed voice matches; in a case where the first voice scene is a streaming scene, inputting the to-be-processed voice into a trained second voice recognition model matched with the language code to obtain second recognition text, the second voice recognition model including a second encoder network and a second decoder network, the second decoder network including a plurality of decoders respectively corresponding to different languages, each decoder being configured to output a corresponding decoding result when the language code of the to-be-processed voice matches. In the above scheme, the first voice recognition model and the second voice recognition model are respectively set according to different voice scenes, the first voice recognition model can effectively enhance the feature extraction capability of the model corresponding to different languages through the plurality of first linear layers and the first selection layer of the first encoder network, and is suitable for sentence recognition application in a non-streaming scene, and the second voice recognition model corresponding to each language can effectively enhance the text recognition capability of the model corresponding to each language through the plurality of decoders of the second decoder network, and is suitable for real-time recognition application in a streaming scene, so that the recognition performance of the voice recognition model for each language in different voice scenes can be effectively optimized, the word-level accuracy of a small language can be improved, and strong support can be provided for related businesses of voice interaction or voice translation.

[0129] In one possible embodiment, the method further includes a third voice recognition module configured to:

[0130] inputting the to-be-processed voice into a trained third voice recognition model matched with the language code to obtain third recognition text, the third voice recognition model including a second encoder network and a third decoder network, the third decoder network including a plurality of second linear layers respectively corresponding to different languages and a second selection layer connected to the second linear layers, each second selection layer being configured to output a processing result of a corresponding connected second linear layer when the language code of the to-be-processed voice matches.

[0131] In one possible embodiment, the method further includes a first model training module configured to:

[0132] obtaining a plurality of training voices of a plurality of languages, training text corresponding to each training voice, and a second voice scene;

[0133] In the case that the second voice scene is a non-streaming scene, each training voice and the corresponding training text are input to the pre-constructed first voice recognition model for training.

[0134] In the case that the second voice scene is a streaming scene, each training voice and the corresponding training text are input to the pre-constructed second voice recognition model corresponding to each language for training.

[0135] In one possible embodiment, further comprising a second model training module configured to:

[0136] Obtain a plurality of training voices of a plurality of languages, training texts corresponding to each training voice, and a second voice scene;

[0137] In the case that the second voice scene is a non-streaming scene, each training voice and the corresponding training text are input to the pre-constructed first voice recognition model for training.

[0138] In the case that the second voice scene is a streaming scene, each training voice and the corresponding training text are input to the pre-constructed third voice recognition model corresponding to each language for training.

[0139] In one possible embodiment, the first model training module is further configured to:

[0140] Extract an acoustic feature vector corresponding to each training voice, and splice the acoustic feature vector and a corresponding one-hot vector according to the language code of each training voice to obtain a first input vector corresponding to each training voice, the one-hot vector being used for configuring the gating function of a plurality of first selection layers in the first encoder network;

[0141] Map the training text to obtain a first text vector corresponding to each training voice;

[0142] Input the first input vector to the first encoder network to obtain a first encoding vector, and input the first encoding vector and the first text vector to the first decoder network to obtain a first probability score sequence;

[0143] According to the loss value results of the loss function calculation based on the first encoding vector and the first probability score sequence respectively, the optimal first voice recognition model is obtained through parameter iteration optimization.

[0144] In one possible embodiment, the first model training module is further configured to:

[0145] Based on the first encoding vector and the corresponding real label vector, a first loss function is calculated to obtain a first loss value, and based on the first probability score sequence and the corresponding real text sequence, a second loss function is calculated to obtain a second loss value.

[0146] According to the weighted calculation result of the first loss value and the second loss value, parameter iterative optimization is performed on the first speech recognition model through back propagation until the first loss function and the second loss function converge to obtain an optimal first speech recognition model.

[0147] In one possible embodiment, the second model training module is further configured to:

[0148] extract an acoustic feature vector corresponding to each training speech;

[0149] map the training text to obtain a first text vector corresponding to each training speech, map a language index corresponding to each training speech to obtain a language index vector, and splice the first text vector and the language index vector to obtain a second input vector corresponding to each training speech, the language index vector being used to determine a decoder corresponding to each training speech;

[0150] input the acoustic feature vector into the second encoder network to obtain a second encoding vector, and input the second encoding vector and the second input vector into the second decoder network to obtain a second probability score sequence;

[0151] According to the loss value results of the loss function calculation based on the second encoding vector and the second probability score sequence, parameter iterative optimization is performed to obtain an optimal second speech recognition model corresponding to each language.

[0152] In one possible embodiment, the first encoder network includes a first encoder block, a plurality of third selection layers, a second encoder block, and a plurality of fourth selection layers, the first encoder block including a head-end encoder and a plurality of third linear layers connected to the head-end encoder, and the second encoder block including a plurality of intermediate encoders connected in series, a tail-end encoder, and a plurality of fourth linear layers connected to the tail-end encoder.

[0153] Correspondingly, the first model training module is further configured to:

[0154] input the first input vector into the first encoder block to obtain a plurality of first feature vectors, and extract a first target feature vector matching the language of the current training speech from the plurality of first feature vectors through the plurality of third selection layers;

[0155] input a first result vector obtained by adding the first target feature vector and a first intermediate encoding vector generated by the head-end encoder into the second encoder block to obtain a plurality of second feature vectors, and extract a second target feature vector matching the language of the current training speech from the plurality of second feature vectors through the plurality of fourth selection layers;

[0156] add the second target feature vector and a second intermediate encoding vector generated by the tail-end encoder to obtain a first encoding vector.

[0157] In a possible embodiment, the second decoder network further includes a fifth linear layer, a sixth linear layer, and a normalized exponential function layer;

[0158] The second model training module is further configured to:

[0159] input the second input vector into the fifth linear layer to obtain an intermediate vector;

[0160] input the intermediate vector and the second encoding vector into a decoder matched with the language index of the current training speech to obtain a predicted sequence;

[0161] input the predicted sequence into the sixth linear layer and the normalized exponential function layer in sequence to obtain a second probability score sequence.

[0162] In a possible embodiment, the plurality of training speeches include training speeches corresponding to a plurality of dialects in the same language;

[0163] The second model training module is further configured to:

[0164] based on the training speech corresponding to each dialect, fine-tune training the first speech recognition model that has been trained to enable the first speech recognition model to perform parameter adjustment adapted to the speech recognition task corresponding to each dialect;

[0165] based on the training speech corresponding to each dialect, fine-tune training the second speech recognition model that has been trained to enable the second speech recognition model to perform parameter adjustment adapted to the speech recognition task corresponding to each dialect.

[0166] FIG. 15 is a structural schematic diagram of a speech recognition device provided in an embodiment of the present application. As shown in FIG. 15, the device includes a processor 501, a memory 502, an input device 503, and an output device 504. The number of processors 501 in the device can be one or more, and one processor 501 is taken as an example in FIG. 15. The processor 501, the memory 502, the input device 503, and the output device 504 in the device can be connected through a bus or other means, and connection through a bus is taken as an example in FIG. 15. The memory 502, as a computer readable storage medium, can be configured to store software programs, computer executable programs, and modules, such as program instructions / modules of the speech recognition method in the embodiment of the present application. The processor 501 executes various function applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 502, that is, implements the speech recognition method described above. The input device 503 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 504 can include a display device such as a display screen.

[0167] The embodiment of the present application further provides a nonvolatile storage medium containing computer executable instructions, which are configured to perform the voice recognition method described in the above embodiment when executed by a computer processor, and the method comprises the following steps:

[0168] obtaining to-be-processed voice containing a language code and a corresponding first voice scene;

[0169] in a case where the first voice scene is a non-streaming scene, inputting the to-be-processed voice into a trained first voice recognition model to obtain first recognition text, the first voice recognition model comprising a first encoder network and a first decoder network, the first encoder network comprising a plurality of first linear layers respectively corresponding to different language settings and a first selection layer connected with the first linear layers, each first selection layer being configured to output a processing result of a corresponding connected first linear layer when matching the language code of the to-be-processed voice;

[0170] in a case where the first voice scene is a streaming scene, inputting the to-be-processed voice into a trained second voice recognition model matched with the language code to obtain second recognition text, the second voice recognition model comprising a second encoder network and a second decoder network, the second decoder network comprising a plurality of decoders respectively corresponding to different language settings, each decoder being configured to output a corresponding decoding result when matching the language code of the to-be-processed voice.

[0171] It is worth noting that the embodiments of the voice recognition device described above include various units and modules only according to logical functions, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy mutual distinction, and do not configure to limit the protection scope of the embodiments of the present application.

[0172] In some possible implementation manners, each aspect of the method provided by the present application can also be implemented in the form of a program product, which includes program codes. When the program product runs on a computer device, the program codes are configured to make the computer device execute the steps in the method according to various exemplary embodiments of the present application described in the specification, for example, the computer device can execute the voice recognition method described in the embodiments of the present application. The program product can be realized by any combination of one or more readable media.

Claims

1. A speech recognition method, wherein, The method comprises the following steps: acquiring a to-be-processed voice containing a language code and a corresponding first voice scene; in a case where the first voice scene is a non-streaming scene, inputting the to-be-processed voice into a trained first voice recognition model to obtain first recognition text, the first voice recognition model comprising a first encoder network and a first decoder network, the first encoder network comprising a plurality of first linear layers respectively corresponding to different language settings and a first selection layer connected to the first linear layers, each first selection layer being configured to output a processing result of a corresponding connected first linear layer when the language code of the to-be-processed voice matches the first selection layer; in a case where the first voice scene is a streaming scene, inputting the to-be-processed voice into a trained second voice recognition model matched with the language code to obtain second recognition text, the second voice recognition model comprising a second encoder network and a second decoder network, the second decoder network comprising a plurality of decoders respectively corresponding to different language settings, each decoder being configured to output a corresponding decoding result when the language code of the to-be-processed voice matches the decoder.

2. The voice recognition method of claim 1, wherein, in a case where the first voice scene is a streaming scene, the method further comprises the following steps: inputting the to-be-processed voice into a trained third voice recognition model matched with the language code to obtain third recognition text, the third voice recognition model comprising the second encoder network and a third decoder network, the third decoder network comprising a plurality of second linear layers respectively corresponding to different language settings and a second selection layer connected to the second linear layers, each second selection layer being configured to output a processing result of a corresponding connected second linear layer when the language code of the to-be-processed voice matches the second selection layer.

3. The voice recognition method according to claim 1 or 2, wherein, before the step of acquiring the to-be-processed voice containing the language code and the corresponding first voice scene, the method further comprises the following steps: acquiring a plurality of training voices of a plurality of languages, training text corresponding to each training voice, and a second voice scene; in a case where the second voice scene is a non-streaming scene, inputting each training voice and corresponding training text into a pre-constructed first voice recognition model for training; in a case where the second voice scene is a streaming scene, inputting each training voice and corresponding training text into a pre-constructed second voice recognition model corresponding to each language for training.

4. The speech recognition method according to claim 1 or 2, wherein, before the step of acquiring the to-be-processed voice containing the language code and the corresponding first voice scene, the method further comprises the following steps: acquiring a plurality of training voices of a plurality of languages, training text corresponding to each training voice, and a second voice scene; in a case where the second voice scene is a non-streaming scene, inputting each training voice and corresponding training text into a pre-constructed first voice recognition model for training; in a case where the second voice scene is a streaming scene, inputting each training voice and corresponding training text into a pre-constructed third voice recognition model corresponding to each language for training.

5. The speech recognition method according to claim 3 or 4, wherein, the step of inputting each training voice and corresponding training text into a pre-constructed first voice recognition model for training comprises the following steps: extracting an acoustic feature vector corresponding to each of the training speeches, concatenating the acoustic feature vector and a corresponding one-hot vector according to a language code of each of the training speeches to obtain a first input vector corresponding to each of the training speeches, the one-hot vector being configured to control gating of a plurality of first selection layers in the first encoder network; mapping the training texts to obtain a first text vector corresponding to each of the training speeches; inputting the first input vector into the first encoder network to obtain a first encoding vector, and inputting the first encoding vector and the first text vector into the first decoder network to obtain a first probability score sequence; performing parameter iterative optimization according to loss value results of loss function calculation based on the first encoding vector and the first probability score sequence to obtain an optimal first speech recognition model.

6. The voice recognition method of claim 5, wherein, The parameter iterative optimization according to the loss value results of the loss function calculation based on the first encoding vector and the first probability score sequence to obtain the optimal first speech recognition model comprises: performing first loss function calculation based on the first encoding vector and a corresponding real label vector to obtain a first loss value, and performing second loss function calculation based on the first probability score sequence and a corresponding real text sequence to obtain a second loss value; performing parameter iterative optimization by back-propagating the first speech recognition model according to a weighted calculation result of the first loss value and the second loss value until the first loss function and the second loss function converge to obtain the optimal first speech recognition model.

7. The voice recognition method of claim 3, wherein, The training of each of the training speeches and the corresponding training texts in the pre-constructed second speech recognition model corresponding to each of the languages comprises: extracting an acoustic feature vector corresponding to each of the training speeches; mapping the training texts to obtain a first text vector corresponding to each of the training speeches, mapping a language index corresponding to each of the training speeches to obtain a language index vector, and concatenating the first text vector and the language index vector to obtain a second input vector corresponding to each of the training speeches, the language index vector being configured to determine a decoder corresponding to each of the training speeches; inputting the acoustic feature vector into the second encoder network to obtain a second encoding vector, and inputting the second encoding vector and the second input vector into the second decoder network to obtain a second probability score sequence; performing parameter iterative optimization according to loss value results of loss function calculation based on the second encoding vector and the second probability score sequence to obtain an optimal second speech recognition model corresponding to each of the languages.

8. The voice recognition method of claim 5, wherein, The first encoder network comprises a first encoder block, a plurality of third selection layers, a second encoder block and a plurality of fourth selection layers, the first encoder block comprises a head-end encoder and a plurality of third linear layers connected to the head-end encoder, the second encoder block comprises a plurality of intermediate encoders connected in series, a tail-end encoder and a plurality of fourth linear layers connected to the tail-end encoder; Correspondingly, the inputting the first input vector into the first encoder network to obtain a first encoding vector comprises: inputting the first input vector into the first encoder block to obtain a plurality of first feature vectors, and extracting a first target feature vector matching the language of the current training speech from the plurality of first feature vectors through the plurality of third selection layers; inputting a first result vector obtained by adding the first target feature vector and a first intermediate encoding vector generated by the first end encoder into a second encoder block to obtain a plurality of second feature vectors, and extracting a second target feature vector matching the language of the current training speech from the plurality of second feature vectors through the plurality of fourth selection layers; adding the second target feature vector and a second intermediate encoding vector generated by the last end encoder to obtain a first encoding vector.

9. The voice recognition method of claim 7, wherein, The second decoder network further comprises a fifth linear layer, a sixth linear layer, and a normalized exponential function layer; Correspondingly, the inputting the second encoding vector and the second input vector into the second decoder network to obtain a second probability score sequence comprises: inputting the second input vector into the fifth linear layer to obtain an intermediate vector; inputting the intermediate vector and the second encoding vector into a decoder matching the language index of the current training speech to obtain a prediction sequence; inputting the prediction sequence into the sixth linear layer and the normalized exponential function layer in sequence to obtain a second probability score sequence.

10. The speech recognition method according to claim 3, 7 or 9, wherein, The plurality of training speeches comprises training speeches corresponding to a plurality of dialects in the same language; Correspondingly, after the inputting each of the training speeches and the corresponding training text into the pre-constructed first speech recognition model for training, the method further comprises: based on each of the training speeches corresponding to the dialect, fine-tuning the trained first speech recognition model to adjust the parameters of the first speech recognition model to adapt to the speech recognition task corresponding to each of the dialects; after the inputting each of the training speeches and the corresponding training text into each of the pre-constructed second speech recognition models corresponding to the language for training, the method further comprises: based on each of the training speeches corresponding to the dialect, fine-tuning the trained second speech recognition model to adjust the parameters of the second speech recognition model to adapt to the speech recognition task corresponding to each of the dialects.

11. A speech recognition apparatus, wherein, comprises: an acquisition module configured to acquire a to-be-processed speech containing a language code and a corresponding first speech scene; a first speech recognition module configured to, in a case where the first speech scene is a non-streaming scene, input the to-be-processed speech into a trained first speech recognition model to obtain a first recognition text, the first speech recognition model comprising a first encoder network and a first decoder network, the first encoder network comprising a plurality of first linear layers respectively corresponding to different languages and a first selection layer connected to the first linear layers, each of the first selection layers being configured to output a processing result of a corresponding connected first linear layer when the language code of the to-be-processed speech matches the language code; a second speech recognition module, configured to, in a case that the first speech scenario is a streaming scenario, input the to-be-processed speech into a trained second speech recognition model matched with the language code to obtain second recognition text, the second speech recognition model comprising a second encoder network and a second decoder network, the second decoder network comprising a plurality of decoders respectively corresponding to different language settings, each of the decoders being configured to output a corresponding decoding result when matched with the language code of the to-be-processed speech.

12. A speech recognition device, the device comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the speech recognition method according to any one of claims 1-10.

13. A non-transitory storage medium storing computer-executable instructions that, when executed by a computer processor, are configured to perform the speech recognition method according to any one of claims 1-10.

14. A computer program product comprising a computer program, wherein, the computer program, when executed by a processor, implements the speech recognition method according to any one of claims 1-10.

Images