End-to-end multilingual speech recognition method based on hybrid expert model

Abstract

The present invention discloses an end-to-end multilingual speech recognition method based on a hybrid expert model, comprising: obtaining speech to be recognized from a user end; processing the speech to be recognized using a trained multilingual speech recognition model based on a hybrid expert model, outputting speech recognition results and language categories; and feeding back the speech recognition results and language categories to the user end. The present invention also provides an end-to-end multilingual speech recognition training device based on a hybrid expert model, an electronic device for an end-to-end multilingual speech recognition method based on a hybrid expert model, and a storage medium. The end-to-end multilingual speech recognition method based on a hybrid expert model provided by the present invention can realize multilingual speech recognition, supports end-to-end multilingual models, and has high recognition efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a multilingual speech recognition method, system, electronic device and storage medium based on a hybrid expert model. Background Art

[0002] With the development of artificial intelligence, speech recognition has been widely used in all walks of life. In multilingual speech recognition methods and systems, Transformer networks are often used as acoustic models. With the development of MOE (Mix of Expert) models, multilingual speech recognition methods and systems based on MOE models have gradually become a research hotspot.

[0003] However, in the prior art, speech methods and / or systems based on MOE models have many technical problems, such as poor speech recognition effect, low speech recognition efficiency, weak generalization ability of speech recognition models, complex training process and high training cost of speech recognition models, and inability to realize speech recognition in multiple languages. Summary of the invention

[0004] In view of this, the present invention provides a multilingual speech recognition method, system, electronic device and storage medium based on a hybrid expert model, in order to solve at least one of the above problems.

[0005] According to a first aspect of the present invention, there is provided an end-to-end multilingual speech recognition method based on a hybrid expert model, comprising:

[0006] Obtain the speech to be recognized from the user end;

[0007] Using the trained multilingual speech recognition model based on the hybrid expert model to process the speech to be recognized, and output the speech recognition result and language category;

[0008] Feedback the speech recognition results and language categories to the user end;

[0009] The trained multilingual speech recognition model based on the hybrid expert model is obtained through the following training process:

[0010] Using the audio training data and the language encoding, multilingual audio training data is obtained, wherein the multilingual audio training data includes audio data and / or audio feature frame data with annotated text and language category;

[0011] Randomly extract multiple data from the multilingual audio training data as a batch data group;

[0012] Randomly initialize the parameters of the multilingual speech recognition model based on the hybrid expert model, and use the multilingual speech recognition model based on the hybrid expert model to perform speech processing on the batch data group to obtain speech recognition results and language recognition results;

[0013] Input the annotated text of the batch data group, the language category of the batch data group, the speech recognition result, and the language recognition result into the cross loss function for calculation to obtain a cross loss value;

[0014] According to the cross loss value, optimize the parameters of the multilingual speech recognition model based on the hybrid expert model;

[0015] The extraction operation, speech processing operation, calculation operation and optimization operation are iterated until the cross loss value meets the preset conditions, thereby obtaining a trained multilingual speech recognition model based on the hybrid expert model.

[0016] According to an embodiment of the present invention, the above-mentioned use of audio training data and language encoding to obtain multilingual audio training data includes:

[0017] Using one-hot encoding, a language code is added to the header of the annotated text of the audio training data. The language code is used to indicate the language category of the audio training data.

[0018] According to an embodiment of the present invention, the above-mentioned hybrid expert model includes an encoder and a decoder, wherein the encoder includes a shared embedding network and a multi-layer hybrid expert neural network, the shared embedding network is shared by all layers of the hybrid expert neural network, each layer of the hybrid expert neural network includes a routing layer, a non-expert component and multiple expert components, the non-expert component includes a convolutional neural network and a self-attention module, and each expert component includes a feedforward neural network or a fully connected layer.

[0019] According to an embodiment of the present invention, each layer of the hybrid expert neural network determines the expert component to be activated from multiple expert components based on the output value of the routing layer of the hybrid expert neural network in the previous layer; wherein the shared embedding network and the routing layer of each layer of the hybrid expert neural network are used to control the path selection strategy of each layer of the hybrid expert neural network.

[0020] According to a second aspect of the present invention, there is provided an end-to-end multilingual speech recognition training device based on a hybrid expert model, comprising:

[0021] A language encoding module, used to obtain multi-language audio training data using audio training data and language encoding, wherein the multi-language audio training data includes audio data and / or audio feature frame data with annotated text and language category;

[0022] A data extraction module, used to randomly extract multiple pieces of data from multilingual audio training data as a batch processing data group;

[0023] A speech processing module, used for randomly initializing the parameters of the multilingual speech recognition model based on the hybrid expert model, and performing speech processing on the batch data group using the multilingual speech recognition model based on the hybrid expert model to obtain speech recognition results and language recognition results;

[0024] A loss calculation module, used for inputting the annotated text of the batch data group, the language category of the batch data group, the speech recognition result and the language recognition result into the cross loss function for calculation to obtain a cross loss value;

[0025] A parameter optimization module, used for optimizing the parameters of the multilingual speech recognition model based on the hybrid expert model according to the cross loss value;

[0026] The iterative update module is used to iteratively perform extraction operations, speech processing operations, calculation operations, and optimization operations until the cross loss value meets the preset conditions, thereby obtaining a trained multilingual speech recognition model based on a hybrid expert model.

[0027] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0028] one or more processors;

[0029] a storage device for storing one or more programs,

[0030] When one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned end-to-end multilingual speech recognition method and training method based on the hybrid expert model.

[0031] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute an end-to-end multilingual speech recognition method and training method based on a hybrid expert model.

[0032] The speech recognition method provided by the present invention can obtain a multilingual speech recognition model that can efficiently and accurately recognize speech in multiple languages ​​by training a multilingual speech model based on a hybrid expert model. Compared with speech recognition of a single language, the performance is greatly improved; at the same time, end-to-end rapid speech recognition can be achieved, the dependence on hardware is reduced, and the user experience is greatly improved; in addition, the multilingual speech recognition training method based on the hybrid expert model provided by the present invention can simplify the training, data processing, testing, model management and online links, train an end-to-end multilingual model, support N (N is a positive integer greater than or equal to 2) language recognition, is simple and efficient, and because the hybrid expert model is used, the recognition performance is significantly improved when the parameters are increased, but the calculation amount of the model will not be increased at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of an end-to-end multilingual speech recognition method based on a hybrid expert model according to an embodiment of the present invention;

[0034] Figure 2 is a flow chart of an end-to-end multilingual speech recognition training method based on a hybrid expert model according to an embodiment of the present invention;

[0035] Figure 3 is a structural diagram of an end-to-end multilingual speech recognition training device based on a hybrid expert model according to an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of language encoding of audio training data according to an embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of the training process of a multilingual speech recognition model according to an embodiment of the present invention;

[0038] Figure 6 is a schematic diagram of a multilingual speech recognition method according to an embodiment of the present invention;

[0039] Figure 7 The block diagram of an electronic device suitable for implementing an end-to-end multilingual speech recognition method and a training method thereof based on a hybrid expert model according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0041] Figure 1 4 is a flow chart of an end-to-end multilingual speech recognition method based on a hybrid expert model according to an embodiment of the present invention.

[0042] like Figure 1 As shown, the above end-to-end multi-language speech recognition method based on the hybrid expert model includes operations S110 to S130.

[0043] In operation S110, a speech to be recognized is acquired from a user terminal.

[0044] In operation S120, the trained multilingual speech recognition model based on the hybrid expert model is used to process the speech to be recognized, and the speech recognition result and language category are output.

[0045] In operation S130, the speech recognition result and the language category are fed back to the user end.

[0046] The speech recognition method provided by the present invention can obtain a multilingual speech recognition model that can efficiently and accurately recognize speech in multiple languages ​​by training a multilingual speech model based on a hybrid expert model. Compared with speech recognition in a single language, the performance is greatly improved; at the same time, end-to-end rapid speech recognition can be achieved, the dependence on hardware is reduced, and the user experience is greatly improved.

[0047] Figure 2 4 is a flowchart of an end-to-end multilingual speech recognition training method based on a hybrid expert model according to an embodiment of the present invention.

[0048] The multilingual speech recognition model provided by the present invention is as follows Figure 2 The training method shown is obtained, wherein the above-mentioned multilingual speech model is based on a mixed expert model and uses audio training data of multiple language categories to obtain a speech recognition model that can recognize multiple categories of languages.

[0049] like Figure 2 As shown, the above end-to-end multi-language speech recognition training method based on the hybrid expert model includes operations S210 to S260.

[0050] In operation S210, multi-language audio training data is obtained by using the audio training data and the language encoding, wherein the multi-language audio training data includes audio data and / or audio feature frame data with annotated text and language category.

[0051] The above-mentioned use of audio training data and language encoding to obtain multilingual audio training data includes using one-hot encoding to add a language code to the header of the annotated text of the audio training data, and the language code is used to indicate the language category of the audio training data.

[0052] For example, if there are K languages, then add language embedding codes 1-K to the head of the annotated text corresponding to the audio training data, where the embedding code uses one-hot coding.

[0053] The audio training data may be speech segments in any language, such as English audio segments or audio feature frames, or Portuguese audio segments or audio feature frames.

[0054] The audio training data processed as above has a total of N hours, where N is a positive number.

[0055] In operation S220, a plurality of data are randomly extracted from the multi-language audio training data as a batch data group.

[0056] B data are randomly selected from N hours to form a batch, denoted as (x, y_dict), where x is the audio or audio feature, and y_dic includes the speech recognition annotation results and the audio language category.

[0057] In operation S230, the parameters of the multilingual speech recognition model based on the hybrid expert model are randomly initialized, and the multilingual speech recognition model based on the hybrid expert model is used to perform speech processing on the batch data group to obtain a speech recognition result and a language recognition result.

[0058] Take x in batch(x,y_dict) as input and input it into M to get the recognition result y_result, which includes the language type result and speech recognition result.

[0059] In operation S240, the annotated text of the batch data set, the language category of the batch data set, the speech recognition result, and the language recognition result are input into a cross loss function for calculation to obtain a cross loss value.

[0060] Send y_result and y_dict to the CE loss function to calculate the loss and update the neural network M (i.e., the multilingual speech recognition model).

[0061] In operation S250, parameters of the multilingual speech recognition model based on the hybrid expert model are optimized according to the cross loss value.

[0062] In operation S260, an extraction operation, a speech processing operation, a calculation operation, and an optimization operation are iteratively performed until the cross loss value meets a preset condition, thereby obtaining a trained multilingual speech recognition model based on a hybrid expert model.

[0063] This patent invents an end-to-end multilingual speech recognition method based on hybrid experts and its training method, which solves the problem of a single model supporting speech recognition in multiple languages. It does not rely on other language classification models or language classification modules, and at the same time, it improves the multilingual speech recognition performance with larger model parameters without increasing the computational complexity of the model.

[0064] According to an embodiment of the present invention, the above-mentioned hybrid expert model includes an encoder and a decoder, wherein the encoder includes a shared embedding network and a multi-layer hybrid expert neural network, the shared embedding network is shared by all layers of the hybrid expert neural network, each layer of the hybrid expert neural network includes a routing layer, a non-expert component and multiple expert components, the non-expert component includes a convolutional neural network and a self-attention module, and each expert component includes a feedforward neural network or a fully connected layer.

[0065] According to an embodiment of the present invention, each layer of the hybrid expert neural network determines the expert component to be activated from multiple expert components based on the output value of the routing layer of the hybrid expert neural network in the previous layer; the shared embedding network and the routing layer of each layer of the hybrid expert neural network are used to control the path selection strategy of each layer of the hybrid expert neural network.

[0066] The expert components to be activated in the first layer of the hybrid expert neural network are selected according to preset conditions, and the expert components to be activated in the hybrid expert neural network in other layers are selected according to the output values ​​of the hybrid expert neural network in the previous layer.

[0067] According to an embodiment of the present invention, the output value of the routing layer is determined by formula (1):

[0068] r l =W l *concat(e;O l-1 ) (1),

[0069] Among them, r l represents the output value of the routing layer of the l-layer hybrid expert neural network, W l represents the weight of the routing layer of the l-layer hybrid expert neural network, e represents the output of the shared embedding network, O l-1 Represents the output of the l-1 layer mixed expert neural network, where l is a positive integer greater than 1.

[0070] According to an embodiment of the present invention, the above path selection strategy is determined by formula (2):

[0071]

[0072] Where k represents the number of expert components in each layer of the hybrid expert neural network, 1≤i≤k, represents the probability of the i-th expert component being selected in the l-th layer of the hybrid expert neural network, represents the output value of the i-th layer of the routing layer in the l-th layer hybrid expert neural network;

[0073] Among them, the output of the l-th layer of hybrid expert neural network is determined by formula (3):

[0074]

[0075] in, Represents the output of the i-th expert component in the l-th layer expert neural network.

[0076] The above formulas (2) to (3) show that It represents the probability of the i-th expert component of the l-th layer of hybrid expert neural network being selected, and its value range is [0-1]. Only one expert component is selected in each layer of hybrid expert neural network. Therefore, the output of each layer of expert neural network is shown in formula (3).

[0077] According to an embodiment of the present invention, the cross loss function includes a shared embedding loss function, a classification loss function, and a sparse loss function based on the L1 paradigm.

[0078] According to an embodiment of the present invention, the cross loss function includes a shared embedding loss function, a classification loss function, and a sparse loss function based on the L1 paradigm.

[0079] Figure 3 4 is a structural diagram of an end-to-end multilingual speech recognition training device based on a hybrid expert model according to an embodiment of the present invention.

[0080] like Figure 3 As shown, the end-to-end multi-language speech recognition training device 300 based on the hybrid expert model includes a language encoding module 310, a data extraction module 320, a speech processing module 330, a loss calculation module 340, a parameter optimization module 350, and an iterative update module 360.

[0081] The language encoding module 310 is used to obtain multi-language audio training data using the audio training data and the language encoding, wherein the multi-language audio training data includes audio data and / or audio feature frame data with annotated text and language category.

[0082] The data extraction module 320 is used to randomly extract multiple pieces of data from the multi-language audio training data as a batch processing data group.

[0083] The speech processing module 330 is used to randomly initialize the parameters of the multilingual speech recognition model based on the hybrid expert model, and use the multilingual speech recognition model based on the hybrid expert model to perform speech processing on the batch data group to obtain speech recognition results and language recognition results.

[0084] The loss calculation module 340 is used to input the annotated text of the batch data group, the language category of the batch data group, the speech recognition result and the language recognition result into the cross loss function for calculation to obtain the cross loss value.

[0085] The parameter optimization module 350 is used to optimize the parameters of the multilingual speech recognition model based on the hybrid expert model according to the cross loss value.

[0086] The iterative update module 360 ​​is used to iteratively perform extraction operations, speech processing operations, calculation operations, and optimization operations until the cross loss value meets a preset condition, thereby obtaining a trained multilingual speech recognition model based on a hybrid expert model.

[0087] Figure 4 2 is a schematic diagram of language encoding of audio training data according to an embodiment of the present invention.

[0088] Figure 5 Schematic diagram of the training process of a multilingual speech recognition model according to an embodiment of the present invention.

[0089] Figure 6 is a schematic diagram of a multilingual speech recognition method according to an embodiment of the present invention.

[0090] Combine the following Figures 4 to 6 The above-mentioned speech recognition method and speech model training method provided by the present invention are further described in detail.

[0091] like Figure 4 As shown, the audio training data itself carries annotated text for representing the speech recognition result. The dictionary module uses one-hot encoding to perform language encoding on the audio training data with the annotated text to obtain multilingual audio training data.

[0092] exist Figure 5 In Figure 4 The multilingual audio training data obtained by the process shown is input into the MOE Transformer model (a multilingual speech recognition model based on a hybrid expert model) for model training. In the model, a hybrid expert Transformer is used. A hybrid expert Transformer means that TransformerFFN is used as an expert. There are multiple FFNs in each layer, such as k experts. The method of selecting which expert to use is to use the network output of the previous layer of the neural network as O l-1, and shared embeddings and weights W l To select FFN, FFN1, FFN2, FFN3…FFNk of each layer, first use formulas (1) and (2), that is: l =W l *concat(e;O l-1 ), j from 1 to k, k is the number of experts set in advance, such as 32, 48, etc.; the one with the largest value is selected The corresponding i, as the corresponding activated expert, is That is, in the lth layer, the i-th FFN is selected for activation. Since only one FFN is activated at a time, the amount of calculation is consistent with the traditional Transformer when the parameters are greatly increased. The amount of calculation will not increase significantly, and experts are automatically selected according to different voices, which greatly improves the performance of the model. During training, the language encoding embedding corresponding to the audio is first inserted into the header of the speech recognition training data text, and then the language-related annotated text and audio or feature frames are used as inputs of the hybrid expert model to train the model until convergence is reached and the training is terminated. The decoding of the present invention uses a Transformer decoder for decoding, so the training and recognition are consistent, and no additional n-grams are required.

[0093] exist Figure 6 In the process of recognition, the hybrid expert multilingual model trained by the present invention is used, and the input is speech of K languages ​​to obtain the recognition result and the corresponding language signal. The first position of the recognition result is the language code; the other outputs are the speech recognition results of the hybrid expert multilingual model.

[0094] Figure 7 A block diagram of an electronic device suitable for implementing a method according to an embodiment of the present invention is schematically shown.

[0095] like Figure 7As shown, the electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0096] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0097] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0098] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0099] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0100] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0101] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An end-to-end multilingual speech recognition method based on a hybrid expert model, comprising: Obtain the speech to be recognized from the user end; Using the trained multilingual speech recognition model based on the hybrid expert model to process the speech to be recognized, and output the speech recognition result and language category; Feedback the speech recognition result and the language category to the user end; The trained multilingual speech recognition model based on the hybrid expert model is obtained through the following training process: Using the audio training data and the language encoding, multilingual audio training data is obtained, wherein the multilingual audio training data includes audio data and / or audio feature frame data with annotated text and language category; Randomly extracting multiple pieces of data from the multilingual audio training data as a batch data group; Randomly initializing the parameters of a multilingual speech recognition model based on a hybrid expert model, and performing speech processing on the batch data group using the multilingual speech recognition model based on the hybrid expert model to obtain a speech recognition result and a language recognition result; Inputting the annotated text of the batch data group, the language category of the batch data group, the speech recognition result, and the language recognition result into a cross loss function for calculation to obtain a cross loss value; Optimizing the parameters of the multilingual speech recognition model based on the hybrid expert model according to the cross loss value; The extraction operation, the speech processing operation, the calculation operation and the optimization operation are iterated until the cross loss value meets the preset condition, thereby obtaining the trained multilingual speech recognition model based on the hybrid expert model.

2. The method according to claim 1, wherein: The method of obtaining multilingual audio training data by using audio training data and language encoding includes: By using one-hot encoding, a language code is added to the header of the annotated text of the audio training data, and the language code is used to indicate the language category of the audio training data.

3. The method according to claim 1, wherein: The hybrid expert model includes an encoder and a decoder, wherein the encoder includes a shared embedding network and a multi-layer hybrid expert neural network, the shared embedding network is shared by all layers of the hybrid expert neural network, each layer of the hybrid expert neural network includes a routing layer, a non-expert component and multiple expert components, the non-expert component includes a convolutional neural network and a self-attention module, and each of the expert components includes a feedforward neural network or a fully connected layer.

4. The method according to claim 3, wherein: Each layer of the hybrid expert neural network determines the expert component to be activated from the multiple expert components according to the output value of the routing layer of the hybrid expert neural network in the previous layer; The shared embedding network and the routing layer of each layer of the hybrid expert neural network are used to control the path selection strategy of each layer of the hybrid expert neural network.

5. The method according to claim 1, wherein: The cross loss function includes a shared embedding loss function, a classification loss function, and a sparse loss function based on the L1 paradigm.

6. An end-to-end multilingual speech recognition training device based on a hybrid expert model, comprising: A language encoding module, used to obtain multi-language audio training data using audio training data and language encoding, wherein the multi-language audio training data includes audio data and / or audio feature frame data with annotated text and language category; A data extraction module, used for randomly extracting a plurality of data from the multilingual audio training data as a batch processing data group; A speech processing module, used for randomly initializing the parameters of the multilingual speech recognition model based on the hybrid expert model, and performing speech processing on the batch data group using the multilingual speech recognition model based on the hybrid expert model to obtain a speech recognition result and a language recognition result; A loss calculation module, used for inputting the annotated text of the batch data group, the language category of the batch data group, the speech recognition result and the language recognition result into a cross loss function for calculation to obtain a cross loss value; A parameter optimization module, used for optimizing the parameters of the multilingual speech recognition model based on the hybrid expert model according to the cross loss value; The iterative update module is used to iteratively perform extraction operations, speech processing operations, calculation operations and optimization operations until the cross loss value meets a preset condition, thereby obtaining the trained multilingual speech recognition model based on the hybrid expert model.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 5.

Images