Speech recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the application discloses a speech recognition method and device, electronic equipment and storage medium, and the method comprises the steps of obtaining a to-be-recognized speech; identifying the to-be-recognized speech through a mixed speech recognition model to obtain a speech recognition result; the mixed speech recognition model comprises a target coding module, a mixed expert model (MoE) module and a target decoding module, the MoE module comprises a routing network and a plurality of expert network groups corresponding to a plurality of language categories one by one, the routing network is used for determining a target expert network group corresponding to an initial speech representation from the plurality of expert network groups according to the initial speech representation, and processing the initial speech representation through the target expert network group to obtain a target speech representation. The embodiment of the application improves the recognition accuracy of the mixed speech containing multiple language categories.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech recognition method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Automatic speech recognition is a technology that automatically transcribes speech into corresponding text, which can provide support for speech content understanding and human-computer interaction, and is widely used in various basic businesses, such as search, voice assistants, automatic subtitles, etc. The accuracy of current speech recognition models for speech containing one language category is already at a usable level, but in many scenarios, speech data can include multiple language categories, such as professional terms in English appearing in Chinese communication. When facing speech containing multiple language categories, such as speech containing Chinese and English, the accuracy of speech recognition is low. SUMMARY

[0003] Embodiments of the present application disclose a speech recognition method, device, electronic device, and storage medium, which improve the recognition accuracy of mixed speech containing multiple language categories.

[0004] Embodiments of the present application disclose a speech recognition method, comprising:

[0005] obtaining speech to be recognized;

[0006] recognizing the speech to be recognized through a mixed speech recognition model to obtain a speech recognition result; the mixed speech recognition model comprises a target encoding module, a MoE module, and a target decoding module; the target encoding module is configured to perform encoding processing on the speech to be recognized to obtain an initial speech representation corresponding to the speech to be recognized; the MoE module comprises a routing network and a plurality of expert network groups corresponding one-to-one to a plurality of language categories; the routing network is configured to determine a target expert network group corresponding to the initial speech representation from the plurality of expert network groups according to the initial speech representation, and perform processing on the initial speech representation through the target expert network group to obtain a target speech representation; and the target decoding module is configured to perform decoding processing according to the target speech representation to obtain the speech recognition result.

[0007] In one embodiment, each expert network group comprises a routing unit and a plurality of expert networks; the routing unit is configured to determine one or more target expert networks from the plurality of expert networks according to the initial speech representation, and perform processing on the initial speech representation through the one or more target expert networks to obtain a target speech representation.

[0008] In one embodiment, after obtaining the speech to be recognized, the method further comprises:

[0009] frame the to-be-recognized speech to obtain a plurality of frames of sub-speech data;

[0010] The speech recognition result is obtained by recognizing the to-be-recognized speech through the mixed speech recognition model, and the method comprises the following steps:

[0011] The plurality of frames of sub-speech data are encoded through the target encoding module to obtain initial speech features corresponding to the plurality of frames of sub-speech data respectively;

[0012] The target frame sub-speech data is any frame of sub-speech data.

[0013] One or more target expert networks are determined from the target expert network group through a routing unit in the target expert network group according to the initial speech feature corresponding to the target frame sub-speech data;

[0014] The initial speech feature corresponding to the target frame sub-speech data is processed through the one or more target expert networks to obtain a fusion speech feature corresponding to the target frame sub-speech data;

[0015] The target speech feature is obtained according to the fusion speech features corresponding to the plurality of frames of sub-speech data respectively;

[0016] The target speech feature is decoded through the target decoding module to obtain a speech recognition result.

[0017] In one embodiment, the target frame sub-speech data corresponding target expert network group is determined from the plurality of expert network groups through the routing network according to the initial speech feature corresponding to the target frame sub-speech data, comprising:

[0018] The target frame sub-speech data corresponding target language category is determined through the routing network according to the initial speech feature corresponding to the target frame sub-speech data, and the expert network group corresponding to the target language category is determined as the target frame sub-speech data corresponding target expert network group.

[0019] In one embodiment, the one or more target expert networks are determined from the target expert network group through the routing unit in the target expert network group according to the initial speech feature corresponding to the target frame sub-speech data, comprising:

[0020] determining one or more target expert networks and a network weight corresponding to each of the target expert networks from the target expert network group according to the initial speech feature corresponding to the target frame sub-speech data by a routing unit in the target expert network group;

[0021] the processing of the initial speech feature corresponding to the target frame sub-speech data by the one or more target expert networks to obtain the fusion speech feature corresponding to the target frame sub-speech data, comprising:

[0022] processing the initial speech feature corresponding to the target frame sub-speech data by the one or more target expert networks respectively to obtain a processing result corresponding to each of the target expert networks;

[0023] weighting and fusing the processing result corresponding to each of the target expert networks based on the network weight corresponding to each of the target expert networks to obtain the fusion speech feature corresponding to the target frame sub-speech data.

[0024] In one embodiment, the target encoding module is obtained based on self-supervised training of a first data set and supervised training of a second data set, and the MoE module and the target decoding module are obtained based on supervised training of the second data set;

[0025] The first data set includes a plurality of unlabeled first sample speeches, each of which corresponds to a language category, and the second data set includes a plurality of second sample speeches and labeled information corresponding to each of the second sample speeches, each of which corresponds to one or more language categories.

[0026] In one embodiment, the method further comprises:

[0027] based on a first data set, self-supervised training of an initial encoding module to obtain a first encoding module, the first encoding module including a plurality of encoding layers;

[0028] based on a second data set, supervised training of a second encoding module to obtain a target encoding module; the second encoding module being obtained by adding a conditional decoding layer between each adjacent two encoding layers in the first encoding module; the conditional decoding layer being used to calculate the posterior probability of the connected previous encoding layer.

[0029] In one embodiment, the method further comprises:

[0030] normalizing the first speech feature extracted by the current encoding layer to obtain a second speech feature;

[0031] calculating the posterior probability corresponding to the second speech feature by the conditional decoding layer connected to the current encoding layer.

[0032] linearly project the posterior probability by a linear projection layer, so that a dimension of the posterior probability is aligned with a dimension of the second speech representation;

[0033] perform residual connection between the mapped posterior probability and the second speech representation to obtain a third speech representation, and input the third speech representation into a next encoding layer of the current encoding layer.

[0034] Embodiments of the present application disclose a speech recognition device, comprising:

[0035] a speech obtaining module configured to obtain a speech to be recognized;

[0036] a speech recognition module configured to recognize the speech to be recognized by using a hybrid speech recognition model to obtain a speech recognition result; the hybrid speech recognition model comprises a target encoding module, a hybrid expert model (MoE) module and a target decoding module; the target encoding module is configured to perform encoding processing on the speech to be recognized to obtain an initial speech representation corresponding to the speech to be recognized; the MoE module comprises a routing network and a plurality of expert network groups corresponding to a plurality of language categories one by one; the routing network is configured to determine a target expert network group corresponding to the initial speech representation from the plurality of expert network groups according to the initial speech representation, and perform processing on the initial speech representation by using the target expert network group to obtain a target speech representation; and the target decoding module is configured to perform decoding processing according to the target speech representation to obtain the speech recognition result.

[0037] In an embodiment, each of the expert network groups comprises a routing unit and a plurality of expert networks; the routing unit is configured to determine one or more target expert networks from the plurality of expert networks according to the initial speech representation, and perform processing on the initial speech representation by using the one or more target expert networks to obtain a target speech representation.

[0038] In an embodiment, the voice acquisition module is further configured to perform frame processing on the voice to be recognized to obtain a plurality of frames of sub-voice data; the voice recognition module is further configured to encode the plurality of frames of sub-voice data by using the target encoding module to obtain initial voice representations corresponding to the plurality of frames of sub-voice data respectively; determine, by using the routing network according to the initial voice representation corresponding to the target frame of sub-voice data, a target expert network group corresponding to the target frame of sub-voice data from the plurality of expert network groups; the target frame of sub-voice data is any frame of sub-voice data; determine, by using the routing unit in the target expert network group according to the initial voice representation corresponding to the target frame of sub-voice data, one or more target expert networks from the target expert network group; process, by using the one or more target expert networks, the initial voice representation corresponding to the target frame of sub-voice data to obtain a fusion voice representation corresponding to the target frame of sub-voice data; obtain a target voice representation according to the fusion voice representations corresponding to the plurality of frames of sub-voice data respectively; and decode, by using the target decoding module, the target voice representation to obtain a voice recognition result.

[0039] In an embodiment, the voice recognition module is further configured to determine, by using the routing network according to the initial voice representation corresponding to the target frame of sub-voice data, a target language category corresponding to the target frame of sub-voice data, and determine the expert network group corresponding to the target language category as the target expert network group corresponding to the target frame of sub-voice data.

[0040] In an embodiment, the voice recognition module is further configured to determine, by using the routing unit in the target expert network group according to the initial voice representation corresponding to the target frame of sub-voice data, one or more target expert networks from the target expert network group and a network weight corresponding to each target expert network; process, by using the one or more target expert networks respectively, the initial voice representation corresponding to the target frame of sub-voice data to obtain a processing result corresponding to each target expert network; and perform weighted fusion on the processing result corresponding to each target expert network based on the network weight corresponding to each target expert network to obtain a fusion voice representation corresponding to the target frame of sub-voice data.

[0041] In an embodiment, the target encoding module is obtained by self-supervised training based on a first data set and supervised training based on a second data set, the MoE module and the target decoding module are obtained by supervised training based on the second data set; the first data set includes a plurality of first sample voices without labels, each first sample voice corresponds to a language category, and the second data set includes a plurality of second sample voices and label information corresponding to each second sample voice, each second sample voice corresponds to one or more language categories.

[0042] In an embodiment, the speech recognition apparatus can further include a model training module configured to perform self-supervised training on the initial encoding module based on a first data set to obtain a first encoding module, the first encoding module including a plurality of encoding layers; perform supervised training on a second encoding module based on a second data set to obtain a target encoding module, the second encoding module being obtained by adding a conditional decoding layer between each two adjacent encoding layers in the first encoding module; and the conditional decoding layer being configured to calculate a posterior probability of a connected previous encoding layer.

[0043] In an embodiment, the model training module is further configured to perform normalization processing on the first speech feature extracted by a current encoding layer to obtain a second speech feature; calculate a posterior probability corresponding to the second speech feature through a conditional decoding layer connected to the current encoding layer; perform linear projection on the posterior probability through a linear projection layer, so that a dimension of the posterior probability is aligned with a dimension of the second speech feature; perform residual connection on the mapped posterior probability and the second speech feature to obtain a third speech feature, and input the third speech feature to a next encoding layer of the current encoding layer.

[0044] Embodiments of the present application disclose an electronic device, comprising:

[0045] a memory storing executable program code;

[0046] a processor coupled to the memory;

[0047] The processor invokes the executable program code stored in the memory to execute the method of any of the above embodiments.

[0048] Embodiments of the present application disclose a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to execute the method of any of the above embodiments.

[0049] The voice recognition method and device, electronic device and storage medium disclosed by the embodiment of the present application can obtain the to-be-recognized voice, and recognize the voice through a mixed voice recognition model to obtain a voice recognition result, wherein the mixed voice recognition model comprises a target encoding module, an MoE module and a target decoding module. The target encoding module can be used for encoding processing on the to-be-recognized voice to obtain an initial voice feature corresponding to the to-be-recognized voice. The MoE module comprises a routing network and a plurality of expert network groups corresponding to a plurality of language categories. The routing network can be used for determining a target expert network group corresponding to the initial voice feature from the plurality of expert network groups according to the initial voice feature, and processing the initial voice feature through the target expert network group to obtain a target voice feature. The target decoding module is used for decoding processing according to the target voice feature to obtain the voice recognition result. According to the embodiment, a plurality of expert network groups can be obtained in the MoE according to the language categories. The routing module in the MoE can route the initial voice feature corresponding to the to-be-recognized voice to the corresponding target expert network group. The language category contained in the to-be-recognized voice can correspond to the language category corresponding to the target expert network. The initial voice feature is processed through the target expert network group, which can accurately capture the voice features of different language categories to obtain the target voice feature, reduces the confusion and misrecognition of the language categories, improves the accuracy of the target voice feature, and thus improves the recognition accuracy of the mixed voice containing a plurality of language categories. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1-A is a structural diagram of a voice recognition model disclosed in the related art;

[0052] Figure 1-A is a structural diagram of another voice recognition model disclosed in the related art;

[0053] Figure 2 is an application scenario diagram of a differential upgrade method disclosed in the embodiments of the present application;

[0054] Figure 3 is a flowchart of a voice recognition method disclosed in the embodiments of the present application;

[0055] Figure 4 is a structural diagram of an MoE module disclosed in the embodiments of the present application;

[0056] Figure 5 is a flowchart of a speech recognition method according to an embodiment of the present application;

[0057] Figure 6 is a flowchart of a training process of a target coding module according to an embodiment of the present application;

[0058] Figure 7 is a schematic diagram of a target coding module according to an embodiment of the present application;

[0059] Figure 8 is a modular schematic diagram of a speech recognition apparatus according to an embodiment of the present application;

[0060] Figure 9 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0062] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present application are intended to cover the inclusions not exclusively, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0063] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first data set can be referred to as the second data set, and similarly, the second data set can be referred to as the first data set. The first data set and the second data set are both data sets, but they are not the same data set.

[0064] For mixed speech containing multiple language types, since the pronunciations of different language types are usually similar, such as the English "I" and the Chinese "ai", it is difficult for a speech recognition model to recognize such words in mixed speech. In the early stage, a cascaded system was usually used to recognize mixed speech. The cascaded system can include a speech boundary detection system, a language type recognition system, and speech recognition systems corresponding to multiple language types respectively. First, the speech is segmented by the language boundary detection system to obtain multiple speech segments, then the language types of the multiple speech segments are classified by the language type recognition system, and finally the corresponding speech recognition system is used to perform speech recognition on each speech segment. However, this cascaded system is prone to cumulative propagation of errors, resulting in low accuracy of speech recognition, and this cascaded system has a high system latency, which greatly affects the experience of human-computer interaction.

[0065] Therefore, the end-to-end neural network model has gradually become the mainstream solution for recognizing mixed speech. The speech recognition model in the related technology can include encoders corresponding to multiple language types. Each encoder can learn the ability to distinguish speech of different language types during the training process, so as to extract speech representations corresponding to multiple language types respectively, and then perform weighted fusion on the speech representations output by the multiple encoders respectively to obtain the final speech representation, thereby improving the accuracy of speech recognition.

[0066] As Figure 1-A shown, Figure 1-A is a structural diagram of a speech recognition model disclosed in the related technology. Among them, the Chinese encoder can extract and process the semantic information in the speech to be recognized according to Chinese rules, and generate the corresponding Chinese speech representation The English encoder can extract and process the semantic information in the speech to be recognized according to English rules, and generate the corresponding English speech representation The gating network can perform weighted fusion on the Chinese speech representation generated by the Chinese encoder and the English speech representation generated by the English encoder to obtain the fused speech representation. Among them, the gating network can determine the weight corresponding to the Chinese encoder as and determine the weight corresponding to the English encoder as Thus, according to the weight corresponding to the Chinese encoder and the weight However, the training process of the voice recognition model is complex and difficult, and the encoder corresponding to each language type needs to be pre-trained, and the voice recognition model has high computational complexity, and the voice recognition efficiency is low.

[0067] As shown in Figure 1-B , Figure 1-B is a structural diagram of another voice recognition model disclosed in the related art, wherein the routing network can calculate the mixed voice through the Softmax function to obtain the probability distribution corresponding to each expert network, and select part of the expert networks from the n expert networks according to the probability distribution corresponding to each expert network, thereby activating the part of the expert networks to perform voice recognition on the mixed voice, and fusing the recognition results of the part of the expert networks to obtain the final recognition result, thereby effectively reducing the computational complexity while keeping the task complexity unchanged. However, the voice recognition model in Figure 1-B needs a large amount of manually labeled data for training to improve the recognition accuracy of the voice recognition model, and the accuracy of the mixed voice containing multiple language types is still not high.

[0068] Embodiments of the present application disclose a voice recognition method and device, electronic equipment and storage medium, by grouping according to language types to obtain a plurality of expert network groups, thereby determining a target expert network group corresponding to the language type corresponding to the voice to be recognized, and processing the initial voice representation corresponding to the voice to be recognized through the target expert network group, thereby obtaining a more accurate target voice representation, which can improve the recognition accuracy of the mixed voice containing multiple language types.

[0069] The following will be described in detail with reference to the accompanying drawings.

[0070] As shown in Figure 2 , Figure 2 is an application scenario diagram of a differential upgrade method disclosed by embodiments of the present application. The application scenario can include an electronic device 210, which can include but is not limited to a mobile phone, a tablet computer, a wearable device, a notebook computer, a PC (Personal Computer), and the like.

[0071] In one embodiment, the electronic device 210 can include a mixed voice recognition model, and the electronic device 210 can obtain a voice to be recognized and recognize the voice to be recognized through the mixed voice recognition model to obtain a voice recognition result. For example, in Figure 2 , the voice to be recognized can be input into the electronic device 210, and the electronic device 210 recognizes the voice to be recognized through the mixed voice recognition model to obtain a voice recognition result "good morning, everyone good morning".

[0072] In view of the feature that the electronic device 210 can perform voice recognition, in some application software, a voice recognition function can be set to facilitate the user to complete a specific operation through voice. For example, in the scenario of the user using a mobile phone voice assistant, the user can send a control instruction to the mobile phone through voice, the mobile phone can detect the user voice, and convert the user voice into target text through a voice recognition model to determine the user's control instruction and respond; in the scenario of conference recording, the conference voice of one or more users can be detected, converted into conference text and recorded.

[0073] As shown in Figure 3 , Fig. 1 is a flow diagram of a voice recognition method according to an embodiment of the present application. The voice recognition method can be applied to the electronic device in the above embodiments. The voice recognition method can include the following steps: Figure 3

[0074] Step 310, obtaining a to-be-recognized voice.

[0075] The to-be-recognized voice can refer to a to-be-recognized original voice signal or voice data. In different scenarios, the to-be-recognized voice can be obtained in different ways. Optionally, in the scenario of real-time voice recognition, the to-be-recognized voice can be collected by the electronic device in real time through a voice collection device such as a microphone, and in the scenario of offline voice processing, the to-be-recognized voice can be pre-stored in the electronic device or imported into the electronic device.

[0076] In the case that the electronic device detects the to-be-recognized voice in real time, the electronic device can include a voice detection device and a voice processing device. The voice detection device can be used to collect sound signals outside the electronic device, such as a microphone, and the voice processing device can be used to perform analog-to-digital conversion on the collected sound signals to obtain the to-be-recognized voice, such as an analog-to-digital converter. It can be understood that the collected sound signal can be an analog signal, and the to-be-recognized voice can be a digital signal.

[0077] Step 320, recognizing the to-be-recognized voice through a hybrid voice recognition model to obtain a voice recognition result.

[0078] ​The mixed speech recognition model can comprise a target encoding module, a mixed expert model MoE module, and a target decoding module. The target encoding module is configured to encode the to-be-recognized speech to obtain an initial speech representation corresponding to the to-be-recognized speech. The MoE module comprises a routing network and a plurality of expert network groups corresponding to a plurality of language categories. The routing network is configured to determine a target expert network group corresponding to the initial speech representation according to the initial speech representation, and process the initial speech representation through the target expert network group to obtain a target speech representation. The target decoding module is configured to decode the target speech representation to obtain a speech recognition result.

[0079] Optionally, the target encoding module can refer to an encoding module specially used for speech recognition. The target encoding module can comprise a feature extraction network and an encoder, such as a HuBERT model. The feature extraction network can comprise a convolutional downsampling network, such as a pre-trained CNN (Convolutional Neural Networks). The encoder can comprise a plurality of encoding layers, such as a plurality of transformer layers.

[0080] The encoding process of the target encoding module on the to-be-recognized speech can comprise signal preprocessing, feature extraction, and feature mapping. The signal preprocessing can refer to noise reduction processing of the to-be-recognized speech to obtain a preprocessed signal, such as filtering out high-frequency signals or low-frequency signals to eliminate noise interference. The feature extraction can refer to extracting signal features of the preprocessed signal through the feature extraction network. The signal features can comprise spectral features or time-domain features. The feature mapping can refer to multi-level feature extraction and transformation of the signal features through the plurality of encoding layers to obtain high-dimensional features and map the high-dimensional features to a hidden state sequence in a low-dimensional vector space as a speech representation. It can be understood that noise can be added to the preprocessed signal to improve the robustness of speech recognition.

[0081] It can be understood that the speech representation is an abstract representation extracted by a deep neural network. Compared with signal features directly describing physical or statistical attributes of the to-be-recognized speech, the speech representation can represent more semantic information of the to-be-recognized speech and is more suitable for use in the field of speech recognition, which can improve the accuracy of speech recognition.

[0082] Optionally, the target decoding module can refer to a decoding module specially used for speech recognition. The target decoding module is configured to convert the speech representation into a recognition result, such as a CTC (Connectionist Temporal Classification) decoder, which can convert the speech representation into a text sequence.

[0083] Each expert network group in the MoE (Mixture of Experts) module corresponds to a language category, and each expert network group can be used to process speech representation of the language category. For example, the expert network group corresponding to Chinese can process speech representation of Chinese. Each expert network group can include one or more expert networks. The expert networks in the same expert network group process speech representation of the same language category, but the network functions of the expert networks in the same expert network group can be different. For example, in the expert network group corresponding to Chinese, one expert network can correspond to semantic analysis of Chinese speech representation, and another expert network can correspond to sentiment analysis of Chinese speech representation.

[0084] The routing network is responsible for determining a suitable target expert network group from the multiple expert network groups corresponding to the multiple language categories to process the initial speech representation, so that the processed speech representation can more prominently represent specific speech information such as speech text and speech sentiment. In the MoE module, the routing network can determine a target expert network group corresponding to the initial speech representation according to the initial speech representation, and activate the expert networks in the target expert network group, so that the activated expert networks process the initial speech representation. Optionally, the routing network can include a preset specific function such as a Softmax function, so as to calculate scores or weights of the initial speech representation in each target expert network group according to the specific function, and determine the target expert network group corresponding to the initial speech representation according to the scores or weights of the initial speech representation in each target expert network group.

[0085] As an example, assuming that there are four expert network groups in the MoE module, when the routing network receives the initial speech representation, the routing network can generate a weight vector corresponding to the four expert network groups through linear transformation and the Softmax function. The weight vector includes four weights corresponding to the four expert network groups, such as [0.1, 0.7, 0.15, 0.05]. Then, a target expert network group with the largest weight (0.7 corresponding to the expert network group) or two target expert network groups (0.7 and 0.15 corresponding to the target expert network groups) can be selected, and the initial speech representation is sent to the target expert network group.

[0086] In the embodiment of the present application, the electronic device can obtain the to-be-recognized speech, and recognize the speech through a mixed speech recognition model to obtain a speech recognition result. The mixed speech recognition model includes a target encoding module, an MoE module, and a target decoding module. The target encoding module can be used for encoding processing on the to-be-recognized speech to obtain an initial speech representation corresponding to the to-be-recognized speech. The MoE module includes a routing network and a plurality of expert network groups corresponding to a plurality of language categories. The routing network can be used for determining a target expert network group corresponding to the initial speech representation from the plurality of expert network groups according to the initial speech representation, and processing the initial speech representation through the target expert network group to obtain a target speech representation. The target decoding module is used for decoding processing according to the target speech representation to obtain the speech recognition result. Through the embodiment, a plurality of expert network groups can be obtained in the MoE according to language categories. The routing module in the MoE can route the initial speech representation corresponding to the to-be-recognized speech to the corresponding target expert network group. The language category contained in the to-be-recognized speech can correspond to the language category corresponding to the target expert network. Then, the initial speech representation is processed through the target expert network group, which can accurately capture the speech features of different language categories to obtain the target speech representation, reduce the confusion and misrecognition of language categories, improve the accuracy of the target speech representation, and thus improve the recognition accuracy of the mixed speech containing a plurality of language categories.

[0087] In order to further select a suitable expert network to process the initial speech representation, in one embodiment, each expert network group can include a routing unit and a plurality of expert networks. The routing unit is used for determining one or more target expert networks from the plurality of expert networks according to the initial speech representation, and processing the initial speech representation through the one or more target expert networks to obtain a target speech representation.

[0088] The routing unit in the target expert network group can determine one or more target expert networks corresponding to the initial speech representation from the plurality of expert networks included in the target expert network group according to the initial speech representation, and activate the one or more expert networks, so that the activated one or more expert networks process the initial speech representation. Optionally, the routing unit can also include a preset specific function. The specific function included in the routing unit can be the same as the specific function included in the routing network, such as a Softmax function, so as to calculate the score or weight of the initial speech representation in each expert network of the target expert network group according to the specific function, and then determine the one or more expert networks corresponding to the initial speech representation according to the score or weight of the initial speech representation in each expert network of the target expert network group.

[0089] By implementing the embodiment, the target expert network group can be selected again through the routing unit, and the target expert network determined through the routing network and the routing unit twice can further improve the accuracy of the determined target speech representation, so as to improve the accuracy of speech recognition.

[0090] As shown in Figure 4 , Figure 4 is a structural schematic diagram of a MoE module disclosed in the embodiment of the application, wherein the routing network can split the input initial speech representation H pre into a Chinese initial speech representation and an English initial speech representation The Chinese initial speech representation is routed to the Chinese expert network group, and the English initial speech representation is routed to the English expert network group.

[0091] The input initial speech representation H pre may include initial speech representations corresponding to multiple frames of sub-speech data respectively, the MoE module can determine the language category corresponding to each frame of sub-speech data, so as to split the input initial speech representation H pre into a Chinese initial speech representation and an English initial speech representation The multiple frames of sub-speech data can be obtained after the to-be-recognized speech is subjected to frame processing.

[0092] The routing unit in the Chinese expert network group determines one or more target expert networks of the Chinese expert network group from the n expert network groups of the Chinese expert network group according to the Chinese initial speech representation , so that the Chinese initial speech representation is processed according to the one or more target expert networks of the Chinese expert network group, and the output data of the Chinese expert network group, that is, the fusion speech representation corresponding to the Chinese initial speech representation

[0093] The routing unit in the English expert network group determines one or more target expert networks of the English expert network group from the n expert network groups of the English expert network group according to the English initial speech representation , so that the English initial speech representation is processed according to the one or more target expert networks of the English expert network group, and the output data of the English expert network group, that is, the fusion speech representation corresponding to the English initial speech representation

[0094] The MoE module fuses the output data of the Chinese expert network group and the output data of the English expert network group, that is, connects the output data of the Chinese expert network group and the output data of the English expert network group, to obtain the target speech representation H corresponding to the to-be-recognized speech output by the MoE module and . out .

[0095] In order to more clearly explain the specific process of speech recognition, as shown in Figure 5 , Fig. 1 is a flow diagram of a speech recognition method according to an embodiment of the present application. The speech recognition method can be applied to the electronic device in the above embodiment. The speech recognition method can include the following steps: Figure 5

[0096] Step 510: obtaining a to-be-recognized speech.

[0097] Step 520: performing frame processing on the to-be-recognized speech to obtain a plurality of frames of sub-speech data.

[0098] Optionally, the frame processing can be performed according to one of a fixed-length frame processing, an adaptive frame processing, and a windowed frame processing. The lengths of the plurality of frames of sub-speech data can be the same or different, and no limitation is made thereto.

[0099] In the fixed-length frame processing, the electronic device can divide the to-be-recognized speech into a plurality of frames according to a fixed time length. The adjacent two frames of sub-speech data can have overlapping speech data to ensure the continuity and information integrity between the frames. As an example, the to-be-recognized speech is 1 second, the fixed time length can be set to 25 milliseconds, and the time length of the overlapping speech data can be 10 milliseconds. The first frame of sub-speech data can include speech data from the 0th millisecond to the 25th millisecond, the first frame of sub-speech data can include speech data from the 10th millisecond to the 35th millisecond, and so on until the to-be-recognized speech data is divided. The fixed-length frame processing is simple and efficient in calculation and has wide applicability.

[0100] In the adaptive frame processing, the electronic device can dynamically adjust the time length of each frame of sub-speech data according to the acoustic characteristics of the to-be-recognized speech. For example, vowels and consonants can use different time lengths of frames, respectively, to improve the uniformity of the features contained in a frame of sub-speech data.

[0101] In the windowed frame processing, the electronic device can divide the to-be-recognized speech into a plurality of frames according to a fixed time length, and process each frame of speech using a specific windowing function, such as a Hamming window, to obtain a plurality of frames of sub-speech data, thereby reducing the signal edge effect caused by the frame processing and reducing the spectral leakage phenomenon.

[0102] ​At step 530, the target encoding module encodes the multiple frames of sub-phonetic data to obtain initial phonetic representations corresponding to the multiple frames of sub-phonetic data respectively.

[0103] As an example, the target encoding module can include a CNN and a transformer. The CNN can extract features of the multiple frames of sub-phonetic data respectively, and extract phonetic features of each frame of sub-phonetic data through multi-layer convolution operation. The phonetic features of each frame of sub-phonetic data are input into the transformer. The transformer can process the phonetic features of the multiple frames of sub-phonetic data respectively by using a self-attention mechanism, and generate initial phonetic representations corresponding to the multiple frames of sub-phonetic data respectively.

[0104] Optionally, the feature extraction network can further include one of a CNN, a long short-term memory network, and a ResNet feature extraction network. The encoder can further include one of a transformer and an autoencoder. The types of the feature extraction network and the encoder are not limited in the embodiments of the present application.

[0105] At step 540, the routing network determines a target expert network group corresponding to the target frame of sub-phonetic data according to the initial phonetic representation corresponding to the target frame of sub-phonetic data, from the multiple expert network groups. The target frame of sub-phonetic data is any frame of sub-phonetic data.

[0106] Optionally, the electronic device can input the initial phonetic representations corresponding to the frames of sub-phonetic data into the MoE module in a time sequence, so that the MoE module processes the initial phonetic representations corresponding to the frames of sub-phonetic data in sequence. Alternatively, the electronic device can input the initial phonetic representations corresponding to the multiple frames of sub-phonetic data into the MoE module together. The MoE module can process the initial phonetic representations corresponding to the multiple frames of sub-phonetic data in a time sequence or in a random manner. The embodiments of the present application do not limit this.

[0107] In one embodiment, the electronic device can determine a target language category corresponding to the target frame of sub-phonetic data according to the initial phonetic representation corresponding to the target frame of sub-phonetic data through the routing network, and determine the expert network group corresponding to the target language category as the target expert network group corresponding to the target frame of sub-phonetic data.

[0108] The routing network is a deep neural network, and can learn a language category classification capability for the initial phonetic representation in a training process. Thus, the target language category corresponding to the target frame of sub-phonetic data can be determined according to the initial phonetic representation corresponding to the target frame of sub-phonetic data in a speech recognition scenario. Alternatively, the routing network can calculate probabilities of the initial phonetic representation corresponding to the target frame of sub-phonetic data belonging to various language categories, and select a language category corresponding to the maximum probability as the target language category.

[0109] By implementing the embodiment, the electronic device can determine the target language category corresponding to the target sub-voice data, and then determine the target expert network group corresponding to the target language category as the target expert network group corresponding to the target frame sub-voice data, thereby improving the accuracy of the determined target expert network group.

[0110] At step 550, the routing unit in the target expert network group determines one or more target expert networks from the target expert network group according to the initial voice representation corresponding to the target frame sub-voice data.

[0111] Optionally, the target expert network group can be preconfigured with a preset number of expert networks. The electronic device can determine the preset number of target expert networks from the target expert network group according to the initial voice representation corresponding to the target frame sub-voice data through the routing unit in the target expert network group. Alternatively, the electronic device can determine one or more target expert networks from the target expert network group according to the initial voice representation corresponding to the target frame sub-voice data through the routing unit in the target expert network group, and the number of the one or more target expert networks is not greater than the preset number. By implementing the embodiment, the initial voice representation can be processed by a proper number of expert networks, thereby reducing the computational complexity and improving the processing efficiency of the initial voice representation.

[0112] At step 560, the initial voice representation corresponding to the target frame sub-voice data is processed by the one or more target expert networks to obtain a fused voice representation corresponding to the target frame sub-voice data.

[0113] The one or more target expert networks respectively correspond to a network weight, which can be determined by the routing unit of the target expert network group or preconfigured. Each target expert network can process the initial voice representation corresponding to the target frame sub-voice data to obtain a processing result corresponding to the target expert network, and then the processing results corresponding to the target expert networks are fused by weighting to obtain the fused voice representation corresponding to the target frame sub-voice data.

[0114] In one embodiment, step 550 can include determining one or more target expert networks and a network weight corresponding to each target expert network from the target expert network group according to the initial voice representation corresponding to the target frame sub-voice data through the routing unit in the target expert network group, and step 560 can include processing the initial voice representation corresponding to the target frame sub-voice data by the one or more target expert networks respectively to obtain a processing result corresponding to each target expert network, and fusing the processing results corresponding to the target expert networks by weighting based on the network weight corresponding to each target expert network to obtain the fused voice representation corresponding to the target frame sub-voice data.

[0115] The routing unit can also be a deep neural network, and can learn the representation processing capabilities of each target expert network on the speech representation during the training process, so that in a speech recognition scenario, one or more target expert networks and the network weights corresponding to each target expert network can be determined according to the initial speech representation corresponding to the target frame sub-speech data. The network weight can be used to represent the representation processing effect of the target expert network on the initial speech representation. The greater the network weight, the better the representation processing capability of the target expert network on the representation processing effect of the initial speech representation.

[0116] For example, in a Chinese expert network group, two target expert networks can be included. The first target expert network has stronger representation processing capability on the speech representation corresponding to the accent of people in Guangdong, and the second target expert network has stronger representation processing capability on the speech representation corresponding to the accent of people in Hong Kong. If the routing unit analyzes that the initial speech representation is closer to the speech representation corresponding to the accent of people in Guangdong, the network weight of the determined first target expert network is greater than the network weight of the second target expert network.

[0117] By implementing this embodiment, the electronic device can weight and fuse the processing results corresponding to each target expert network based on the network weights corresponding to each target expert network determined by the routing unit, to obtain the fused speech representation corresponding to the target frame sub-speech data, thereby improving the accuracy of the fused speech representation, improving the network comprehensive performance of the target expert network group through the dynamic weight method, and improving the accuracy of speech recognition.

[0118] Step 570: obtaining a target speech representation according to the fused speech representations corresponding to the multiple frame sub-speech data respectively.

[0119] After the MoE completes processing of the multiple frame sub-speech data, the electronic device can connect the fused speech representations corresponding to the multiple frame sub-speech data respectively to obtain the target speech representation.

[0120] Step 580: decoding the target speech representation through a target decoding module to obtain a speech recognition result.

[0121] In the embodiments of the present application, the electronic device can obtain the to-be-recognized speech, and perform frame processing on the to-be-recognized speech to obtain multiple frame sub-speech data, so as to process the initial speech representation corresponding to each frame sub-speech data, and finally obtain the target speech representation according to the fused speech representation corresponding to each frame sub-speech data. Compared with generating a speech representation of a long speech, the method of frame processing of speech can reduce the computational complexity, and for mixed speech containing multiple language types, frame processing can make each frame sub-speech data include only one language type, thereby improving the convenience and accuracy of speech recognition.

[0122] The above explains the speech recognition process, and the following explains the training process of the mixed speech recognition model, which can be obtained through one or more training.

[0123] In one embodiment, the target encoding module is obtained through self-supervised training based on a first data set and supervised training based on a second data set, and the MoE module and the target decoding module are obtained through supervised training based on the second data set. The first data set includes a plurality of unlabeled first sample speeches, each corresponding to a language category, and the second data set includes a plurality of second sample speeches and the corresponding labeled information of each second sample speech, each corresponding to one or more language categories.

[0124] In the self-supervised training process, the target encoding module can mine the plurality of unlabeled first sample speeches in the first data set through a preset auxiliary task to generate the corresponding discrete labels of each first sample speech, and then train the target encoding module according to the corresponding discrete labels of each first sample speech to minimize the prediction error and improve the encoding ability of the target encoding module until the self-supervised training is completed. The target encoding module can train the target encoding module to predict the masked part of the features of the first sample speech, such as a certain frame of speech in the first sample speech, and then convert the prediction result into a discrete label.

[0125] In the supervised training process, the plurality of second sample speeches can be input into the mixed speech recognition model to recognize each second sample speech through the mixed speech recognition model to obtain the corresponding sample recognition result of each second sample speech, and according to the corresponding sample recognition result of each first sample speech and the corresponding sample information, the corresponding target loss of each first sample speech can be obtained, so that the parameters of the target encoding module, the MoE module and the target decoding module can be adjusted according to the corresponding target loss of each first sample speech until the supervised training is completed. Optionally, the labeled information can be the text corresponding to the second sample speech.

[0126] It can be understood that due to the different training purposes, the self-supervised training is to improve the feature extraction ability of the target encoding module, and the supervised training is to adapt the mixed speech recognition model to a multi-language environment, so the first sample speech can correspond to only one language category, and the second sample speech can correspond to one or more language categories, for example, two first sample speeches can correspond to Chinese and English respectively, and a second sample speech can contain both Chinese and English. Moreover, the number of second sample speeches in the second data set can be much smaller than the number of first sample speeches in the first data set, reducing the cost of manual labeling.

[0127] The embodiment is implemented, the target coding module can be once self-supervised training and once supervised training, the extraction ability of the target coding module to the speech feature can be realized through the self-supervised training, and the adaptation of the speech recognition model to the speech recognition scene and the improvement of the speech recognition accuracy can be realized through the supervised training.

[0128] As Figure 6 shown, Figure 6 is a flowchart of a training process of a target coding module disclosed in an embodiment of the application, wherein:

[0129] In step 610, based on the first data set, the initial coding module is self-supervised trained to obtain the first coding module, and the first coding module includes multiple coding layers.

[0130] The coding layer can refer to a transformer layer, and the first coding module library includes multiple transformer layers.

[0131] In step 620, the second coding module is supervised trained based on the second data set to obtain the target coding module; the second coding module is obtained by adding a conditional decoding layer between each adjacent two coding layers in the first coding module.

[0132] The self-supervised training is to improve the extraction ability of the multiple coding layers to the speech feature, and the decoder can not be introduced at this stage to avoid over-fitting the distribution of a specific task or specific data, but the supervised training is to let the speech recognition model learn the accurate mapping relationship in the speech recognition task of the mixed speech through a small amount of first sample speech and the annotation information corresponding to the first sample speech, and at this time, introducing the decoder to optimize the framework of the coding module is helpful to improve the performance of the mixed speech recognition model.

[0133] The conditional decoding layer is used to calculate the posterior probability of the connected previous coding layer. In the supervised training process, the posterior probability calculated by the conditional decoding can be connected with the speech representation extracted by the previous coding layer, and the connected information is input into the next coding layer.

[0134] In the embodiment of the application, the electronic device can optimize the structure of the coding module after completing the self-supervised training, and by adding a conditional decoding layer between each adjacent two coding layers, the conditional independence of the target coding module can be relieved to improve the performance of the speech recognition model.

[0135] In one embodiment, the electronic device can normalize the first speech representation extracted by the current encoding layer to obtain a second speech representation, calculate the posterior probability corresponding to the second speech representation through the conditional decoding layer connected to the current encoding layer, linearly project the posterior probability through the linear projection layer to align the dimension of the posterior probability with the dimension of the first speech representation, and then perform residual connection between the mapped posterior probability and the first speech representation to obtain a third speech representation, and input the third speech representation into the next encoding layer of the current encoding layer.

[0136] The current encoding layer can refer to any one of the multiple encoding layers except the last encoding layer. By performing residual connection between the linearly projected posterior probability and the first speech representation, each encoding layer can obtain the original information of the previous encoding layer, which helps the model learn cross-layer features and enhance context dependence.

[0137] As shown in Figure 7 , Figure 7 is a schematic diagram of a target encoding module disclosed by an embodiment of the present application, wherein the encoding layer of the target encoding module is a transformer layer, the layer normalization algorithm is in Layer Norm, the conditional decoding layer is a conditional CTC decoder, the current transformer layer can output a first speech representation to Layer Norm, Layer Norm performs normalization processing on the first speech representation to obtain a second speech representation, the second speech representation is then input to the conditional CTC decoder to obtain a posterior probability, a linear projection layer maps the posterior probability to a latitude space corresponding to the second speech representation, and performs residual connection between the mapped posterior probability and the first speech representation to obtain a third speech representation, and the third speech representation is input to the next transformer layer.

[0138] By implementing this embodiment, the optimized model structure can improve the recognition ability of mixed speech containing multiple language categories and improve the accuracy of speech recognition.

[0139] As shown in Figure 8 , Figure 8 is a modular schematic diagram of a speech recognition device disclosed by an embodiment of the present application. The speech recognition device 800 can be applied to the electronic device in the above-mentioned embodiments. The speech recognition device 800 can include a speech acquisition module 810 and a speech recognition module 820, wherein:

[0140] The speech acquisition module 810 is configured to acquire a speech to be recognized.

[0141] The voice recognition module 820 is configured to recognize the to-be-recognized voice by using a mixed voice recognition model to obtain a voice recognition result. The mixed voice recognition model includes a target coding module, a mixed expert model (MoE) module, and a target decoding module. The target coding module is configured to perform coding processing on the to-be-recognized voice to obtain an initial voice feature corresponding to the to-be-recognized voice. The MoE module includes a routing network and a plurality of expert network groups corresponding to a plurality of language categories. The routing network is configured to determine a target expert network group corresponding to the initial voice feature from the plurality of expert network groups according to the initial voice feature, and perform processing on the initial voice feature by using the target expert network group to obtain a target voice feature. The target decoding module is configured to perform decoding processing on the target voice feature to obtain the voice recognition result.

[0142] In one embodiment, each expert network group includes a routing unit and a plurality of expert networks. The routing unit is configured to determine one or more target expert networks from the plurality of expert networks according to the initial voice feature, and perform processing on the initial voice feature by using the one or more target expert networks to obtain a target voice feature.

[0143] In one embodiment, the voice acquisition module 810 is further configured to perform frame processing on the to-be-recognized voice to obtain a plurality of frame sub-voice data. The voice recognition module 820 is further configured to perform coding on the plurality of frame sub-voice data by using the target coding module to obtain an initial voice feature corresponding to each frame sub-voice data. The routing network is configured to determine a target expert network group corresponding to the target frame sub-voice data from the plurality of expert network groups according to the initial voice feature corresponding to the target frame sub-voice data. The target frame sub-voice data is any frame sub-voice data. The routing unit in the target expert network group is configured to determine one or more target expert networks from the target expert network group according to the initial voice feature corresponding to the target frame sub-voice data. The one or more target expert networks are configured to perform processing on the initial voice feature corresponding to the target frame sub-voice data to obtain a fused voice feature corresponding to the target frame sub-voice data. The target voice feature is obtained according to the fused voice features corresponding to the plurality of frame sub-voice data. The target decoding module is configured to perform decoding on the target voice feature to obtain the voice recognition result.

[0144] In one embodiment, the voice recognition module 820 is further configured to determine a target language category corresponding to the target frame sub-voice data according to the initial voice feature corresponding to the target frame sub-voice data by using the routing network, and determine the expert network group corresponding to the target language category as the target expert network group corresponding to the target frame sub-voice data.

[0145] In an embodiment, the speech recognition module 820 is further configured to determine one or more target expert networks and a network weight corresponding to each target expert network from the target expert network group according to the initial speech representation corresponding to the target frame sub-speech data through a routing unit in the target expert network group; process the initial speech representation corresponding to the target frame sub-speech data through the one or more target expert networks respectively to obtain a processing result corresponding to each target expert network; and perform weighted fusion on the processing result corresponding to each target expert network based on the network weight corresponding to each target expert network to obtain a fused speech representation corresponding to the target frame sub-speech data.

[0146] In an embodiment, the target encoding module is obtained based on self-supervised training of a first data set and supervised training of a second data set, the MoE module and the target decoding module are obtained based on supervised training of the second data set; the first data set includes a plurality of unlabeled first sample speeches, each first sample speech corresponds to one language category, and the second data set includes a plurality of second sample speeches and labeled information corresponding to each second sample speech, each second sample speech corresponds to one or more language categories.

[0147] In an embodiment, the speech recognition apparatus can further include a model training module configured to perform self-supervised training on the initial encoding module based on a first data set to obtain a first encoding module, the first encoding module including a plurality of encoding layers; and perform supervised training on a second encoding module based on a second data set to obtain a target encoding module; the second encoding module being obtained by adding a conditional decoding layer between each two adjacent encoding layers in the first encoding module; and the conditional decoding layer being configured to calculate a posterior probability of a connected previous encoding layer.

[0148] In an embodiment, the model training module is further configured to perform normalization processing on the first speech representation extracted by the current encoding layer to obtain a second speech representation; calculate a posterior probability corresponding to the second speech representation through a conditional decoding layer connected to the current encoding layer; perform linear projection on the posterior probability through a linear projection layer, so that a dimension of the posterior probability is aligned with a dimension of the second speech representation; perform residual connection on the mapped posterior probability and the second speech representation to obtain a third speech representation, and input the third speech representation to a next encoding layer of the current encoding layer.

[0149] In the embodiments of the present application, the electronic device can obtain the to-be-recognized speech, and recognize the speech through a mixed speech recognition model to obtain a speech recognition result. The mixed speech recognition model includes a target encoding module, an MoE module, and a target decoding module. The target encoding module can be used to perform encoding processing on the to-be-recognized speech to obtain an initial speech representation corresponding to the to-be-recognized speech. The MoE module includes a routing network and a plurality of expert network groups corresponding to a plurality of language categories. The routing network can be used to determine a target expert network group corresponding to the initial speech representation from the plurality of expert network groups according to the initial speech representation, and process the initial speech representation through the target expert network group to obtain a target speech representation. The target decoding module is configured to perform decoding processing on the target speech representation to obtain the speech recognition result. Through the embodiments, a plurality of expert network groups can be obtained in the MoE according to language categories. The routing module in the MoE can route the initial speech representation corresponding to the to-be-recognized speech to the corresponding target expert network group. The language category contained in the to-be-recognized speech can correspond to the language category corresponding to the target expert network. The initial speech representation is processed through the target expert network group, which can accurately capture the speech features of different language categories to obtain the target speech representation, reduce the confusion and misrecognition of language categories, improve the accuracy of the target speech representation, and thus improve the recognition accuracy of the mixed speech containing a plurality of language categories.

[0150] As shown in Figure 9 In one embodiment, an electronic device is provided, which can include:

[0151] a memory 910 storing executable program codes;

[0152] a processor 920 coupled with the memory 910;

[0153] The processor 920 invokes the executable program codes stored in the memory 910 to implement the speech recognition method provided in the above embodiments.

[0154] The memory 910 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 910 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 910 can include a program storage area and a data storage area. The program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above various method embodiments, etc. The data storage area can also store data created by the electronic device during use, etc.

[0155] The processor 920 can include one or more processing cores. The processor 920 connects various parts within the entire electronic device by various interfaces and lines, performs various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 910, and calling data stored in the memory 910. Alternatively, the processor 920 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 920 can integrate a combination of one or several of a central processing unit (CPU), a graphics processor (GPU), and a modem. Among them, the CPU mainly processes an operating system, a user interface, and an application program; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 920, but can be implemented by a separate communication chip.

[0156] It can be understood that the electronic device can include more or less structural elements than the above structural block diagram, for example, including a power module, a physical key, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., and can also not be limited herein.

[0157] The embodiments of the present application disclose a computer readable storage medium storing a computer program, wherein the computer program causes a computer to execute the method described in the above embodiments.

[0158] In addition, the embodiments of the present application further disclose a computer program product, when the computer program product runs on a computer, causes the computer to execute all or part of the steps of any one of the voice recognition methods described in the above embodiments.

[0159] Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data in a computer readable manner.

[0160] The voice recognition method, device, electronic device and storage medium disclosed in the embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation on the present application.

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech to be recognized; The speech to be recognized is identified by a hybrid speech recognition model to obtain the speech recognition result; The hybrid speech recognition model includes a target encoding module, a hybrid expert model (MoE) module, and a target decoding module. The target encoding module encodes the speech to be recognized to obtain an initial speech representation. The MoE module includes a routing network and multiple expert network groups corresponding one-to-one with various language types. The routing network determines the target expert network group corresponding to the initial speech representation from the multiple expert network groups, and processes the initial speech representation through the target expert network group to obtain a target speech representation. The target decoding module decodes the target speech representation to obtain the speech recognition result. The target encoding module is obtained by self-supervised training based on a first data set and supervised training based on a second data set. The MoE module and the target decoding module are obtained by supervised training based on the second data set. The first data set includes multiple unlabeled first sample speech, each first sample speech corresponding to a language. The second data set includes multiple second sample speech and labeling information corresponding to each second sample speech, each second sample speech corresponding to one or more language types. Furthermore, the method further includes: performing self-supervised training on an initial encoding module based on a first data set to obtain a first encoding module, the first encoding module including multiple encoding layers; and performing supervised training on a second encoding module based on a second data set to obtain a target encoding module; the second encoding module is obtained by adding a conditional decoding layer between every two adjacent encoding layers in the first encoding module; the conditional decoding layer is used to calculate the posterior probability of the connected previous encoding layer.

2. The method according to claim 1, characterized in that, Each of the expert network groups includes a routing unit and multiple expert networks; the routing unit is used to determine one or more target expert networks from the multiple expert networks based on the initial speech representation, and to process the initial speech representation through the one or more target expert networks to obtain a target speech representation.

3. The method according to claim 2, characterized in that, After acquiring the speech to be recognized, the method further includes: The speech to be recognized is segmented into frames to obtain multi-frame sub-speech data; The process of recognizing the speech to be recognized using a hybrid speech recognition model to obtain the speech recognition result includes: The target encoding module encodes the multi-frame sub-speech data to obtain the initial speech representations corresponding to the multi-frame sub-speech data respectively. The routing network determines the target expert network group corresponding to the target frame sub-speech data from among the plurality of expert network groups based on the initial speech representation corresponding to the target frame sub-speech data; the target frame speech data can be any frame speech data. One or more target expert networks are determined from the target expert network group by the routing unit in the target expert network group based on the initial speech representation corresponding to the target frame sub-speech data; The initial speech representation corresponding to the sub-speech data of the target frame is processed by the one or more target expert networks to obtain the fused speech representation corresponding to the sub-speech data of the target frame; The target speech representation is obtained based on the fused speech representation corresponding to the multi-frame sub-speech data respectively; The target speech representation is decoded by the target decoding module to obtain the speech recognition result.

4. The method according to claim 3, characterized in that, The step of determining the target expert network group corresponding to the target frame sub-speech data from the plurality of expert network groups based on the initial speech representation corresponding to the target frame sub-speech data through the routing network includes: The routing network determines the target language type corresponding to the target frame sub-speech data based on the initial speech representation corresponding to the target frame sub-speech data, and determines the expert network group corresponding to the target language type as the target expert network group corresponding to the target frame sub-speech data.

5. The method according to claim 3, characterized in that, The step of determining one or more target expert networks from the target expert network group through routing units in the target expert network group based on the initial speech representation corresponding to the target frame sub-speech data includes: The routing unit in the target expert network group determines one or more target expert networks and the network weights corresponding to each target expert network from the target expert network group based on the initial speech representation corresponding to the target frame sub-speech data. The step of processing the initial speech representation corresponding to the target frame sub-speech data through the one or more target expert networks to obtain the fused speech representation corresponding to the target frame sub-speech data includes: The initial speech representations corresponding to the sub-speech data of the target frame are processed by one or more target expert networks respectively to obtain the processing results corresponding to each target expert network. Based on the network weights corresponding to each of the target expert networks, the processing results of each target expert network are weighted and fused to obtain the fused speech representation corresponding to the target frame sub-speech data.

6. The method according to claim 1, characterized in that, The method further includes: The first speech representation extracted from the current coding layer is normalized to obtain the second speech representation; The posterior probability corresponding to the second speech representation is calculated through the conditional decoding layer connected to the current coding layer. The posterior probability is linearly projected through a linear projection layer, so that the dimension of the posterior probability is aligned with the dimension of the second speech representation. The mapped posterior probability is residually concatenated with the second speech representation to obtain the third speech representation, and the third speech representation is input into the next coding layer of the current coding layer.

7. A voice recognition device, characterized in that, include: The speech acquisition module is used to acquire the speech to be recognized; A speech recognition module is used to recognize the speech to be recognized using a hybrid speech recognition model to obtain a speech recognition result. The hybrid speech recognition model includes a target encoding module, a hybrid expert model (MoE) module, and a target decoding module. The target encoding module is used to encode the speech to be recognized to obtain an initial speech representation corresponding to the speech to be recognized. The MoE module includes a routing network and multiple expert network groups that correspond one-to-one with multiple language types. The routing network is used to determine the target expert network group corresponding to the initial speech representation from the multiple expert network groups based on the initial speech representation, and to process the initial speech representation through the target expert network group to obtain a target speech representation. The target decoding module is used to decode the target speech representation to obtain the speech recognition result. The target encoding module is obtained by self-supervised training based on a first data set and supervised training based on a second data set. The MoE module and the target decoding module are obtained by supervised training based on the second data set. The first data set includes multiple unlabeled first sample speech, each first sample speech corresponding to a language. The second data set includes multiple second sample speech and labeling information corresponding to each second sample speech, each second sample speech corresponding to one or more language types. Furthermore, the speech recognition device further includes a model training module, which is used to perform self-supervised training on an initial encoding module based on a first data set to obtain a first encoding module, the first encoding module including multiple encoding layers; and to perform supervised training on a second encoding module based on a second data set to obtain a target encoding module; the second encoding module is obtained by adding a conditional decoding layer between every two adjacent encoding layers in the first encoding module; the conditional decoding layer is used to calculate the posterior probability of the connected upper encoding layer.

8. An electronic device, characterized in that, include: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when executed by a processor, the computer program causes the processor to perform the method according to any one of claims 1 to 6.

Images