Speech recognition, speech recognition model training method, device, medium and apparatus

Abstract

This application discloses a speech recognition method, apparatus, storage medium, and device. The method includes: adding a hybrid expert network and an embedding module to the encoding network module of a speech recognition model; using the embedding module to extract speech features from the target acoustic feature sequence of the target speech data to obtain target speech embedding features; inputting the target speech embedding features and the target acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing, so that the neural network module of the hybrid expert network can obtain language-related features and target acoustic feature sequences, thereby improving the accuracy of the output results of the neural network module of the hybrid expert network; and inputting the output results into the decoding network module for decoding processing to obtain the recognized text of the target speech data, which can improve the accuracy of speech recognition in various mixed language scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a speech recognition method, a speech recognition model training method, an apparatus, a computer-readable storage medium, and a computer device. Background Technology

[0002] Human-computer voice interaction / voice recognition has become very common in modern life. Typical human-computer voice interaction scenarios, such as automobiles, smart healthcare, smart homes, smart customer service, robots, and even virtual digital humans, all involve the application of voice recognition.

[0003] The world is home to a variety of languages, and even within my country, due to its vast territory and significant regional differences, people in different areas speak a wide variety of dialects. The written forms of dialects differ from Standard Mandarin; many regional dialects have their own standard characters and cannot be simply expressed using Standard Mandarin. Currently, most voice interaction systems only support Standard Mandarin conversations, while many people are accustomed to using dialects in their daily lives, which may lead to situations where Standard Mandarin and dialects are mixed during interactions.

[0004] In human-computer voice interaction, there are scenarios where various languages ​​are mixed, and current speech recognition methods cannot recognize them well, resulting in low accuracy. Summary of the Invention

[0005] This application provides a speech recognition model training method, a speech recognition method, an apparatus, a computer-readable storage medium, and a computer device, which can improve the accuracy of speech recognition in various mixed language scenarios.

[0006] This application provides a speech recognition method, including:

[0007] Acquire target speech data and invoke a speech recognition model, wherein the speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network, and the encoding network module includes a neural network module and an embedding module of the hybrid expert network;

[0008] The target speech data is subjected to acoustic feature extraction to obtain the target acoustic feature sequence of the target speech data;

[0009] The target acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain the target speech embedding features of the target speech data;

[0010] The target speech embedding features and the target acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain the target semantic encoding features of the target speech data;

[0011] The target semantic encoding features and the preset text start character are input into the decoding network module for decoding processing to obtain the recognized text of the target speech data.

[0012] This application also provides a speech recognition model training method, including:

[0013] A speech dataset and an initial speech recognition model in multiple languages ​​are obtained. The speech dataset includes speech sample data in each language, language text labels for the corresponding language of the speech sample data, and target text labels for the target language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network.

[0014] The speech sample data is subjected to acoustic feature extraction to obtain the acoustic feature sequence of the speech sample data;

[0015] The acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data, and the network parameters of the embedding module are updated according to the speech embedding features and the language text label;

[0016] The speech embedding features and the acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain the semantic encoding features of the speech sample data;

[0017] The semantic encoding features and the target text labels are input into the decoding network module for decoding processing to obtain the predicted text of the speech sample data;

[0018] The network parameters of the initial speech recognition model are updated based on the predicted text and the target text label to obtain the speech recognition model.

[0019] This application embodiment also provides a speech recognition model training device, including:

[0020] The first acquisition unit is used to acquire speech datasets and initial speech recognition models in multiple different languages. The speech datasets include speech sample data for each different language, language text labels for the corresponding language of the speech sample data, and target text labels for the target language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network.

[0021] The first acoustic extraction unit is used to extract acoustic features from the speech sample data to obtain the acoustic feature sequence of the speech sample data.

[0022] The first embedding extraction unit is used to input the acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data;

[0023] The first expert coding unit is used to input the speech embedding features and the acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing to obtain the semantic coding features of the speech sample data;

[0024] The first decoding unit is used to input the semantic encoding features and the target text label into the decoding network module for decoding processing, so as to obtain the predicted text of the speech sample data;

[0025] The second update unit is used to update the network parameters of the initial speech recognition model according to the predicted text and the target text label, so as to obtain the speech recognition model.

[0026] This application also provides a voice recognition device, including:

[0027] The second acquisition unit is used to acquire target speech data and call a speech recognition model, wherein the speech recognition model includes an embedding module, a neural network module of hybrid expert network and a decoding network module;

[0028] The second acoustic extraction unit is used to extract acoustic features from the target speech data to obtain the target acoustic feature sequence of the target speech data.

[0029] The second embedding extraction unit is used to input the target acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the target speech embedding features of the target speech data;

[0030] The second expert coding unit is used to input the target speech embedding features and the target acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing, so as to obtain the target semantic coding features of the target speech data;

[0031] The second decoding unit is used to input the target semantic encoding features and the preset text start character into the decoding network module for decoding processing, so as to obtain the recognition text of the target speech data.

[0032] This application also provides a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the steps described in any of the above embodiments.

[0033] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the steps in the method described in any of the above embodiments by calling the computer program stored in the memory.

[0034] The speech recognition model training method, speech recognition method, apparatus, computer-readable storage medium, and computer device provided in this application embodiment, by adding a hybrid expert network and an embedding module to the encoding network module of the speech recognition model, utilizes the embedding module to perform speech feature extraction processing on the target acoustic feature sequence of the target speech data to obtain target speech embedding features. These target speech embedding features include shallow language-related features of the target speech data. After obtaining the target speech embedding features, the target speech embedding features and the target acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing, so that the neural network module of the hybrid expert network can obtain... Language-related target speech embedding features and target acoustic feature sequences improve the accuracy of the output results of the neural network module of the hybrid expert network. Moreover, due to the presence of the hybrid expert network, different expert neural networks in the hybrid expert network can be trained using speech datasets of different languages. Theoretically, the hybrid expert network can be arbitrarily extended to add new languages ​​without increasing the overall computational load, demonstrating extremely strong scalability. Furthermore, the presence of different expert neural networks enables the speech recognition model to recognize target speech data in different languages ​​or target speech data in multiple languages, thereby improving the accuracy of speech recognition. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1a This is a schematic diagram of an application scenario provided for an embodiment of this application.

[0037] Figure 1b This is a schematic diagram of another application scenario provided for an embodiment of this application.

[0038] Figure 2This is a flowchart illustrating the speech recognition model training method provided in an embodiment of this application.

[0039] Figure 3 This is a schematic diagram of the coding network module provided in an embodiment of this application.

[0040] Figure 4 This is a schematic diagram of a sub-process of the speech recognition model training method provided in an embodiment of this application.

[0041] Figure 5 This is a schematic diagram of the decoding network module provided in an embodiment of this application.

[0042] Figure 6 This is a schematic diagram of a sub-process of the speech recognition model training method provided in the embodiments of this application.

[0043] Figure 7 This is a schematic diagram of another sub-process of the speech recognition model training method provided in the embodiments of this application.

[0044] Figure 8 This is a schematic diagram of a language decoder for a hybrid expert network provided in an embodiment of this application.

[0045] Figure 9 This is a flowchart illustrating the speech recognition method provided in an embodiment of this application.

[0046] Figure 10 This is a schematic diagram of the structure of the speech recognition model training device provided in the embodiments of this application.

[0047] Figure 11 This is a schematic diagram of the structure of the speech recognition device provided in the embodiments of this application.

[0048] Figure 12 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] This application provides a speech recognition model training method, a speech recognition method, an apparatus, a computer-readable storage medium, and a computer device. Specifically, the speech recognition model training method and / or speech recognition method of this application can be executed by a computer device, and the speech recognition model training apparatus and / or speech recognition apparatus of this application are integrated into the computer device, wherein they can be integrated into one or more computer devices. For example, if the speech recognition model training method is executed in one computer device and the speech recognition method is executed in another computer device, correspondingly, the speech recognition model training apparatus is integrated into one computer device, and the speech recognition apparatus is integrated into another computer device.

[0051] The computer equipment can be a terminal device or a server. The terminal device can be a smartphone, tablet, laptop, touchscreen, game console, personal computer (PC), smart vehicle terminal, robot, etc. The server can be a standalone physical server, a service node in a blockchain system, a server cluster consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, and big data and artificial intelligence platforms.

[0052] Before providing a detailed description of the solutions in the embodiments of this application, we will further analyze the prior art.

[0053] Current speech recognition systems are mainly designed as cascaded models, which can be viewed as a combination of two subsystems: speech recognition and machine translation. The speech recognition subsystem is responsible for converting dialect audio signals into dialect text. However, dialect text is difficult to read and understand, which also places a greater interactive burden on the machine: to understand the text, an additional complex machine translation subsystem is needed to translate the dialect text into standard Mandarin, thus significantly reducing speech recognition efficiency.

[0054] Meanwhile, some end-to-end speech recognition systems, unlike the combined speech recognition and machine translation subsystems, can accept speech signals from a specific dialect and directly convert them into easily understandable Mandarin text. However, these end-to-end speech recognition systems have a significant drawback: users need to manually select the language they use for interactive dialogue.

[0055] In in-vehicle interaction scenarios, users must first select a fixed language such as Mandarin, Cantonese, or Northeastern Mandarin in the vehicle's settings before using voice control. When Mandarin is selected, the user's spoken Mandarin is highly likely to be correctly recognized, allowing successful control of the vehicle. However, if the user's Mandarin is unintentionally mixed with dialect, this part is almost impossible to recognize. Conversely, once a dialect is selected, the recognition success rate of Mandarin drops significantly. This severely limits daily interaction, restricting it to a single language. Users must switch between languages ​​when speaking different languages, greatly impacting the convenience and comfort of the interaction.

[0056] Therefore, current speech recognition methods have low accuracy in scenarios where various languages ​​are spoken together.

[0057] To address this problem, this application proposes a speech recognition model training method, a speech recognition method, a device, a computer-readable storage medium, and a computer equipment. The speech recognition model in this application is switch-free, eliminating the need for language selection. Furthermore, it is no longer limited to a single language during interaction, allowing users to freely speak any supported language, and even mix multiple languages, achieving barrier-free interaction across any language.

[0058] Figure 1a This is a schematic diagram illustrating an application scenario of the speech recognition method provided in this application embodiment. In this scenario, the speech recognition model is integrated into the terminal. The user speaks, the terminal receives the speech, and then calls the speech recognition model to recognize the speech. The recognized Mandarin text can be output simultaneously with the speech recognition process. Alternatively, after receiving a complete speech from the user, the terminal can call the speech-to-text function to invoke the speech recognition model to recognize the complete speech and obtain the recognized Mandarin text. The recognized Mandarin text can be displayed on the terminal's graphical user interface.

[0059] In other embodiments, the speech recognition model can be integrated into the server. After receiving the speech, the terminal sends the received speech to the server. The server then calls the speech recognition model to perform recognition based on the received speech, returns the recognized Mandarin text to the terminal, and displays the returned Mandarin text on the terminal. There are many other application scenarios, which will not be listed here.

[0060] Figure 1b This is a simplified schematic diagram illustrating the application scenario of the speech recognition method provided in this application embodiment. As can be seen, when using the speech recognition model provided in this application embodiment, the input speech can be multilingual, such as Mandarin, Minnan, Cantonese, Sichuanese, Northeastern Mandarin, Shanghainese, etc. After being processed by the speech recognition model, all of them are output as text in the target language, such as Mandarin text.

[0061] Figure 2 This is a flowchart illustrating the speech recognition model training method provided in this application embodiment. The method can be applied in computer devices and includes the following steps.

[0062] 101. Obtain speech datasets and initial speech recognition models for multiple different languages. The speech datasets include speech sample data for each different language, language text labels for the corresponding language of the speech sample data, and target text labels for the target language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network.

[0063] The "multiple languages" can include various minority languages ​​such as Tibetan and Uyghur, and / or various dialects such as dialect A and dialect B, and / or various other non-native languages ​​such as Spanish and Hindi. The "multiple languages" also include the target language, such as Mandarin. The target language is the language used in the majority of cases in the current application scenario or the official language in the current application scenario.

[0064] It should be noted that the target language in the various languages ​​in this application embodiment varies depending on the specific application scenario of the speech recognition model. Assuming that the speech recognition model in this application embodiment is applied to a country where English is the main language, then the target language is English, and the other languages ​​are languages ​​other than English.

[0065] This application uses various languages, including Mandarin and different dialects, as examples for illustration.

[0066] The speech dataset contains speech samples in multiple languages, including Mandarin Chinese and various dialects such as Cantonese, Northeastern Mandarin, and Sichuanese. It also includes language-specific text labels for each speech sample, such as dialect labels (Dialect Label), and target language labels, such as Mandarin Chinese labels (Target Label). The corresponding language refers to the language to which the speech sample data belongs. The language-specific text label maps the speech sample data to / translates it into written text in that language (e.g., dialect label). The target language label maps the speech sample data to written text in the target language (e.g., Mandarin Chinese label).

[0067] For a dialect, the language text / written text of the corresponding language of the speech sample data can also be referred to as the orthographic text of the dialect corresponding to the speech sample data. The orthographic text of a dialect is the text form that directly translates the speech of the dialect into the written words of the language to which the dialect belongs. For example, when expressing the meaning of "What did you say?" in Cantonese, the sentence "你讲乜嘢" is often used. The former is the Mandarin text, and the latter is the orthographic text of the dialect. In addition, in other languages such as English and Japanese, the orthographic text is the corresponding written text of that language.

[0068] In a speech dataset, there can be multiple speech sample data for each different speech. Each piece of data in the speech dataset includes speech sample data, a language text label, and a target text label. For example, one piece of data includes: Cantonese speech data (such as the speech data of "What did you say?" in Cantonese), "你讲乜嘢", and "What did you say?". For the language of Mandarin, the language text label and the target text label are the same.

[0069] Among them, speech datasets of multiple different languages can be determined according to the corresponding application scenarios. For example, for the intelligent customer service scenario, the speech conversations between users and customers in the corresponding language can be collected or recorded, and their corresponding language text labels and target text labels are manually marked, such as the corresponding orthographic text and Mandarin text. For the smart home scenario, the interactive speech data for controlling smart home appliances in the corresponding language can be obtained and marked.

[0070] In some embodiments, the speech sample data of a multi-language mixed speaking scenario can also be directly obtained. Using the speech sample data of a multi-language mixed speaking scenario for training can reduce the number of speech datasets and have less requirements for training data.

[0071] After obtaining the speech sample data, language text labels, and target text labels of each different language, these data can also be preprocessed. The preprocessing mainly includes data cleaning and data augmentation, etc. Data cleaning is mainly for text, in order to remove unreasonable bad data in the labeled data such as language text labels and target text labels, and to unify the expressions of sentences such as numbers, times, and dates, so as to improve the quality of the labeled data and enhance the accuracy of the speech dataset. Data augmentation is mainly for speech sample data, and the limited speech sample data is transformed, including but not limited to changing the speech sample data in ways such as adding noise, reducing noise, changing speed, and reverberation. This is equivalent to expanding the speech sample data, which can effectively improve the robustness in different scenarios. For small language dialects with insufficient speech sample data, the requirements for the amount of labeled data are also relatively reduced. The dataset after data cleaning and data augmentation is used as the speech dataset.

[0072] The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network.

[0073] The initial speech recognition model refers to the speech recognition model that needs to have its network parameters updated. The initial speech recognition model can be obtained in the following way.

[0074] For example, an initial speech recognition model can be obtained by improving the RNN-T (Recurrent Neural Network Transducer) model structure. The RNN-T model structure includes an encoder, a decoder, and a joint network. The encoder can be any network structure capable of encoding, and the decoder can be any network structure capable of decoding. By adding a hybrid expert network and an embedding module to the RNN-T encoder, a neural network module (MoE Encoder) including a hybrid expert network and an embedding module can be obtained. Further, based on the decoding network module composed of the RNN-T decoder and the joint network, another dialect decoding network module can be added, called a language decoding network module. A hybrid expert network is then added to the language decoder within the language decoding network module, thus obtaining the initial speech recognition model.

[0075] The embedding module can be a simple linear connection layer, or it can be several convolutional layers or other common forms. The Mixture of Experts (MoE) network consists of multiple parallel expert neural networks such as FFN1, FFN2, ..., FFNn-1, FFNn and a router. Each FFN is an expert neural network, and one expert neural network corresponds to one expert. The router can be regarded as a classifier.

[0076] For example, in one scenario, the encoder of an RNN-T sequentially includes a pre-shared convolutional layer, multiple long short-term memory (LSTM) network modules, and a post-shared convolutional layer, following the data flow direction. Then, after adding the embedding module to the pre-shared convolutional layer, and the hybrid expert network after the multiple LSTM network modules and before the post-shared convolutional layer, the outputs of the embedding module and the multiple LSTM network modules are used as inputs to the router in the hybrid expert network to obtain the neural network module and the embedding module of the hybrid expert network. All other modules besides the embedding module are referred to as the neural network modules of the hybrid expert network. Figure 3The diagram shown is a schematic of the coding network module of the hybrid expert network provided in an embodiment of this application.

[0077] In some embodiments, the encoder may not include forward convolution modules and / or backward convolution modules, and the multiple long short-term memory neural networks in the encoder may be replaced with other network modules that can extract temporal acoustic features.

[0078] 102. Acoustic features are extracted from the speech sample data to obtain the acoustic feature sequence of the speech sample data.

[0079] Acoustic features are extracted from each speech sample data, for example, by performing Melp feature extraction to obtain an acoustic feature sequence of the speech sample data, such as a Melp feature sequence. Acoustic feature extraction can also use other methods, and the resulting acoustic feature sequence can also be an F-bank (Filter Bank) feature sequence, a PLP feature sequence, etc.

[0080] This step is used to extract useful acoustic features from the speech sample data of the speech dataset to obtain an acoustic feature sequence, which is used as input to the initial speech recognition model.

[0081] In general, acoustic feature extraction can also be included in the preprocessing part, that is, the acoustic feature sequence of each speech sample data is preprocessed to improve the training speed.

[0082] 103. The acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data.

[0083] Since the embedding module is a linear connection layer, it typically maps high-dimensional acoustic features to low-dimensional acoustic features. Therefore, the embedding module can extract shallow information from speech sample data, such as information about language and / or dialect.

[0084] The acoustic feature sequence is input into the embedding module for speech feature extraction processing, which involves mapping the high-dimensional acoustic feature sequence to obtain the speech embedding features of the speech sample data. The speech embedding features can be low-dimensional acoustic feature sequences, which include shallow information of the speech sample data.

[0085] In one embodiment, after obtaining the speech embedding features of the speech sample data, the network parameters of the embedding module are updated based on the speech embedding features and language text labels. Therefore, the embedding module is independently trained, and the network parameters of an independently trained embedding module are more accurate, enabling it to extract shallow, language-related features from the speech sample data more precisely.

[0086] Specifically, the speech loss value of the embedding module can be determined based on the speech embedding features and language text labels, and the network parameters of the embedding module can be updated based on the speech loss value. For example, the speech embedding features can be pre-decoded according to a preset decoding method to obtain the language text of the corresponding language of the speech sample data, and then the speech loss value of the embedding module can be determined based on the language text and language text labels.

[0087] The preset decoding method can be Connectionist Temporal Classification (CTC) decoding, and the preset decoding process can be CTC decoding. CTC decoding is used to obtain the language text of the speech sample data. Then, based on the CTC loss function, the language text and the language text label of the speech sample data are used to determine the CTC loss value, which is then used as the speech loss value for the embedding module. After obtaining the speech loss value, the network parameters of the embedding module are updated based on the speech loss value.

[0088] In other embodiments, the preset decoding method and preset decoding process can also be other decoding methods and decoding processes.

[0089] 104. The speech embedding features and acoustic feature sequences are input into the neural network module of the hybrid expert network for encoding processing to obtain the semantic encoding features of the speech sample data.

[0090] The input to the neural network module of the hybrid expert network includes speech embedding features and acoustic feature sequences. Thus, it includes both shallow features such as language and / or language type, as well as acoustic features, thereby improving the accuracy of the output results of the neural network module of the hybrid expert network.

[0091] The neural network module of the hybrid expert network includes a first preset neural network and a hybrid expert network. The first preset neural network is used to extract acoustic features (also known as temporal acoustic features) that include the temporal content of speech sample data. For example, the first preset neural network can be a Long Short-Term Memory (LSTM) network model, a model obtained by splicing multiple LSTM networks such as LSTMs, or other models that can extract acoustic features that include the temporal content of speech sample data.

[0092] Correspondingly, step 104 includes: inputting the acoustic feature sequence into a first preset neural network for acoustic feature extraction processing to obtain temporal acoustic features including the temporal content of the speech sample data; performing a first fusion processing on the speech embedding features and the temporal acoustic features to obtain a first fused feature; and inputting the first fused feature and the temporal acoustic features into a hybrid expert network for expert coding processing to obtain the semantic coding features of the speech sample data.

[0093] The first preset neural network can be an LSTM module, wherein the obtained temporal acoustic features include temporal information of the acoustic feature sequence, which includes information of all frames from the first frame to the current frame.

[0094] The temporal acoustic features and speech embedding features are subjected to a first fusion process, which may include splicing or other methods, to obtain a first fused feature. Thus, the first fused feature includes both shallow linguistic information from the speech sample data and deep acoustic information with temporal content from the speech sample data, thereby improving the accuracy of subsequent coding processes.

[0095] Specifically, the first fused feature and the temporal acoustic feature can be input into a hybrid expert network for expert coding to obtain the semantic coding features of the speech sample data. The first fused feature can be input into a router for classification to obtain expert probabilities corresponding one-to-one with multiple expert neural networks (multiple FFNs); the temporal acoustic feature can be input into the target expert neural network with the highest expert probability for expert coding to obtain the expert semantic coding features of the speech sample data; the semantic coding features of the speech sample data are determined based on the expert semantic coding features and the highest expert probability.

[0096] The number of classification categories in the router is consistent with the number of expert neural networks in the hybrid expert network. There is a one-to-one correspondence between the classification categories in the router and the expert neural networks. For example, if there are 10 expert neural networks, then there are also 10 classification categories, and they correspond one-to-one.

[0097] The router's output is used to select which expert neural network to use. The first fusion feature is input into the router for classification processing to obtain expert probabilities corresponding to multiple expert neural networks. Since the first fusion feature integrates shallow speech / language information and deep acoustic information including temporal content, the calculation of expert probabilities in the router is more accurate, and the selected target expert neural network is also more accurate, improving both training efficiency and accuracy.

[0098] In this context, multiple expert probabilities represent the likelihood of assigning the temporal acoustic features corresponding to the current speech sample data to each expert neural network (each FFN). The expert neural network with the highest probability is ultimately selected; that is, the target expert probability with the highest expert probability is chosen from multiple expert probabilities, and the expert neural network (FFN) corresponding to the target expert probability is used as the target expert neural network. Figure 3 The highest probability in the range is 0.7, so the first expert neural network corresponding to 0.7 is taken as the target expert neural network.

[0099] The temporal acoustic features output by the first preset neural network are input into the target expert neural network for expert encoding processing to obtain expert semantic encoding features. The expert semantic encoding features are then multiplied by the maximum target expert probability to obtain the semantic encoding features of the speech sample data.

[0100] Although there are multiple expert neural networks in a hybrid expert network, only one expert neural network is trained at a time, allowing that expert neural network to learn the semantic information in the temporal acoustic features. In this way, speech sample data from different languages ​​can be used to train different expert neural networks, improving the specialization of each expert neural network.

[0101] The number of expert neural networks in the hybrid expert network can be freely configured, but must be set to be no less than the number of supported languages. When a new language is added, a new expert neural network is added to the hybrid expert network, a new category is added to the router, and the speech dataset for the new language is obtained, mixed with the original speech dataset, and then the model is trained. Thus, the speech recognition model in this embodiment has extremely high scalability.

[0102] Assuming the coding network module of the hybrid expert network in this application embodiment is as follows: Figure 3 As shown, the entire encoding network module in this application can be implemented according to the following steps. Figure 4 As shown, the specific steps include the following.

[0103] 201. The acoustic feature sequence is input into the forward convolution module for the first convolution process to obtain the first convolution feature.

[0104] The forward convolution module includes multiple convolutional layers. These layers are used for the first convolutional processing to extract the first convolutional features of the acoustic feature sequence. This first convolutional processing using the forward convolution module further extracts shallower features from the acoustic feature sequence, improving the accuracy of the speech embedding features used later.

[0105] 202. The first convolutional feature is input into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data.

[0106] Since the first convolutional feature extraction already extracts relatively shallow features from the acoustic feature sequence, these shallow features are further input into the embedding module for speech feature extraction processing. For example, the high-dimensional first convolutional features are mapped to obtain low-dimensional speech embedding features. If the first convolutional feature has a large number of channels, such as 512 channels, after processing by the embedding module, it is mapped to 256 or 128 channels, etc. The resulting speech embedding features include shallower features to represent speech / language information, making the representation of speech / language information more accurate.

[0107] In one embodiment, after obtaining the speech embedding features, the network parameters of the embedding module are updated based on the speech embedding features and the language text label. Since a forward convolutional module is included before the embedding module, the network parameters of both the embedding module and the forward convolutional module can be updated.

[0108] 203. The first convolutional feature is input into the first preset neural network for acoustic feature extraction processing to obtain temporal acoustic features including the temporal content of the speech sample data.

[0109] After obtaining the first convolutional feature, the first convolutional feature is simultaneously input into the embedding module and the first preset neural network module for corresponding processing. Accordingly, steps 202 and 203 can be executed in parallel.

[0110] 204. Perform a first fusion process on the temporal acoustic features and speech embedding features to obtain the first fused features.

[0111] 205. The first fused feature is input into the router of the hybrid expert network for classification processing to obtain the expert probabilities corresponding one-to-one with the multiple expert neural networks.

[0112] 206. The temporal acoustic features are input into the target expert neural network with the highest expert probability for expert encoding processing to obtain the expert semantic encoding features of the speech sample data.

[0113] 207. The semantic coding features of the speech sample data are determined based on the expert semantic coding features and the highest expert probability.

[0114] 208. The semantic encoding features are input into the backward convolution module for a second convolution process to obtain the final semantic encoding features of the speech sample data.

[0115] In this process, the semantic coding features are subjected to a second convolution to obtain the final semantic coding features of the speech sample data.

[0116] Figure 4 For steps not described in detail, please refer to the corresponding sections above; they will not be repeated here. Figure 4 It describes in detail Figure 3 The entire process of obtaining semantic coding features from speech sample data by the coding network module of the hybrid expert network is the entire process of coding by the coding network module. Among them, the forward convolution module, the first preset neural network such as LSTMs, the backward convolution module, and the embedding module are all modules that share parameters.

[0117] The entire processing procedure of the decoding network module will be described in detail below.

[0118] 105. The semantic encoding features and target text labels are input into the decoding network module for decoding to obtain the predicted text of the speech sample data. The network parameters of the initial speech recognition model are updated according to the predicted text and target text labels to obtain the speech recognition model.

[0119] In one scenario, the decoding network module in this application embodiment only includes the target decoding network module, such as a Mandarin decoding network module. Figure 5 As shown, the target decoding network module includes a target decoder (e.g., a Mandarin decoder) and a target joint network module (e.g., a Mandarin joint network module). The network structures in the target decoder and the target joint network module are consistent with the network structures of the decoder and joint network module in current RNN-T models.

[0120] Correspondingly, step 105 includes: inputting the target text label, such as the Mandarin text label, into the target decoder for text feature extraction processing to obtain the text feature extraction result of the target text label; performing a third fusion processing on the text feature extraction result and the semantic encoding feature of the encoding network module in the above application to obtain a third fusion feature, wherein the third fusion processing may be splicing processing, etc.; inputting the third fusion feature into the target joint network module for target decoding processing, such as Mandarin decoding processing, to obtain the target predicted text of the target language of the speech sample data, such as the Mandarin predicted text; and updating the network parameters of the initial speech recognition model according to the target predicted text and the target text label to obtain the speech recognition model.

[0121] Because the accuracy of semantic encoding features is improved, decoding these more accurate features results in more accurate target prediction text, further improving training efficiency. Correspondingly, after obtaining the target prediction text, the loss value of the initial speech recognition model is determined based on the target prediction text and the target text label. The network parameters of the initial speech recognition model are then updated based on the loss value until training stopping conditions are met, such as loss convergence or the number of training epochs reaching a preset number. Training then stops, and the speech recognition model is obtained.

[0122] In one case, such as Figure 5 As shown, the decoding network module in the training process includes two sets of parallel decoders and a joint network module, referred to as the language decoding network module (e.g., dialect decoding network module) and the target decoding network module (e.g., Mandarin decoding network module). The language decoding network module outputs the orthographic text result in the corresponding language. For example, after decoding the Cantonese speech sample data, the language decoding network module outputs the Cantonese orthographic text result of the speech sample data. This embodiment uses two parallel decoding network modules for training to improve the accuracy of the final target predicted text.

[0123] Correspondingly, such as Figure 6 As shown, step 105 includes the following steps.

[0124] 301. Input the semantic encoding features and language text labels into the language decoding network module for the first decoding process to obtain the language prediction text of the corresponding language of the speech sample data.

[0125] The semantic encoding features and language text labels, such as dialect text labels (Dialect Label) (text labels of orthographic text of dialects), are input into the language decoding network module for the first decoding process to obtain language prediction text, such as dialect prediction text (Dialect Text), such as the predicted dialect text, and the predicted orthographic text of dialects.

[0126] Among them, such as Figure 5 As shown, the language decoding network module includes a hybrid expert network language decoder (DialectMoE Decoder) such as a hybrid expert network dialect decoder and a language joint network module (Dialect Jointer).

[0127] Correspondingly, the step of inputting semantic encoding features and language text labels into a language decoding network module for first decoding processing to obtain the language prediction text of the corresponding language of the speech sample data includes: inputting language text labels into a hybrid expert network language decoder for language text decoding processing to obtain language text prediction features of language text labels, such as dialect text prediction features of dialect text labels; performing a second fusion processing on the semantic encoding features and language text prediction features to obtain a second fused feature; and inputting the second fused feature into a language joint network module for language decoding processing to obtain the language prediction text of the corresponding language of the speech sample data, such as dialect prediction text. In this embodiment, the language decoder of a hybrid expert network is used for language text decoding processing to obtain more accurate language text prediction features and improve the accuracy of language prediction text. In addition, the second fusion processing on the more accurate semantic encoding features and the more accurate language text prediction features makes the obtained second fused feature more accurate, further improving the accuracy of language text prediction features.

[0128] The language decoder of the hybrid expert network includes a second preset neural network and a hybrid expert network. The second preset neural network can be an existing network in the decoder capable of decoding text features of temporal content, such as an LSTM network or LSTMs network, or other network structures. This hybrid expert network is consistent with the hybrid expert network in the encoding network module, and new expert neural networks can be arbitrarily extended to the hybrid expert network in the language decoder.

[0129] In one embodiment, the step of inputting language text labels into a language decoder of a hybrid expert network for language text decoding processing to obtain language text prediction features of the corresponding language of the speech sample data includes: inputting language text labels into a second preset neural network for text feature extraction processing to obtain text feature extraction results; and inputting the text feature extraction results into a hybrid expert network for expert decoding processing to obtain language text prediction features of the language text labels, such as dialect text prediction features. In this embodiment, the text feature extraction results are further input into the corresponding expert neural network in the hybrid expert network for expert decoding processing to improve the accuracy of the obtained language text prediction features.

[0130] Furthermore, such as Figure 7 As shown, step 301 includes the following steps.

[0131] 401. Input the language text label into the second preset neural network for text feature extraction processing to obtain the text feature extraction result.

[0132] 402. The text feature extraction results are input into the router of the hybrid expert network for classification processing to obtain the expert probabilities corresponding one-to-one with multiple expert neural networks.

[0133] 403. The text feature extraction results are input into the target expert neural network with the highest expert probability for expert decoding processing to obtain the language text prediction features of the corresponding language text label.

[0134] 404. The final language text prediction features for the language text label are determined based on the language text prediction features and the highest expert probability. For example, the language text prediction features and the highest expert probability are multiplied together to obtain the final language text prediction features.

[0135] 405. The semantic encoding features and language text prediction features are subjected to a second fusion process to obtain the second fused features. This second fusion process can include splicing or similar methods.

[0136] 406. The second fusion feature is input into the language joint network module for language decoding processing to obtain the language prediction text of the corresponding language of the speech sample data.

[0137] Assuming the existing decoder includes a second pre-defined neural network such as an LSTM module and linear connection layers such as a softmax layer, a hybrid expert network can be added before the linear connection layers to obtain a language decoder using the hybrid expert network, such as... Figure 8 As shown. It should be noted that the original decoder can also be used for other network structures. In Figure 8 In the language decoder of the hybrid expert network shown, after multiplying the language text prediction features with the maximum expert probability, the result of the multiplication process needs to be input into the linear connection layer for further processing to obtain the final language text prediction features.

[0138] 302. Input the semantic encoding features and target text labels into the target decoding network module for the second decoding process to obtain the target predicted text of the target language of the speech sample data.

[0139] In this process, semantic encoding features and target text labels (such as Mandarin text labels) are input into the target decoding network module for a second decoding process to obtain target predicted text (such as Mandarin predicted text, i.e., the predicted Mandarin text).

[0140] The process involves inputting the target text label into the target decoder for text feature extraction to obtain the text feature extraction result of the target text label. The text feature extraction result and semantic encoding features are then subjected to a third fusion process to obtain the third fusion feature. The third fusion process can be concatenation, etc. The third fusion feature is then input into the target joint network module for target decoding to obtain the target predicted text of the target language of the speech sample data, such as the predicted text of Mandarin of the speech sample data.

[0141] During training, the semantic encoded features output by the encoding network module of the hybrid expert network are simultaneously input into the language joint network module (Dialect Jointer) of the language decoding network module and the target joint network module (Target Jointer) of the target decoding network module. At the same time, the language decoder (Dialect MoEDecoder) and the target decoder (Target Decoder) of the hybrid expert network are respectively input into dialect text labels (language text labels) and Mandarin text labels (target text labels). The language decoding network module uses dialect text labels to assist the training of the target decoding network module (Mandarin network module). When the corresponding dialect text labels are missing in the speech sample data, this part of the language decoding network module can be omitted.

[0142] The language joint network module and the target joint network module have the same network structure. Beam search can be used for decoding to obtain the corresponding predicted text.

[0143] 303. Update the network parameters of the initial speech recognition model based on the language predicted text, target predicted text, language text label, and target text label to obtain the speech recognition model.

[0144] The process involves determining the ASR (Advanced Speech Recognition) loss value based on the predicted language text (Dialect Text) and the predicted language text label (Dialect Label), such as determining the dialect orthographic recognition loss value based on the predicted dialect text and the dialect text label. Similarly, the ST (Speed ​​Transmission) loss value is determined based on the predicted target text (Target Text) and the target text label (Target Label), such as determining the translation loss value for translating speech sample data into Mandarin based on the predicted Mandarin text and the Mandarin text label. The network parameters of the initial speech recognition model are updated based on the ASR and ST losses until the training stopping condition is met, at which point training stops, resulting in the speech recognition model. Specifically, the overall loss value of the initial speech recognition model is determined based on the ASR and ST losses, and the network parameters of the initial speech recognition model are updated based on this overall loss value. Thus, the network parameters of the speech recognition model are updated together based on the corresponding loss values ​​of the two sets of decoding network modules, especially the language decoding network module, which helps improve the accuracy of the target text decoding results.

[0145] The above-mentioned speech recognition model training method introduces a hybrid expert network on the RNN-T structure, which enables the obtained speech recognition model to support various dialects very flexibly. It can add new languages ​​at will without losing the original language performance, and can expand the model capacity at will without increasing the amount of computation. It is extremely friendly to limited computing power conditions and can freely switch and adapt to dialects, which greatly improves the accuracy of speech recognition in various mixed language scenarios.

[0146] like Figure 9 The diagram shown is a flowchart of the speech recognition method provided in this embodiment of the application. The method includes the following steps.

[0147] 501. Obtain the target speech data and call the speech recognition model. The speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network.

[0148] The speech recognition model is trained using the speech recognition model training method described in any of the above embodiments. The target speech data can be existing speech data, such as the target speech data during speech-to-text conversion, or it can be speech data generated while the user is speaking, such as frame-by-frame speech data or multi-frame speech data.

[0149] 502. Perform acoustic feature extraction on the target speech data to obtain the target acoustic feature sequence of the target speech data.

[0150] 503. Input the target acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the target speech embedding features of the target speech data.

[0151] The target acoustic feature sequence is input into the embedding module for speech feature extraction processing, which involves mapping the high-dimensional target acoustic feature sequence to obtain the low-dimensional target speech embedding feature of the target speech data, i.e., the low-dimensional acoustic feature sequence. This target speech embedding feature includes shallow information of the target speech data.

[0152] 504. The target speech embedding features and target acoustic feature sequences are input into the neural network module of the hybrid expert network for encoding processing to obtain the target semantic encoding features of the target speech data.

[0153] The neural network module of the hybrid expert network includes a first preset neural network and a hybrid expert network. Correspondingly, step 504 includes: inputting the target acoustic feature sequence into the first preset neural network for acoustic feature extraction processing to obtain target temporal acoustic features including the temporal content of the target speech data; performing a first fusion processing on the target speech embedding features and the target temporal acoustic features to obtain a first target fusion feature; and inputting the first target fusion feature and the target temporal acoustic features into the hybrid expert network for expert coding processing to obtain target semantic coding features of the target speech data.

[0154] The hybrid expert network includes a router and multiple expert networks. The step of inputting the first target fusion feature and the target temporal acoustic feature into the hybrid expert network for expert coding processing to obtain the target semantic coding feature of the target speech data includes: inputting the first target fusion feature into the router for classification processing to obtain expert probabilities corresponding one-to-one with multiple expert neural networks (multiple FFNs); inputting the target temporal acoustic feature into the target expert neural network with the highest expert probability for expert coding processing to obtain the target expert semantic coding feature of the target speech data; and determining the target semantic coding feature of the target speech data based on the target expert semantic coding feature and the highest expert probability.

[0155] Assuming the encoding network module in the embodiments of this application is as follows: Figure 3As shown, correspondingly, after obtaining the target acoustic feature sequence of the target speech data, the method further includes: inputting the target acoustic feature sequence into a forward convolution module for a first convolution process to obtain a first target convolution feature. Correspondingly, the step of inputting the target acoustic feature sequence into an embedding module for speech feature extraction processing includes: inputting the first target convolution feature into the embedding module for speech feature extraction processing; the step of inputting the target acoustic feature sequence into the first preset neural network for acoustic feature extraction processing includes: inputting the first target convolution feature into the first preset neural network for acoustic feature extraction processing; after obtaining the target semantic encoding features of the target speech data, the method further includes: inputting the target semantic encoding features into a backward convolution module for a second convolution process to obtain the final target semantic encoding features. Specifically, please refer to... Figure 4 The embodiments described herein will not be repeated here.

[0156] 505. Input the target semantic encoding features and the preset text start character into the decoding network module for decoding processing to obtain the recognized text of the target speech data.

[0157] When using a speech recognition model, the corresponding decoding network module only includes the target decoding network module, that is, the target decoder and the target joint network, such as the Mandarin decoder and the Mandarin joint network module.

[0158] The first frame of semantic encoding features from the target semantic encoding features and a preset text start character are input into the target decoding network module for decoding processing to obtain the target recognition text of the first frame. The preset text start character is as follows: <sos>The preset text start character represents the beginning of a sentence. After obtaining the target recognition text of the first frame, the semantic encoding features of the second frame and the target recognition text of the first frame are input into the target decoding network module for decoding to obtain the target recognition text of the second frame. The semantic encoding features of the third frame and the target recognition text of the second frame are input into the target decoding network module for decoding to obtain the target recognition text of the third frame. This process continues until all frames are decoded to obtain the final recognition text.

[0159] Specifically, for example, a preset text start character is input into the target decoder for text feature extraction to obtain the text feature extraction result of the preset text start character. The text feature extraction result and the semantic encoding feature of the first frame are then fused together to obtain the third target fusion feature. The third target fusion feature is then input into the target joint network module for target decoding to obtain the target recognition text of the first frame. The target recognition text of the first frame is then input into the target decoder for text feature extraction to obtain the text feature extraction result of the target recognition text of the first frame. The text feature extraction result and the semantic encoding feature of the second frame are then fused together to obtain the third target fusion feature. The third target fusion feature is then input into the target joint network module for target decoding to obtain the target recognition text of the second frame. This process continues in an autoregressive manner until all frames are decoded to obtain the final recognition text, such as the recognition text of Mandarin Chinese.

[0160] Among them, such as Figure 5 As shown, during the training of the semantic recognition model, the decoding network module includes two sets of parallel decoders and a joint network module, including... Figure 5 The speech recognition model uses both solid and dashed lines, but only the solid lines are used during the semantic recognition process. This is because the language decoding network module using the dashed lines has already updated its model parameters during training. When using the speech recognition model, the updated model parameters using the dashed lines are already in use. Thus, during the decoding process, the translated text is directly obtained, reducing the computational load during inference and making the decoding results of the target decoding network module more accurate.

[0161] The speech recognition method in this embodiment eliminates the need for users to manually select their dialect and is no longer limited to a single language during interaction. It automatically identifies the language based on the user's spoken words and directly translates them into more readable Mandarin text, significantly improving the convenience, comfort, and efficiency of human-computer interaction. This speech recognition method can be widely applied to scenarios such as smart cars and smart homes.

[0162] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0163] To facilitate better implementation of the speech recognition model training method of this application, this application also provides a speech recognition model training apparatus. Please refer to... Figure 10 , Figure 10 This is a schematic diagram of the structure of a speech recognition model training device provided in an embodiment of this application. The speech recognition model training device 600 may include a first acquisition unit 601, a first acoustic extraction unit 602, a first embedding extraction unit 603, a first expert coding unit 605, a first decoding unit 606, and a second update unit 607.

[0164] The first acquisition unit 601 is used to acquire speech datasets and initial speech recognition models in multiple different languages. The speech datasets include speech sample data for each different language, language text labels for the corresponding language of the speech sample data, and target text labels for the target language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network.

[0165] The first acoustic extraction unit 602 is used to extract acoustic features from the speech sample data to obtain the acoustic feature sequence of the speech sample data.

[0166] The first embedding extraction unit 603 is used to input the acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data.

[0167] In one embodiment, such as Figure 10 As shown, the device 600 further includes a first update unit 604. The first update unit 604 is used to update the network parameters of the embedding module based on the speech embedding features and the language text labels. Specifically, a loss value can be determined based on the speech embedding features and the language text labels, and the network parameters of the embedding module can be updated based on the loss value. For example, the speech embedding features can be pre-decoded according to a preset decoding method to obtain the language text of the corresponding language of the speech sample data; the speech loss value of the embedding module can be determined based on the language text and the language text labels; and the network parameters of the embedding module can be updated based on the speech loss value.

[0168] The first expert coding unit 605 is used to input the speech embedding features and the acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing to obtain the semantic coding features of the speech sample data.

[0169] The hybrid expert network's neural network module includes a first preset neural network and a hybrid expert network. A first expert coding unit 605 is specifically used to input the acoustic feature sequence into the first preset neural network for acoustic feature extraction processing to obtain temporal acoustic features including the temporal content of the speech sample data; to perform a first fusion processing on the speech embedding features and the temporal acoustic features to obtain a first fused feature; and to input the first fused feature and the temporal acoustic features into the hybrid expert network for expert coding processing to obtain the semantic coding features of the speech sample data.

[0170] The hybrid expert network includes a router and multiple expert neural networks. The step of inputting the fused features and the temporal acoustic features into the hybrid expert network for expert coding processing to obtain the semantic coding features of the speech sample data includes: inputting the first fused features into the router for classification processing to obtain expert probabilities corresponding one-to-one with the multiple expert neural networks; inputting the temporal acoustic features into the target expert neural network with the highest expert probability for expert coding processing to obtain the expert semantic coding features of the speech sample data; and determining the semantic coding features of the speech sample data based on the expert semantic coding features and the highest expert probability.

[0171] In one embodiment, the neural network module of the hybrid expert network further includes a forward convolution module and a backward convolution module, correspondingly, as shown below. Figure 10 As shown, the device 600 further includes a first forward convolution unit 608 and a first backward convolution unit 609.

[0172] The system includes a first forward convolution unit 608, which inputs the acoustic feature sequence into the forward convolution module for a first convolution process to obtain the first convolutional feature. A first embedding extraction unit 603 specifically inputs the first convolutional feature into the embedding module for speech feature extraction processing to obtain the speech embedding feature of the speech sample data. A first expert coding unit 605 inputs the first convolutional feature into the first preset neural network for acoustic feature extraction processing to obtain temporal acoustic features including the temporal content of the speech sample data. Correspondingly, after obtaining the semantic encoding feature of the speech sample data, a first backward convolution unit 609 inputs the semantic encoding feature into the backward convolution module for a second convolution process to obtain the final semantic encoding feature.

[0173] The first decoding unit 606 is used to input the semantic encoding features and the target text label into the decoding network module for decoding processing to obtain the predicted text of the speech sample data.

[0174] The decoding network module includes a target decoding network module and a language decoding network module. The first decoding unit 606 is specifically used to input the semantic encoding features and the language text labels into the language decoding network module for a first decoding process to obtain the language prediction text of the corresponding language of the speech sample data; and to input the semantic encoding features and the target text labels into the target decoding network module for a second decoding process to obtain the target prediction text of the target language of the speech sample data.

[0175] The language decoding network module includes a language decoder of a hybrid expert network and a language joint network module. The step of inputting the semantic encoding features and the language text labels into the language decoding network module for a first decoding process to obtain the language prediction text of the corresponding language of the speech sample data includes: inputting the language text labels into the language decoder of the hybrid expert network for language text decoding processing to obtain the language text prediction features of the language text labels; performing a second fusion process on the semantic encoding features and the language text prediction features to obtain a second fused feature; and inputting the second fused feature into the language joint network module for language decoding processing to obtain the language prediction text of the corresponding language of the speech sample data.

[0176] The language decoder of the hybrid expert network includes a second preset neural network and a hybrid expert network. The step of inputting the language text label into the language decoder of the hybrid expert network for language text decoding processing to obtain the language text prediction features of the corresponding language of the speech sample data includes: inputting the language text label into the second preset neural network for text feature extraction processing to obtain text feature extraction results; and inputting the text feature extraction results into the hybrid expert network for expert decoding processing to obtain the language text prediction features of the speech text label.

[0177] The second update unit 607 is used to update the network parameters of the initial speech recognition model according to the predicted text and the target text label to obtain the speech recognition model.

[0178] Wherein, when the decoding network module includes a target decoding network module and a language decoding network module, the second update unit 607 is specifically used to update the network parameters of the initial speech recognition model based on the language predicted text, the target predicted text, the language text label, and the target text label. Specifically, it determines the language text loss value based on the language predicted text and the language text label; determines the target text loss value based on the target predicted text and the target text label; determines the overall loss value of the initial speech recognition model based on the language text loss value and the target text loss value; and updates the network parameters of the initial speech recognition model based on the overall loss value.

[0179] To facilitate better implementation of the speech recognition method of this application, this application also provides a speech recognition device. Please refer to... Figure 11 , Figure 11 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application. The speech recognition device 700 may include a second acquisition unit 701, a second acoustic extraction unit 702, a second embedding extraction unit 703, a second expert coding unit 704, and a second decoding unit 705.

[0180] The second acquisition unit 701 is used to acquire target speech data and call a speech recognition model, wherein the speech recognition model includes an embedding module, a neural network module of hybrid expert network and a decoding network module.

[0181] The second acoustic extraction unit 702 is used to extract acoustic features from the target speech data to obtain the target acoustic feature sequence of the target speech data.

[0182] The second embedding extraction unit 703 is used to input the target acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the target speech embedding features of the target speech data.

[0183] The second expert coding unit 704 is used to input the target speech embedding features and the target acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing, so as to obtain the target semantic coding features of the target speech data.

[0184] The hybrid expert network's neural network module includes a first preset neural network and a hybrid expert network. A second expert coding unit 704 is specifically used to input the target acoustic features into the first preset neural network for acoustic feature extraction processing to obtain target temporal acoustic features including the temporal content of the target speech data; to perform a first fusion processing on the target speech embedding features and the target temporal acoustic features to obtain a first target fusion feature; and to input the first target fusion feature and the target temporal acoustic features into the hybrid expert network for expert coding processing to obtain target semantic coding features of the target speech data.

[0185] The second decoding unit 705 is used to input the target semantic encoding features and the preset text start character into the decoding network module for decoding processing, so as to obtain the recognition text of the target speech data.

[0186] In one embodiment, such as Figure 11 As shown, the device 700 may further include a second forward convolution unit 706 and a second backward convolution unit 707. The second forward convolution unit 706 is used to input the target acoustic feature sequence into the forward convolution module for a first convolution process to obtain a first target convolution feature. Correspondingly, the second embedding extraction unit 703 is used to input the first target convolution feature into the embedding module for speech feature extraction processing to obtain target speech embedding features of the target speech data; the second expert coding unit 704 is used to input the first target convolution feature into the first preset neural network for acoustic feature extraction processing to obtain target temporal acoustic features including the temporal content of the target speech data. Correspondingly, after obtaining the target semantic coding features of the target speech data, the first backward convolution unit 709 is used to input the target semantic coding features into the backward convolution module for a second convolution process to obtain the final target semantic coding features.

[0187] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0188] Accordingly, embodiments of this application also provide a computer device, which can be a terminal or a server. For example... Figure 12 As shown, Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 800 includes a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, and a computer program stored in the memory 802 and executable on the processor. The processor 801 is electrically connected to the memory 802.

[0189] The processor 801 is the control center of the computer device 800. It connects various parts of the computer device 800 through various interfaces and lines. By running or loading software programs (computer programs) and / or modules stored in the memory 802, and calling data stored in the memory 802, it performs various functions of the computer device 800 and processes data, thereby monitoring the computer device 800 as a whole.

[0190] In this embodiment, the processor 801 in the computer device 800 loads the instructions corresponding to the processes of one or more applications into the memory 802 according to the following steps, and the processor 801 runs the applications stored in the memory 802 to achieve the functions of any of the above method embodiments, such as any speech recognition model training method and / or any of the above speech recognition methods, for example:

[0191] Acquire speech datasets and an initial speech recognition model for multiple different languages. The speech datasets include speech sample data for each language, language text labels for the corresponding languages ​​of the speech sample data, and target text labels for the target languages ​​of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network. Acoustic features are extracted from the speech sample data to obtain an acoustic feature sequence. The acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain speech embedding features of the speech sample data. The speech embedding features and the acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain semantic encoding features of the speech sample data. The semantic encoding features and the target text labels are input into the decoding network module for decoding processing to obtain predicted text of the speech sample data. The network parameters of the initial speech recognition model are updated based on the predicted text and the target text labels to obtain the speech recognition model; and / or,

[0192] The process involves acquiring target speech data and invoking a speech recognition model. This model includes an encoding network module and a decoding network module within a hybrid expert network. The encoding network module further includes a neural network module and an embedding module within the hybrid expert network. Acoustic features are extracted from the target speech data to obtain a target acoustic feature sequence. This target acoustic feature sequence is then input into the embedding module for speech feature extraction to obtain target speech embedding features. The target speech embedding features and the target acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding to obtain target semantic encoding features. Finally, the target semantic encoding features and a preset text start character are input into the decoding network module for decoding to obtain the recognized text of the target speech data.

[0193] The specific implementation and beneficial effects of the various operations that the processor can execute can be found in the preceding method embodiments, and will not be repeated here.

[0194] Optional, such as Figure 12 As shown, the computer device 800 also includes: a touch screen display 803, a radio frequency circuit 804, an audio circuit 805, an input unit 806, and a power supply 807. The processor 801 is electrically connected to the touch screen display 803, the radio frequency circuit 804, the audio circuit 805, the input unit 806, and the power supply 807. Those skilled in the art will understand that... Figure 12 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0195] The touch display screen 803 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 803 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar devices. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the data to the processor 801 to determine the type of touch event. Subsequently, the processor 801 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and display panel can be integrated into the touch display screen 803 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 803 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 803 can also be used as part of the input unit 806 to achieve input functions.

[0196] In this embodiment, the touch display screen 803 is used to present a graphical user interface and receive operation commands generated by the user interacting with the graphical user interface.

[0197] The radio frequency circuit 804 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0198] Audio circuitry 805 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 805 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 805, converted back into audio data, and output to processor 801 for processing. The audio data is then transmitted via radio frequency circuitry 804 to, for example, another computer device, or output to memory 802 for further processing. Audio circuitry 805 may also include an earphone jack to facilitate communication between peripheral headphones and computer devices.

[0199] The input unit 806 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0200] Power supply 807 is used to supply power to various components of computer device 800. Optionally, power supply 807 can be logically connected to processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 807 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0201] although Figure 12 As not shown in the diagram, the computer device 800 may also include a camera, sensors, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0202] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0203] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0204] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute steps in any of the methods provided in embodiments of this application, such as any of the above-described speech recognition model training methods and any of the speech recognition methods. For example, the computer program can execute the following steps:

[0205] Acquire speech datasets and an initial speech recognition model for multiple different languages. The speech datasets include speech sample data for each language, language text labels for the corresponding languages ​​of the speech sample data, and target text labels for the target languages ​​of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network. Acoustic features are extracted from the speech sample data to obtain an acoustic feature sequence. The acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain speech embedding features of the speech sample data. The speech embedding features and the acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain semantic encoding features of the speech sample data. The semantic encoding features and the target text labels are input into the decoding network module for decoding processing to obtain predicted text of the speech sample data. The network parameters of the initial speech recognition model are updated based on the predicted text and the target text labels to obtain the speech recognition model; and / or,

[0206] The process involves acquiring target speech data and invoking a speech recognition model. This model includes an encoding network module and a decoding network module within a hybrid expert network. The encoding network module further includes a neural network module and an embedding module within the hybrid expert network. Acoustic features are extracted from the target speech data to obtain a target acoustic feature sequence. This target acoustic feature sequence is then input into the embedding module for speech feature extraction to obtain target speech embedding features. The target speech embedding features and the target acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding to obtain target semantic encoding features. Finally, the target semantic encoding features and a preset text start character are input into the decoding network module for decoding to obtain the recognized text of the target speech data.

[0207] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0208] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0209] Since the computer program stored in the storage medium can execute the steps in any of the multilingual fusion semantic representation methods provided in the embodiments of this application, the beneficial effects that any of the multilingual fusion semantic representation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0210] The foregoing has provided a detailed description of a multilingual fusion semantic representation method, apparatus, storage medium, and computer device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / sos>

Claims

1. A speech recognition method, characterized in that, include: Acquire target speech data and invoke a speech recognition model, wherein the speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network, and the encoding network module includes a neural network module and an embedding module of the hybrid expert network; The target speech data is subjected to acoustic feature extraction to obtain the target acoustic feature sequence of the target speech data; The target acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain the target speech embedding features of the target speech data; The target speech embedding features and the target acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain the target semantic encoding features of the target speech data; The target semantic encoding features and the preset text start character are input into the decoding network module for decoding processing to obtain the recognized text of the target speech data.

2. The method according to claim 1, characterized in that, The neural network module of the hybrid expert network includes a first preset neural network and a hybrid expert network. The step of inputting the target speech embedding features and the target acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing to obtain the target semantic encoding features of the target speech data includes: The target acoustic feature sequence is input into the first preset neural network for acoustic feature extraction processing to obtain target temporal acoustic features including the temporal content of the target speech data; The target speech embedding features and the target temporal acoustic features are subjected to a first fusion process to obtain a first target fusion feature; The first target fusion feature and the target temporal acoustic feature are input into the hybrid expert network for expert coding processing to obtain the target semantic coding features of the target speech data.

3. The method according to claim 2, characterized in that, The hybrid expert network includes routers and multiple expert neural networks. The step of inputting the first target fusion feature and the target temporal acoustic feature into the hybrid expert network for expert coding processing to obtain the target semantic coding feature of the target speech data includes: The first target fusion feature is input into the router for classification processing to obtain the expert probabilities that correspond one-to-one with multiple expert neural networks; The target temporal acoustic features are input into the target expert neural network with the highest expert probability for expert encoding processing to obtain the target expert semantic encoding features of the target speech data; The target semantic coding features of the target speech data are determined based on the target expert semantic coding features and the maximum expert probability.

4. The method according to claim 2, characterized in that, The neural network module of the hybrid expert network further includes a forward convolution module and a backward convolution module. After obtaining the target acoustic feature sequence of the target speech data, it also includes: The target acoustic feature sequence is input into the forward convolution module for the first convolution process to obtain the first target convolution feature; The step of inputting the target acoustic feature sequence into the embedding module for speech feature extraction processing includes: inputting the first target convolutional feature into the embedding module for speech feature extraction processing; The step of inputting the target acoustic feature sequence into the first preset neural network for acoustic feature extraction processing includes: inputting the first target convolutional features into the first preset neural network for acoustic feature extraction processing; After obtaining the target semantic coding features of the target speech data, the method further includes: inputting the target semantic coding features into the backward convolution module for a second convolution process to obtain the final target semantic coding features.

5. The method according to any one of claims 1-4, characterized in that, The speech recognition model is trained through the following steps: A speech dataset and an initial speech recognition model in multiple languages ​​are obtained. The speech dataset includes speech sample data in each language, language text labels for the corresponding language of the speech sample data, and target text labels for the target language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network. The speech sample data is subjected to acoustic feature extraction to obtain the acoustic feature sequence of the speech sample data; The acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data; The speech embedding features and the acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain the semantic encoding features of the speech sample data; The semantic encoding features and the target text labels are input into the decoding network module for decoding processing to obtain the predicted text of the speech sample data; The network parameters of the initial speech recognition model are updated based on the predicted text and the target text label to obtain the speech recognition model.

6. The method according to claim 5, characterized in that, After obtaining the speech embedding features of the speech sample data, the method further includes: The speech loss value of the embedding module is determined based on the speech embedding features and the language text labels; The network parameters of the embedding module are updated based on the speech loss value.

7. The method according to claim 6, characterized in that, The step of determining the speech loss value of the embedding module based on the speech embedding features and the language text label includes: According to a preset decoding method, the speech embedding features are subjected to preset decoding processing to obtain the language text of the corresponding language of the speech sample data; The speech loss value of the embedding module is determined based on the language text and the language text label.

8. The method according to claim 5, characterized in that, The decoding network module includes a target decoding network module and a language decoding network module; The step of inputting the semantic encoding features and the target text label into the decoding network module for decoding processing to obtain the predicted text of the speech sample data includes: The semantic encoding features and the language text labels are input into the language decoding network module for the first decoding process to obtain the language prediction text of the corresponding language of the speech sample data; The semantic encoding features and the target text labels are input into the target decoding network module for a second decoding process to obtain the target predicted text of the target language of the speech sample data; The step of updating the network parameters of the initial speech recognition model based on the predicted text and the target text label includes: The network parameters of the initial speech recognition model are updated based on the predicted text in the language, the predicted text in the target language, the text label in the language, and the text label in the target language.

9. The method according to claim 8, characterized in that, The language decoding network module includes a language decoder with a hybrid expert network and a language joint network module. The step of inputting the semantic encoding features and the language text labels into the language decoding network module for a first decoding process to obtain the language prediction text of the corresponding language of the speech sample data includes: The language text label is input into the language decoder of the hybrid expert network for language text decoding processing to obtain the language text prediction features of the language text label; The semantic encoding features and the language text prediction features are then subjected to a second fusion process to obtain a second fused feature; The second fusion feature is input into the language joint network module for language decoding processing to obtain the language prediction text of the corresponding language of the speech sample data.

10. The method according to claim 9, characterized in that, The language decoder of the hybrid expert network includes a second preset neural network and a hybrid expert network. The step of inputting the language text label into the language decoder of the hybrid expert network for language text decoding processing to obtain the language text prediction features of the corresponding language of the speech sample data includes: The language text labels are input into the second preset neural network for text feature extraction processing to obtain the text feature extraction results; The extracted text features are input into a hybrid expert network for expert decoding to obtain the language text prediction features of the language text label.

11. The method according to claim 8, characterized in that, The step of updating the network parameters of the initial speech recognition model based on the language-predicted text, the target predicted text, the language text label, and the target text label includes: The language text loss value is determined based on the predicted language text and the language text label; Determine the target text loss value based on the target predicted text and the target text label; The overall loss value of the initial speech recognition model is determined based on the language text loss value and the target text loss value; The network parameters of the initial speech recognition model are updated based on the overall loss value.

12. A method for training a speech recognition model, characterized in that, include: Acquire speech datasets and initial speech recognition models for multiple different languages. The speech datasets include speech sample data for each different language, language text labels for the corresponding language of the speech sample data, and general text labels for the general language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network. The speech sample data is subjected to acoustic feature extraction to obtain the acoustic feature sequence of the speech sample data; The acoustic feature sequence is input into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data; The speech embedding features and the acoustic feature sequence are input into the neural network module of the hybrid expert network for encoding processing to obtain the semantic encoding features of the speech sample data; The semantic encoding features and the general text labels are input into the decoding network module for decoding processing to obtain the predicted text of the speech sample data; The network parameters of the initial speech recognition model are updated based on the predicted text and the general text label to obtain the speech recognition model.

13. A speech recognition model training device, characterized in that, include: The first acquisition unit is used to acquire speech datasets and initial speech recognition models in multiple different languages. The speech datasets include speech sample data for each different language, language text labels for the corresponding language of the speech sample data, and target text labels for the target language of the speech sample data. The initial speech recognition model includes an encoding network module and a decoding network module of a hybrid expert network. The encoding network module includes a neural network module and an embedding module of the hybrid expert network. The first acoustic extraction unit is used to extract acoustic features from the speech sample data to obtain the acoustic feature sequence of the speech sample data. The first embedding extraction unit is used to input the acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the speech embedding features of the speech sample data; The first expert coding unit is used to input the speech embedding features and the acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing to obtain the semantic coding features of the speech sample data; The first decoding unit is used to input the semantic encoding features and the target text label into the decoding network module for decoding processing, so as to obtain the predicted text of the speech sample data; The second update unit is used to update the network parameters of the initial speech recognition model according to the predicted text and the target text label, so as to obtain the speech recognition model.

14. A voice recognition device, characterized in that, include: The second acquisition unit is used to acquire target speech data and call a speech recognition model, wherein the speech recognition model includes an embedding module, a neural network module of hybrid expert network and a decoding network module; The second acoustic extraction unit is used to extract acoustic features from the target speech data to obtain the target acoustic feature sequence of the target speech data. The second embedding extraction unit is used to input the target acoustic feature sequence into the embedding module for speech feature extraction processing to obtain the target speech embedding features of the target speech data; The second expert coding unit is used to input the target speech embedding features and the target acoustic feature sequence into the neural network module of the hybrid expert network for encoding processing, so as to obtain the target semantic coding features of the target speech data; The second decoding unit is used to input the target semantic encoding features and the preset text start character into the decoding network module for decoding processing, so as to obtain the recognition text of the target speech data.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the method as described in any one of claims 1-12.

16. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the steps of the method as described in any one of claims 1-12 by invoking the computer program stored in the memory.

Images