Speech Recognition Method, Apparatus and Computer Readable Storage Medium

Abstract

The present application discloses a voice recognition method, apparatus and computer-readable storage medium. Among them, the method includes: obtaining voice information to be recognized; respectively extracting features from the voice information by using at least two sub-encoding networks in an encoding network to obtain at least two voice features; wherein each of the sub-encoding networks is trained according to sample data of a corresponding language; using a fusion network to fuse the at least two voice features to obtain a fused voice feature; using a decoding network to decode the fused voice feature to obtain a voice recognition result. By the above method, the present application can improve the accuracy of voice recognition.

Description

Technical Field

[0001] This application relates to the field of speech recognition, and in particular, to a speech recognition method, apparatus, and computer-readable storage medium. Background Art

[0002] Speech recognition enables a machine to automatically convert speech into corresponding text through machine learning methods, thus endowing the machine with a function similar to human hearing and being an important part of artificial intelligence.

[0003] Since the adoption of hybrid modeling based on deep neural networks ten years ago, the accuracy of automatic speech recognition has been significantly improved. With the continuous breakthrough of artificial intelligence technology and the increasing popularity of various intelligent terminal devices, speech recognition, as an important part of human-computer interaction, is widely used on various intelligent terminals, and more and more users are accustomed to using voice input. This breakthrough uses DNN to replace the traditional Gaussian mixture model for acoustic likelihood evaluation, while still retaining all components such as acoustic models, language models, and vocabulary models as a hybrid ASR system. Summary of the Invention

[0004] This application mainly provides a speech recognition method, apparatus, and computer-readable storage medium, which solve the problem of low speech recognition accuracy in the prior art.

[0005] To solve the above technical problem, in the first aspect of this application, a speech recognition method is provided, including: obtaining speech information to be recognized; respectively extracting features of the speech information by using at least two sub-encoding networks in an encoding network to obtain at least two speech features; wherein each sub-encoding network is trained according to sample data of the corresponding language; fusing the at least two speech features by using a fusion network to obtain a fused speech feature; and decoding the fused speech feature by using a decoding network to obtain a speech recognition result.

[0006] To solve the above technical problem, in the second aspect of this application, a speech recognition apparatus is provided, including: an obtaining module, configured to obtain speech information to be recognized; an encoding module, configured to respectively extract features of the speech information by using at least two sub-encoding networks in an encoding network to obtain at least two speech features; wherein each sub-encoding network is trained according to sample data of the corresponding language; a fusion module, configured to fuse the at least two speech features by using a fusion network to obtain a fused speech feature; and a decoding module, configured to decode the fused speech feature by using a decoding network to obtain a speech recognition result.

[0007] To solve the above technical problems, a third aspect of the present application provides a voice recognition device, which includes a processor and a memory coupled to each other; a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the voice recognition method provided in the first aspect as described above.

[0008] To solve the above technical problems, a fourth aspect of the present application provides a computer-readable storage medium, which stores program data. When the program data is executed by a processor, the voice recognition method provided in the first aspect as described above is implemented.

[0009] The beneficial effects of the present application are as follows: Different from the prior art, the present application uses at least two sub-encoding networks in the encoding network to respectively extract features from the voice information to obtain at least two voice features, where each sub-encoding network is trained according to the sample data of the corresponding language, and then the fusion network is used to fuse the at least two voice features to obtain a fused voice feature, and finally the decoding network is used to decode the fused voice feature to obtain a voice recognition result. Each sub-encoding network does not affect each other, the accuracy of the extracted voice features is higher, and when applied to the final decoding, the voice recognition result is more accurate. Description of the Drawings

[0010] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0011] Figure 1 It is a structural schematic block diagram of an embodiment of the voice recognition model of the present application;

[0012] Figure 2 It is a flow schematic block diagram of an embodiment of the training method of the voice recognition model of the present application;

[0013] Figure 3 It is a structural schematic block diagram of an embodiment of the preset language recognition model of the present application;

[0014] Figure 4 It is a flow schematic block diagram of an embodiment of the voice recognition method of the present application;

[0015] Figure 5 It is a flow schematic block diagram of an embodiment of step S12 of the present application;

[0016] Figure 6 It is a flow schematic block diagram of an embodiment of step S13 of the present application;

[0017] Figure 7 It is a flowchart showing an embodiment of step S33 of the present application;

[0018] Figure 8 It is a flowchart showing another embodiment of step S33 of the present application;

[0019] Figure 9 It is a structural schematic diagram of an embodiment of the voice recognition device of the present application;

[0020] Figure 10 It is a structural schematic diagram of another embodiment of the voice recognition device of the present application;

[0021] Figure 11 It is a structural schematic diagram of an embodiment of the computer-readable storage medium of the present application. Detailed implementation manners

[0022] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0023] The terms "first" and "second" in the present application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0024] Referring to "embodiment" in this article means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification not necessarily referring to the same embodiment, nor an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0025] Building a multilingual ASR system with traditional hybrid models is difficult because each language usually has its own language-specific inventory of phonemes and words. Most hybrid multilingual solutions focus on building an acoustic model with a shared hidden layer, while each language has its own lexical model and language model. In contrast, building a multilingual end-to-end ASR system is made very easy by simply using the union set of tokens (such as characters or subwords) of all languages as the output token set and then training an end-to-end model on all the data. This end-to-end model is a general ASR model that can recognize speech in any language as long as that language was used during training.

[0026] However, pooling all languages together to train a multilingual model is a double-edged sword. While it is simple and maximizes cross-language sharing, it also causes confusion between different languages during the recognition process. At the same time, there are other factors to consider when designing a multilingual end-to-end model. When a multilingual model covers a large number of languages, the data from these languages is severely imbalanced. This imbalanced data usually causes the model to perform well on resource-rich languages but fail on resource-poor languages. To address this issue, data sampling is usually used to balance the amount of training data. The model capacity should also be expanded to recognize a large number of languages. In summary, it is very inconvenient to model a multilingual recognition model using traditional models. Although end-to-end models have achieved remarkable success in multilingual modeling, there are still issues such as confusion between different languages during the recognition process and performance differences caused by differences in data ratios.

[0027] This application provides a speech recognition model. For the specific speech recognition model and speech recognition method, please refer to the following embodiments.

[0028] Please refer to Figure 1 , Figure 1 which is a structural schematic block diagram of an embodiment of the speech recognition model of this application. The speech recognition model 400 of this embodiment includes an input layer 410, an encoding network 420, a fusion network 430, and a decoding network 440. Among them, the input layer 410 is used to input the speech information to be recognized, the encoding network 420 is used to obtain the speech features of the speech information to be recognized, the fusion network 430 is used to perform a fusion operation on the speech features, and the decoding network 440 is used to decode the features after the fusion operation and output the recognition result of the speech information to be recognized.

[0029] Among them, the encoding network 420 includes multiple sub-encoding networks, and each sub-encoding network is trained according to the sample data of the corresponding language. The speech recognition model of this embodiment only schematically shows the first sub-encoding network 421 and the second sub-encoding network 422. In fact, it may also include a third sub-encoding network, a fourth sub-encoding network, etc., without limiting the number of sub-encoding networks.

[0030] Please refer to Figure 2 , Figure 2 which is a flowchart showing an embodiment of the training method of the speech recognition model of the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 2 the process sequence shown. The encoding network in this embodiment is obtained through the following steps:

[0031] Step S41: Use sample data in at least two languages to train a preset language recognition model respectively.

[0032] The sample data in various languages for training each sub-encoding network can be language data of different countries, language data of different ethnic groups, or even dialect languages in different regions of the same country. For example, Russian, Mandarin, ethnic minority languages, Cantonese, Wu dialect, etc. can all be used as sample data.

[0033] Among them, please refer to Figure 3 , Figure 3 which is a structural schematic diagram of an embodiment of the preset language recognition model of the present application. The language recognition model 500 includes an input layer 510, an encoder 520, and a decoder 530. Using sample data in a single language to train this language recognition model 500, the input layer 510 inputs sample speech data, the encoder 520 extracts sample speech features of the sample speech data, and the decoder 530 predicts the corresponding semantic information and outputs it according to the sample speech features, and adjusts the parameters of the encoder 520 according to the predicted semantic information.

[0034] Specifically, this step uses sample data in at least two languages to train at least two preset end-to-end language recognition models 500, and each language recognition model 500 is trained using sample data in a single language respectively. In this way, each obtained encoder 520 can achieve high-precision feature extraction of speech information in a single language, reducing the confusion between different languages.

[0035] Step S42: Extract the encoder part from the trained language recognition models respectively as sub-encoding networks.

[0036] Train multiple language recognition models 500, extract the encoder 520 from them, and use it as the sub-encoding network in the above encoding network 420.

[0037] Among them, each sub-encoding network corresponds to a language label, and this language label is the language type of the sample data used when training the corresponding sub-encoding network. For example, if the above first sub-encoding network 421 is trained using Cantonese, then its corresponding language label is Cantonese, and if the second sub-encoding network 422 is trained using Mandarin, then its corresponding language label is Mandarin.

[0038] Step S43: Connect the output layer of the sub-encoding network to the input layer of the fusion network.

[0039] In this step, the output layer of the sub-encoding network is connected to the input layer of the fusion network 430 to form a decoupled speech recognition model, and each sub-encoding network is independent of each other. Since each sub-encoding network is trained with sample data of different languages, it can extract high-precision features for the speech of a single language.

[0040] Since the volumes of language data of different languages vary, for example, there are obvious differences between Cantonese and Mandarin data. The amount of Cantonese data is less, and the ratio to Mandarin data is about 1:20, resulting in a great imbalance in language volume. However, by training the corresponding sub-encoding networks according to the data of each language, whether the data volumes of each language are balanced or not has little impact on the feature extraction of speech information. Therefore, the sample data volumes of each language can be selected without restriction, so that each sub-encoding network can have excellent performance in feature extraction of the speech information of the corresponding language, and the speech features can express the original speech information more accurately.

[0041] Among them, when the parameters of the sub-encoding network are fixed, the fusion network 430 can also be trained with sample data of at least two languages whose data volumes meet a certain degree of balance.

[0042] Among them, the data volumes meeting a certain degree of balance means that the difference in data volumes between any two sample data of different languages is less than or equal to 20%. In order to improve the recognition accuracy for different languages, the difference in data volumes between any two sample data can also be controlled to be smaller, such as 10%, 5%, or even the difference is 0, and the sample data volumes are exactly the same.

[0043] Among them, the sample data for training the fusion network 430 includes the sample data with the language labels corresponding to the sub-encoding network.

[0044] Different from the prior art, in this embodiment, the sub-encoding networks are trained with sample data of various languages respectively. Each language can use as much sample data of that language as possible. Each sub-encoding network has excellent performance in the recognition accuracy of the speech information of the corresponding language, can be free from the constraint of data balance, and reduces the confusion between different languages; at the same time, a single-language model is trained separately, and then the fusion network 430 is trained with balanced data, achieving the effect of evenly improving the recognition effect under the condition of data imbalance.

[0045] Please refer to Figure 4 , Figure 4 which is a flowchart of an embodiment of the speech recognition method of this application. It should be noted that if there are substantially the same results, this embodiment is not limited to the Figure 4 shown process sequence. This embodiment includes the following steps:

[0046] Step S11: Obtain the speech information to be recognized.

[0047] The speech information to be recognized can be pre-recorded by a sound capture device such as a microphone, or can be recorded in real time, or can also be an audio segment intercepted from an audio file in a storage device or a video file containing audio information. For an audio file, its format can be WAV, FLAC, MP3, AAC, etc. This is only for illustrative purposes and does not represent a limitation on the content, format, acquisition method, etc. of the speech information.

[0048] Step S12: Use at least two sub-encoding networks in the encoding network to extract features from the speech information respectively, and obtain at least two speech features.

[0049] Among them, the speech feature is the acoustic feature in the speech information. For example, feature vectors such as MFCC (Mel Frequency Cepstrum Coefficient), FBank, and PLP (Perceptual Linear Predictive) of the speech information to be recognized can be extracted as its speech feature representation.

[0050] The at least two sub-encoding networks can be some or all of the sub-encoding networks pre-selected in the encoding network. For example, the language tags of the sub-encoding networks include European languages and Mandarin Chinese. A language option can be set on its deployment device. The user selects some or all of the language types according to their usage habits. After the language selection is completed, the sub-encoding networks corresponding to the selected language types in the encoding network are enabled, and those not selected are not enabled. During the speech recognition process, the branches of the sub-encoding networks not selected do not participate in the speech feature extraction and have no output.

[0051] Please refer to Figure 5 , Figure 5 which is a flowchart showing an embodiment of step S12 of this application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 5 the process sequence shown. This embodiment uses each sub-encoding network to perform the following steps respectively:

[0052] Step S21: Obtain the temporal features of the speech information.

[0053] Specifically, the sub-encoding network can include a long short-term memory network. In this embodiment, the long short-term memory network is used to extract the temporal features of the speech information, and the single-direction and bidirectional long short-term memory networks are used to extract the single-direction and bidirectional features of the speech information, and finally the speech features are obtained.

[0054] In one embodiment, the sub-encoding network includes a first long short-term memory network, a second long short-term memory network, and a third long short-term memory network. In this embodiment, the temporal features of the voice information are obtained. Specifically, the voice information is input into the first long short-term memory network to obtain the first temporal features, the first temporal features are input into the second long short-term memory network to obtain the second temporal features, and the second temporal features are input into the third long short-term memory network to obtain the final temporal features.

[0055] Other networks that can extract temporal features from voice information can also be used to obtain the temporal features in step S21, which is not limited here.

[0056] Step S22: Perform one-way and two-way feature extraction based on the temporal features to obtain voice features.

[0057] Among them, the sub-encoding network may further include a unidirectional and bidirectional long short-term memory network (UB-LSTM). In this step, the temporal features are input into the unidirectional and bidirectional long short-term memory network to implement the extraction of one-way and two-way features of the voice information to be recognized, and voice features are output.

[0058] In this step, at least two sub-encoding networks are used to extract features from the voice information, and each sub-encoding network outputs a voice feature, resulting in at least two voice features.

[0059] Step S13: Use a fusion network to fuse at least two voice features to obtain a fused voice feature.

[0060] Among them, the fusion operation is to enhance the attention of at least two voice features, and the obtained fused voice feature is more in line with the language characteristics of the voice information to be recognized.

[0061] Please refer to Figure 6 , Figure 6 is a flowchart of an embodiment of step S13 of this application. It should be noted that if there are substantially the same results, this embodiment is not limited to the Figure 6 shown process sequence. This embodiment includes the following steps:

[0062] Step S31: Perform a first fusion operation on at least two voice features to obtain a first fusion feature.

[0063] Among them, the first fusion operation is to perform dimensionality reduction and splicing operations on at least two voice features using a first fully connected layer network. Specifically, the above voice recognition model 400 may further include a first fully connected layer network DNN for performing dimensionality reduction operations on at least two voice features, and then using a splicing algorithm to splice the multiple features obtained by dimensionality reduction.

[0064] Step S32: Classify the language of the voice information based on the first fusion feature to obtain a language feature.

[0065] Among them, the above voice recognition model 400 may further include an LID (Language Identification), and this step uses the language classifier to extract language information to obtain a language feature.

[0066] Specifically, the language classifier includes an input layer, a hidden layer, and an output layer. The first fusion feature is input from the input layer of the language classifier, and the hidden layer extracts features from the first fusion feature to obtain a language feature.

[0067] Step S33: Perform a second fusion operation on the voice feature and the language feature to obtain a fused voice feature.

[0068] In this step, the voice features and language features output by each sub-encoding network are fused. Please refer to Figure 7 , Figure 7 which is a flowchart of an embodiment of step S33 of this application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 7 the process sequence shown. This embodiment includes the following steps:

[0069] Step S331: Multiply each voice feature by the language feature to obtain a corresponding weighted feature.

[0070] In this step, each voice feature output by each sub-encoding network is multiplied by the language feature to respectively obtain a weighted feature corresponding to each voice feature.

[0071] Specifically, in this step, after each voice feature is dimensionally reduced, it is multiplied by the language feature.

[0072] The above step of multiplying the language feature by each voice feature realizes the enhancement of the attention of a single language, and the obtained fusion feature is more in line with the actual language type of the voice information to be recognized.

[0073] Step S332: Sum the weighted features and use a second fully connected layer network to perform dimensional expansion to obtain a fused voice feature.

[0074] In this step, all the weighted features obtained in the previous step are added up and dimensional expansion is performed to obtain a fused voice feature.

[0075] Please refer to Figure 8 , Figure 8 which is a flowchart of another embodiment of step S33 of this application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 8 the process sequence shown. This embodiment includes the following steps:

[0076] Step S333: Determine the probabilities of the speech information to be recognized for the languages corresponding to the respective sub - coding networks according to the language characteristics.

[0077] In this step, language type prediction is performed according to the language characteristics, and according to the language labels of the sub - coding networks participating in the speech feature coding, the probabilities of the language type of the speech information to be recognized for the language types corresponding to the respective language labels are determined.

[0078] Step S334: Perform a weighted fusion operation on the speech features output by the respective sub - coding networks according to the probabilities to obtain the fused speech features.

[0079] Specifically, weighted fusion of the speech features is performed according to the probabilities, that is, using the probability values obtained in the previous step as weights, the speech features output by the corresponding sub - coding networks are weighted and summed respectively to obtain the fused speech features.

[0080] For example, when inputting a piece of speech information, the speech features of this speech information are extracted by using sub - coding network 1, sub - coding network 2, and sub - coding network 3 to obtain feature a, feature b, and feature c. Among them, the language label corresponding to sub - coding network 1 is language 1, the language label corresponding to sub - coding network 2 is language 2, and the language label corresponding to sub - coding network 3 is language 3. The probability that the speech information is predicted to be language 1 in the previous step is A, the probability of being language 2 is B, and the probability of being language 3 is C. Then the obtained fused speech feature can be expressed as: a×A + b×B + c×C.

[0081] Step S14: Use the decoding network to decode the fused speech features to obtain the speech recognition result.

[0082] The decoding network obtains the speech recognition result according to the fused speech features. The speech recognition result can be displayed in text form on an electronic screen. For example, when performing speech - to - text input, the electronic device receives the speech information sent by the user, converts the speech information into text in the above - mentioned manner of this application, and displays it on the display screen of the electronic device; or, the speech recognition result can also be sent to an intelligent terminal to generate a reply according to the speech recognition result. For example, when having a conversation with an artificial intelligence virtual customer service, the user sends speech information, the virtual customer service terminal performs speech recognition to obtain the recognition result, and generates a reply according to the recognition result.

[0083] In one embodiment, the decoding network can be stacked by MoChaAttention (Monotonic ChunkAttention) and Long Short-Term Memory (LSTM) networks. Specifically, in the decoding process, first, through MoChaAttention, the attention and extraction of the effective information in the fused speech features are realized, and then the temporal correlation is enhanced through one layer of LSTM, and finally the speech recognition result is output. Among them, on the one hand, the MoChaAttention mechanism has the advantages of linear time complexity and real-time decoding, and has a high decoding efficiency. On the other hand, MoChaAttention moves a fixed-length window, and soft attention is realized inside the window, which realizes the enhancement of local information attention and the function of quickly purifying and decoding the fused speech features, and efficiently decodes.

[0084] Different from the prior art, the sub-encoding networks in the encoding network of the present application are trained with the sample data of the corresponding languages respectively. Each sub-encoding network has a high accuracy in extracting the features of the speech information corresponding to its language label, and the features have a good expression effect on the speech information. Further, the fusion network fuses multiple speech features and weights the speech features output by each sub-encoding network according to the language recognition result, realizing the enhancement of the attention of a single language, making the fusion of speech features more targeted, reducing the confusion between different languages, and improving the recognition accuracy.

[0085] The above speech recognition model has strong scalability. When a new language recognition function needs to be added, only a new language recognition model needs to be trained separately with the sample data of the new language, and the obtained encoder is used as a sub-encoding network and connected into the above encoding network 420 of the present application. Then, the fusion network 430 is trained with the balanced sample data of the languages corresponding to each sub-encoder, and the parameters in the fusion network 430 are adjusted to complete the training of the model. The updated speech recognition model has the speech recognition function of the new language. For example, the original speech recognition model does not have the function of recognizing Sichuan dialect, that is, there is no sub-encoding network trained with Sichuan dialect sample data. Now, if the function of recognizing Sichuan dialect needs to be added, a language recognition model is trained with Sichuan dialect sample data, the encoder of the language recognition model is extracted and connected into the encoding network 420 to obtain a new speech recognition model structure. Then, the fusion network 430 is trained with the balanced sample data of the languages corresponding to the sub-encoding networks in the encoding network 420 to obtain an updated speech recognition model, and the updated speech recognition model has the function of recognizing Sichuan dialect.

[0086] When it is necessary to update the existing language recognition function, only a new sample data of the language is needed to retrain a language recognition model, extract its encoder, and use the encoder to replace the old sub-encoding network corresponding to the language, so as to improve the speech recognition effect of the language without changing the speech recognition effect of other languages. For example, a sub-encoding network of the original speech recognition model is trained with 2,000 hours of Cantonese sample data. In order to improve the recognition accuracy of Cantonese speech, a language recognition model is trained with 5,000 hours of Cantonese sample data, the encoder therein is extracted, and the encoder is connected to the above-mentioned encoding network 420, and the original sub-encoding network for Cantonese recognition is removed, then an updated speech recognition model can be obtained. The updated speech recognition model has improved accuracy in Cantonese speech recognition, while the speech recognition performance for other languages remains unchanged.

[0087] When updating the speech recognition model in the above manner, the update of the encoding network 420 only needs to train a new sub-encoder with new sample data, without retraining the entire speech recognition model, which is easy to update.

[0088] Please refer to Figure 9 , Figure 9 is a structural schematic diagram of an embodiment of the speech recognition device of the present application. The speech recognition device 600 includes an acquisition module 610, an encoding module 620, a fusion module 630, and a decoding module 640.

[0089] Among them, the acquisition module 610 is used to acquire the speech information to be recognized; the encoding module 620 is used to respectively extract features of the speech information by using at least two sub-encoding networks in the encoding network to obtain at least two speech features, and each sub-encoding network is trained according to the sample data of the corresponding language; the fusion module 630 is used to fuse at least two speech features by using a fusion network to obtain a fused speech feature; the decoding module 640 is used to decode the fused speech feature by using a decoding network to obtain a speech recognition result.

[0090] Among them, the encoding module 620 is further used to respectively execute the following steps for each sub-encoding network: obtain the timing feature of the speech information, and perform single-direction and bi-directional feature extraction according to the timing feature to obtain a speech feature.

[0091] Among them, the fusion module 630 is further used to perform a first fusion operation on at least two speech features to obtain a first fusion feature; classify the language of the speech information based on the first fusion feature to obtain a language feature; perform a second fusion operation on the speech feature and the language feature to obtain a fused speech feature.

[0092] Among them, the fusion module 630 is further used to perform dimensionality reduction and splicing operations on at least two speech features by using a first fully connected layer network.

[0093] Among them, the fusion module 630 is further configured to determine the probabilities that the speech information is in the languages corresponding to the respective sub-encoding networks according to the language features; and perform a weighted fusion operation on the speech features output by the respective sub-encoding networks according to the probabilities to obtain the fused speech features.

[0094] Among them, the speech recognition device 600 further includes a training module (not shown in the figure). The training module is configured to train a preset language recognition model by using sample data in at least two languages respectively; extract the encoder part from the trained language recognition model respectively to be used as the sub-encoding networks; and connect the output layers of the sub-encoding networks to the input layer of the fusion network.

[0095] Among them, the training module is further configured to train the fusion network and the encoding network by using sample data in at least two languages with a certain degree of balance in data volume under the condition that the parameters of the sub-encoding networks are fixed.

[0096] For the descriptions of the functions and processes implemented by the respective functional modules of the speech recognition device 600 and the like, please refer to the descriptions of the corresponding steps in the speech recognition method embodiments of the present application above, and details are not described herein again.

[0097] Please refer to Figure 10 , Figure 10 which is a structural schematic diagram of another embodiment of the speech recognition device of the present application. The speech recognition device 200 includes a processor 210 and a memory 220 that are coupled to each other. A computer program is stored in the memory 220, and the processor 210 is configured to execute the computer program to implement the speech recognition method described in the above embodiments.

[0098] For the descriptions of the respective steps executed by the processor, please refer to the descriptions of the respective steps in the speech recognition method embodiments of the present application above, and details are not described herein again.

[0099] The memory 220 can be used to store program data and modules. The processor 210 executes various functional applications and data processing by running the program data and modules stored in the memory 220. The memory 220 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as speech feature extraction function, feature fusion function, feature decoding function, etc.); the data storage area can store data created according to the use of the speech recognition device 200 (such as speech information data, network parameters, feature vectors, etc.). In addition, the memory 220 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 220 may further include a memory controller to provide the processor 210 with access to the memory 220.

[0100] In various embodiments of the present application, the disclosed methods and apparatuses can be implemented in other ways. For example, the embodiments of the apparatuses described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0103] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product, and this computer software product is stored in a storage medium.

[0104] Refer to Figure 11 , Figure 11 , which is a structural schematic block diagram of an embodiment of the computer-readable storage medium of the present application. The computer-readable storage medium 300 stores program data 310. When the program data 310 is executed, the steps of the above-mentioned voice recognition method embodiments are implemented.

[0105] For the description of the steps executed by the processing, please refer to the description of the steps of the voice recognition method embodiments of the present application above, and details will not be repeated here.

[0106] The computer-readable storage medium 300 can be various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical discs.

[0107] The above are only the embodiments of the present application, and do not thus limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall similarly be included within the patent protection scope of the present application.

Claims

1. A speech recognition method, characterized in that, The method includes: Obtaining the speech information to be recognized; Using at least two sub-encoding networks in the encoding network to respectively extract features from the speech information, obtaining at least two speech features; wherein, each of the sub-encoding networks is trained according to the sample data of the corresponding language; Performing a first fusion operation on the at least two speech features to obtain a first fusion feature; Performing language classification on the speech information based on the first fusion feature to obtain a language feature; Multiplying each of the speech features by the language feature respectively to obtain corresponding weighted features; Summing up the weighted features and using a second fully-connected layer network to perform dimension expansion to obtain a fused speech feature; Using a decoding network to decode the fused speech feature to obtain a speech recognition result; Wherein, before using at least two sub-encoding networks in the encoding network to respectively extract features from the speech information, the method further includes: Using the sample data of at least two languages to respectively train a preset language recognition model; Respectively extracting the encoder part from the trained language recognition model as the sub-encoding network; Connecting the output layer of the sub-encoding network to the input layer of the fusion network, and training the fusion network and the encoding network using the sample data of at least two languages with the data volume meeting a certain balance degree while the parameters of the sub-encoding network are fixed.

2. The method according to claim 1, wherein The using at least two sub-encoding networks in the encoding network to respectively extract features from the speech information includes: Using each of the sub-encoding networks to respectively perform the following steps: Obtaining the temporal features of the speech information; Performing single-directional and bi-directional feature extraction according to the temporal features to obtain the speech features.

3. The method according to claim 1, wherein The performing a first fusion operation on the at least two speech features includes: Using a first fully-connected layer network to perform dimensionality reduction and splicing operations on the at least two speech features.

4. The method according to claim 1, wherein The data volume meeting a certain balance degree means that the difference in data volume between any two of the sample data of different languages is less than or equal to 20%.

5. A voice recognition device, characterized in that, The device includes: An obtaining module, configured to obtain the speech information to be recognized; An encoding module, configured to use at least two sub-encoding networks in the encoding network to respectively extract features from the speech information, obtaining at least two speech features; wherein, each of the sub-encoding networks is trained according to the sample data of the corresponding language; A fusion module, configured to perform a first fusion operation on the at least two speech features to obtain a first fusion feature; performing language classification on the speech information based on the first fusion feature to obtain a language feature; multiplying each of the speech features by the language feature respectively to obtain corresponding weighted features; summing up the weighted features and using a second fully-connected layer network to perform dimension expansion to obtain a fused speech feature; A decoding module, configured to use a decoding network to decode the fused speech feature to obtain a speech recognition result; A training module, configured to train a preset language recognition model respectively using sample data in at least two languages; extract the encoder part from the trained language recognition model respectively to be used as the sub-encoding network; connect the output layer of the sub-encoding network to the input layer of the fusion network, and train the fusion network and the encoding network using sample data in at least two languages with a certain degree of balance in data volume while the parameters of the sub-encoding network are fixed.

6. A voice recognition device, characterized in that, The device includes a processor and a memory coupled to each other; a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program data, and when the program data is executed by a processor, the steps of the method according to any one of claims 1-4 are implemented.

Images