Multilingual speech recognition method, device and storage medium

Abstract

The present disclosure relates to a multilingual speech recognition method, device and storage medium. The method comprises: obtaining audio to be recognized, the audio to be recognized includes audio in one or more languages; extracting acoustic information of the audio to be recognized, and performing feature extraction based on the acoustic information to determine language information; determining target text information based on the language information and multiple text information, wherein the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages; determining the audio recognition result of the audio to be recognized based on the acoustic information and the target text information. According to the embodiments of the present disclosure, the speech recognition effect of multiple languages ​​can be improved, and the recognition accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a multi-language speech recognition method, device and storage medium. Background Art

[0002] Speech recognition technology plays an important role in modern technology. For example, in human-computer interaction scenarios, speech recognition technology improves the way of human-computer interaction, enabling users to communicate with computer systems and smart devices using spoken language; in terms of improving accessibility, for the visually impaired or other physically impaired people, speech recognition technology is an important auxiliary tool that can improve their quality of life, etc.

[0003] In the field of speech recognition technology, there are scenarios for multilingual speech recognition, which requires recognizing spoken speech in multiple languages, not just a single language. The development of this technology is crucial to a globalized society and cross-cultural communication, and is a very important research direction. Current technical solutions have problems with multilingual recognition, such as poor recognition results and low recognition accuracy. Summary of the invention

[0004] In view of this, the present disclosure proposes a multi-language speech recognition method, device and storage medium.

[0005] According to one aspect of the present disclosure, a multilingual speech recognition method is provided. The method comprises:

[0006] Acquire audio to be recognized, where the audio to be recognized includes audio in one or more languages;

[0007] Extracting acoustic information of the audio to be recognized, and performing feature extraction based on the acoustic information to determine language information;

[0008] Determine target text information based on language information and multiple text information, wherein the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages;

[0009] Based on the acoustic information and the target text information, an audio recognition result of the audio to be recognized is determined.

[0010] In a possible implementation, determining the target text information based on the language information and the multiple text information includes:

[0011] Splicing multiple text information to obtain spliced ​​text information;

[0012] Based on the language information and the concatenated text information, the language features are calculated using the cross-attention mechanism, and the first scores corresponding to the languages ​​are determined based on the language features. The first scores represent the probability that the audio to be recognized in the current frame is of the language corresponding to the first score.

[0013] The target text information is determined based on the first scores corresponding to the languages ​​and the text information corresponding to the languages.

[0014] In one possible implementation, the method is used for a neural network model, which includes an encoder, a gating module, a plurality of prediction networks respectively associated with each language, a language recognition module, and a joint network;

[0015] The encoder is used to determine acoustic information, the gating module is used to determine target text information; multiple prediction networks associated with each language are used to determine multiple text information; the language recognition module is used to determine language information; the joint network is used to determine the audio recognition result based on the acoustic information and the target text information.

[0016] In a possible implementation, the method is applied to a trained neural network model, and the method further includes:

[0017] Determine a first audio training sample set, wherein each first audio training sample in the first audio training sample set includes audio and a label in the same language, and the first audio training set includes multiple languages;

[0018] For the current first audio training sample, perform the following training process:

[0019] Extracting acoustic information of the first audio training sample, and determining language information of the first audio training sample based on the acoustic information of the first audio training sample;

[0020] Determine text information associated with the language of the first audio training sample of the current frame based on the audio recognition result of the first audio training sample of the previous frame, and determine text information of the first audio training sample of the current frame associated with other languages ​​based on a preset value;

[0021] Determine a second score corresponding to each language based on the language information of the first audio training sample and the text information of the first audio training sample of each current frame, wherein the second score indicates the probability that the first audio training sample of the current frame is the language corresponding to the second score;

[0022] Determine a first audio training sample recognition result based on the acoustic information of the first audio training sample, the second score, and the text information of the first audio training sample of each current frame;

[0023] Training the initial neural network model based on the second scores corresponding to each language and the recognition results of the first audio training samples to obtain an initial neural network model for executing a training process for the next first audio training sample;

[0024] After completing the training process for each first audio training sample, an intermediate neural network model is obtained;

[0025] Training is performed based on the intermediate neural network model to obtain a trained neural network model.

[0026] In a possible implementation, training an initial neural network model based on the second scores corresponding to each language and the recognition results of the first audio training samples to obtain an initial neural network model for performing a training process for the next first audio training sample includes:

[0027] Calculating a first loss function value based on the second scores corresponding to each language and the labels of the first audio training samples;

[0028] Calculating a second loss function value based on the first audio training sample recognition result and the label of the first audio training sample;

[0029] Based on the first loss function value, the parameters of the encoder, each prediction network, language recognition module and gating module of the initial neural network model are adjusted; based on the second loss function value, the parameters of the encoder, each prediction network, language recognition module, gating module and joint network of the initial neural network model are adjusted; in response to the first loss function value and the second loss function value satisfying the first preset condition, the training is stopped and the initial neural network model for executing the training process for the next first audio training sample is obtained.

[0030] In a possible implementation, training is performed based on the intermediate neural network model to obtain a trained neural network model, including:

[0031] Based on the first audio training sample set, determining a second audio training sample set, wherein the second audio training sample set includes multi-language mixed audio and labels;

[0032] Extracting acoustic information of the second audio training sample in the second audio training sample set, and determining language information of the second audio training sample based on the acoustic information of the second audio training sample;

[0033] Determine text information of the second audio training sample of the current frame associated with each language based on the audio recognition result of the second audio training sample of the previous frame;

[0034] Determining a second audio training sample recognition result based on the acoustic information of the second audio training sample and the text information of the second audio training sample of each current frame;

[0035] Calculating a third loss function value based on the second audio training sample recognition result and the label of the second audio training sample;

[0036] The parameters of the encoder, language recognition module and gating module are fixed, and the parameters of each prediction network and the joint network of the intermediate neural network model are adjusted based on the third loss function value. In response to the third loss function value satisfying the second preset condition, the training is stopped and the trained neural network model is obtained.

[0037] In a possible implementation, determining the second audio training sample set based on the first audio training sample set includes:

[0038] The audio and labels of multiple languages ​​in the first audio training sample set are extracted according to a preset ratio, the audio and labels of multiple languages ​​are spliced ​​respectively, and the audio and labels of multiple languages ​​mixed in the second audio training sample set are determined.

[0039] In a possible implementation, determining the second audio training sample set based on the first audio training sample set includes:

[0040] Determine the boundaries of each word in the audio of the first audio training sample set, and extract some words as target words according to a preset ratio;

[0041] Translate the target word into a word text in another language, and replace the text corresponding to the target word in the label of the first audio training sample set with the translated word text;

[0042] An audio in another language is generated based on the translated word text, and the audio corresponding to the target word in the audio of the first audio training sample set is replaced with the generated audio in the other language.

[0043] In a possible implementation, the encoder, each prediction network and language recognition module in the initial neural network model are obtained through pre-training.

[0044] According to another aspect of the present disclosure, a multilingual speech recognition device is provided. The device comprises:

[0045] An acquisition module, used to acquire audio to be recognized, where the audio to be recognized includes audio in one or more languages;

[0046] A first determination module is used to extract acoustic information of the audio to be recognized, and perform feature extraction based on the acoustic information to determine language information;

[0047] A second determination module is used to determine target text information based on language information and multiple text information, wherein the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages;

[0048] The third determination module is used to determine the audio recognition result of the audio to be recognized based on the acoustic information and the target text information.

[0049] In a possible implementation manner, the second determining module is configured to:

[0050] Splicing multiple text information to obtain spliced ​​text information;

[0051] Based on the language information and the concatenated text information, the language features are calculated using the cross-attention mechanism, and the first scores corresponding to the languages ​​are determined based on the language features. The first scores represent the probability that the audio to be recognized in the current frame is of the language corresponding to the first score.

[0052] The target text information is determined based on the first scores corresponding to the languages ​​and the text information corresponding to the languages.

[0053] In one possible implementation, the device is used for a neural network model, which includes an encoder, a gating module, a plurality of prediction networks respectively associated with each language, a language recognition module, and a joint network;

[0054] The encoder is used to determine acoustic information, the gating module is used to determine target text information; multiple prediction networks associated with each language are used to determine multiple text information; the language recognition module is used to determine language information; the joint network is used to determine the audio recognition result based on the acoustic information and the target text information.

[0055] In a possible implementation, the device is used for a trained neural network model, and the device also includes:

[0056] A fourth determination module is used to determine a first audio training sample set, each first audio training sample in the first audio training sample set includes audio and a label in the same language, and the first audio training set includes multiple languages;

[0057] For the current first audio training sample, perform the following training process:

[0058] a fifth determination module, configured to extract acoustic information of the first audio training sample, and determine language information of the first audio training sample based on the acoustic information of the first audio training sample;

[0059] a sixth determination module, configured to determine text information associated with the language of the first audio training sample of the current frame based on the audio recognition result of the first audio training sample of the previous frame, and to determine text information of the first audio training sample of the current frame associated with other languages ​​based on a preset value;

[0060] A seventh determination module, configured to determine a second score corresponding to each language based on the language information of the first audio training sample and the text information of the first audio training sample of each current frame, wherein the second score indicates a probability that the first audio training sample of the current frame is of the language corresponding to the second score;

[0061] an eighth determination module, configured to determine a first audio training sample recognition result based on the acoustic information of the first audio training sample, the second score, and the text information of the first audio training sample of each current frame;

[0062] A ninth determination module, configured to train the initial neural network model based on the second scores corresponding to the languages ​​and the recognition results of the first audio training samples, to obtain an initial neural network model for executing a training process for the next first audio training sample;

[0063] A tenth determination module, configured to obtain an intermediate neural network model after completing the training process for each first audio training sample;

[0064] The training module is used to perform training based on the intermediate neural network model to obtain a trained neural network model.

[0065] In a possible implementation manner, the ninth determining module is configured to:

[0066] Calculating a first loss function value based on the second scores corresponding to each language and the labels of the first audio training samples;

[0067] Calculating a second loss function value based on the first audio training sample recognition result and the label of the first audio training sample;

[0068] Based on the first loss function value, the parameters of the encoder, each prediction network, language recognition module and gating module of the initial neural network model are adjusted; based on the second loss function value, the parameters of the encoder, each prediction network, language recognition module, gating module and joint network of the initial neural network model are adjusted; in response to the first loss function value and the second loss function value satisfying the first preset condition, the training is stopped and the initial neural network model for executing the training process for the next first audio training sample is obtained.

[0069] In a possible implementation, the training module is used to:

[0070] Based on the first audio training sample set, determining a second audio training sample set, wherein the second audio training sample set includes multi-language mixed audio and labels;

[0071] Extracting acoustic information of the second audio training sample in the second audio training sample set, and determining language information of the second audio training sample based on the acoustic information of the second audio training sample;

[0072] Determine text information of the second audio training sample of the current frame associated with each language based on the audio recognition result of the second audio training sample of the previous frame;

[0073] Determining a second audio training sample recognition result based on the acoustic information of the second audio training sample and the text information of the second audio training sample of each current frame;

[0074] Calculating a third loss function value based on the second audio training sample recognition result and the label of the second audio training sample;

[0075] The parameters of the encoder, language recognition module and gating module are fixed, and the parameters of each prediction network and the joint network of the intermediate neural network model are adjusted based on the third loss function value. In response to the third loss function value satisfying the second preset condition, the training is stopped and the trained neural network model is obtained.

[0076] In a possible implementation, determining the second audio training sample set based on the first audio training sample set includes:

[0077] The audio and labels of multiple languages ​​in the first audio training sample set are extracted according to a preset ratio, the audio and labels of multiple languages ​​are spliced ​​respectively, and the audio and labels of multiple languages ​​mixed in the second audio training sample set are determined.

[0078] In a possible implementation, determining the second audio training sample set based on the first audio training sample set includes:

[0079] Determine the boundaries of each word in the audio of the first audio training sample set, and extract some words as target words according to a preset ratio;

[0080] Translate the target word into a word text in another language, and replace the text corresponding to the target word in the label of the first audio training sample set with the translated word text;

[0081] An audio in another language is generated based on the translated word text, and the audio corresponding to the target word in the audio of the first audio training sample set is replaced with the generated audio in the other language.

[0082] In a possible implementation, the encoder, each prediction network and language recognition module in the initial neural network model are obtained through pre-training.

[0083] According to another aspect of the present disclosure, a multilingual speech recognition device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0084] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0085] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0086] According to the embodiment of the present application, by obtaining audio to be recognized in one or more languages, extracting acoustic information of the audio to be recognized and performing feature extraction based on the acoustic information to determine language information, determining target text information based on the language information and multiple text information, and determining the audio recognition result based on the acoustic information and the target text information, a multi-language speech recognition task can be achieved. And by determining multiple text information based on the audio recognition result of the previous frame of audio to be recognized, different text information corresponds to different languages, and the language information can be fully utilized to improve the robustness of recognition of different languages, effectively improve the multi-language speech recognition effect, and make the recognition accuracy higher.

[0087] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0089] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application.

[0090] Figure 2 A flow chart of a multilingual speech recognition method according to an embodiment of the present application is shown.

[0091] Figure 3 A structural diagram of a neural network model according to an embodiment of the present application is shown.

[0092] Figure 4 A structural diagram of a language recognition model according to an embodiment of the present application is shown.

[0093] Figure 5 A flow chart of a multilingual speech recognition method according to an embodiment of the present application is shown.

[0094] Figure 6 A flow chart of a multilingual speech recognition method according to an embodiment of the present application is shown.

[0095] Figure 7 A structural diagram of a multilingual speech recognition device according to an embodiment of the present application is shown.

[0096] Figure 8 is a block diagram of a device 1900 for multilingual speech recognition according to an exemplary embodiment. DETAILED DESCRIPTION

[0097] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0098] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0099] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0100] Speech recognition technology plays an important role in modern science and technology. For example, in human-computer interaction scenarios, speech recognition technology improves the way of human-computer interaction, enabling users to communicate with computer systems and smart devices using spoken language; in terms of improving accessibility, for the visually impaired or other physically disabled people, speech recognition technology is an important auxiliary tool that can improve their quality of life, and so on. In the field of speech recognition technology, there are scenarios for multilingual speech recognition, which requires the recognition of spoken speech in multiple languages, not just limited to a single language. The development of this technology is crucial to a globalized society and cross-cultural communication, and is a very important research direction. The current technical solutions are not robust enough in multilingual recognition, and the recognition effect of multiple languages ​​is poor.

[0101] In view of this, the present application provides a multilingual speech recognition method, device and storage medium. The method of the embodiment of the present application can achieve multilingual speech recognition tasks by acquiring audio to be recognized in one or more languages, extracting acoustic information of the audio to be recognized and performing feature extraction based on the acoustic information to determine the language information, determining the target text information based on the language information and multiple text information, and determining the audio recognition result based on the acoustic information and the target text information. And by determining multiple text information based on the audio recognition result of the previous frame of audio to be recognized, different text information corresponds to different languages, and the language information can be fully utilized to improve the robustness of recognition of different languages, effectively improve the multilingual speech recognition effect, and make the recognition accuracy higher.

[0102] Figure 1 A schematic diagram of an application scenario according to an embodiment of the present application is shown. The multilingual speech recognition system of the embodiment of the present application can be used in a human-computer interaction scenario of multilingual speech recognition, such as Figure 1 As shown, the user can make voice input. The multilingual speech recognition system of the embodiment of the present application can obtain the audio to be recognized input by the user and obtain the audio recognition result, that is, the text of the audio to be recognized. The text can be used for subsequent translation, intelligent customer service, command control and other tasks.

[0103] In the scenario of the embodiment of the present application, the audio input to be recognized by the user may be composed of one language or multiple languages. For example, it may be a mixed audio of Chinese and English: "hi, my name is Tom", and the multilingual speech recognition system may recognize the audio input by the user and output "hi, my name is Tom". The audio input to be recognized by the user may also include more languages.

[0104] The multilingual speech recognition system of the embodiment of the present application can be deployed in a terminal device or a server. The terminal device can be any one or more of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and a vehicle-mounted device. The embodiment of the present disclosure does not impose any special restrictions on the specific type of the terminal device, and the terminal device can have a wired or wireless communication function.

[0105] The server can be located locally or in the cloud, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. The wireless communication function can be realized, for example, through mobile communication technologies such as 2G / 3G / 4G / 5G, as well as Wi-Fi, Bluetooth, frequency modulation (FM), digital radio, satellite communication, etc. Communication can also be carried out through wired connection to achieve interaction with other devices.

[0106] Figure 2 FIG. 1 is a flow chart of a multilingual speech recognition method according to an embodiment of the present application. Figure 2 As shown, the method may include:

[0107] Step S201, obtaining audio to be recognized.

[0108] The audio to be recognized includes audio in one or more languages. For example, the audio to be recognized may be a voice message spoken by a user, including any one or more languages ​​among Chinese, English, Korean, and Japanese. The audio to be recognized may be audio input by the user, such as a mixed audio message in Chinese and English, such as "hi, my name is Tom".

[0109] Step S202: extracting acoustic information of the audio to be recognized, and performing feature extraction based on the acoustic information to determine language information.

[0110] Step S203: determining target text information based on the language information and the multiple text information.

[0111] Among them, the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages.

[0112] Step S204: determining an audio recognition result of the audio to be recognized based on the acoustic information and the target text information.

[0113] The method of the embodiment of the present application can be used in a neural network model. The audio to be recognized can be input into the neural network model of the embodiment of the present application to obtain an audio recognition result. The present disclosure does not limit the structure of the neural network model.

[0114] For ease of understanding, we first introduce the exemplary structure of the neural network model, see Figure 3, showing a structural diagram of a neural network model according to an embodiment of the present application. The neural network model of the embodiment of the present application can combine the structure of RNN-Transducer (recurrent neural network-transducer, RNN-T) and a mixture of experts (MOE), and combine language identification (LID) with a gating structure, so that the output of MOE can be more robust for different languages.

[0115] like Figure 3 As shown, the neural network model of the embodiment of the present application may include an encoder (Encoder), a plurality of prediction networks 1 to 4 (Predictor_1 to 4, corresponding to 4 languages ​​respectively, and may also include a greater or lesser number of prediction networks) associated with each language, a language identification module (LID embedding), a gating module (GATE) and a joint network (Jointer). The structure of the above-mentioned RNN-T is embodied as a combination of an encoder, a prediction network and a joint network, and the combination of the RNN-T and MOE structures is embodied as making the model include a plurality of prediction networks associated with each language respectively. Language identification is embodied as a language identification module, and the gating structure is embodied as a gating module.

[0116] Among them, the encoder can be used to process audio (such as wav in the figure) to determine acoustic information; the prediction networks associated with each language can be used to determine multiple text information of the current frame audio (such as P-1 to 4 in the figure) based on the audio recognition result corresponding to the previous frame audio (that is, the text of the previous frame audio, such as text in the figure), and different text information can correspond to different languages ​​respectively; the language recognition module can be used to perform feature extraction based on acoustic information to determine the language information of the audio; the gating module can be used to determine the target text information based on the text information output by multiple prediction networks and the language information output by the language recognition module; the joint network can be used to determine the text corresponding to the current frame audio based on the acoustic information output by the encoder and the target text information output by the gating module, such as the text corresponding to a letter or Chinese character in "hi, my name is tom", and the texts corresponding to each frame are summarized to obtain the audio recognition result (such as "hi, my name is tom" in the figure).

[0117] The method of the embodiment of the present application can be used for a trained neural network model, and the trained neural network model can be obtained by training the initial neural network model. The training process of the embodiment of the present application is first introduced below.

[0118] The training process is divided into a pre-training phase and a formal training phase. The formal training phase can include two phases. In the first training phase, the neural network model can be trained using the training sample sets of unmixed languages; and in the second training phase, the neural network model trained in the first phase can be trained using the training sample sets of mixed languages. In this way, the training data of different languages ​​can be more fully utilized to improve the training effect of the model and obtain a neural network model with higher accuracy.

[0119] In the first training stage, the gating module, encoder, and language recognition module of the neural network model are mainly trained; in the second training stage, the parameters of the gating module, encoder, and language recognition module of the neural network model will be fixed, and each prediction network and the joint network will be trained.

[0120] In the first training stage, two loss functions are included, and the two loss function values ​​are respectively used to adjust the parameters of the encoder, each prediction network, language recognition module and gating module of the neural network model, and are used to adjust the parameters of the encoder, each prediction network, language recognition module, gating module and joint network of the neural network model; the loss function in the second training stage is different from the loss function in the first training stage, and the loss function value in the second training stage is used to adjust the parameters of each prediction network and the joint network of the neural network model.

[0121] The first is the pre-training phase:

[0122] Among them, the encoder, each prediction network and language recognition module in the initial neural network model can be obtained through pre-training.

[0123] Unlabeled audio data can be obtained, which can be in multiple languages. Unsupervised pre-training is performed based on models such as wav2vec2.0 to obtain a feature extractor for extracting acoustic information. The parameters of the feature extractor can be set to the encoder in the initial neural network model.

[0124] Each prediction network can be pre-trained based on the structure of a recurrent neural network language model (RNNLM). One RNNLM can be trained for each language, and each RNNLM is trained using texts of a single language. For example, for language i, RNNLM_i (representing the RNNLM model corresponding to language i) can be trained using the text of language i. The pre-training method can be implemented based on existing technologies, so that the text at time 0-t is input, and the pre-trained RNNLM model can output the text at time t+1. The parameters of the pre-trained RNNLM_1~N (N represents the number of languages) can be set to the prediction network of the corresponding language in the initial neural network model.

[0125] The encoder can be used to extract acoustic features from the training data of each language (including audio and labels of unmixed languages) to obtain the corresponding acoustic information (embedding). The language recognition model is pre-trained using the embedding and the input data and labels of the corresponding language type training. The structure of the language recognition model can be found in Figure 4 , showing a schematic diagram of the structure of a language recognition model according to an embodiment of the present application. Figure 4 As shown, the language recognition model is a classification model, which may include: a conformer, attention / max pooling, and a softmax module. The conformer module is used to extract features from embedding to obtain language information (which may be called lid_embed). The attention / max pooling module is used to process lid_embed based on the attention mechanism or the maximum pooling (max pooling). The output result is normalized by the softmax module to output the language type corresponding to the embedding (such as language in the figure). The parameters of the conformer module in the pre-trained language recognition model can be set to the language recognition module in the initial neural network model.

[0126] Thus, an initial neural network model can be obtained. The pre-training stage can save the overall model training time and make the model obtained by subsequent training more accurate.

[0127] The first training phase of the formal training phase can be found in Figure 5 , shows a flow chart of a multilingual speech recognition method according to an embodiment of the present application. Figure 5 As shown, the method may also include:

[0128] Step S501: determine a first audio training sample set.

[0129] The first audio training sample set includes audio and labels in multiple languages ​​and not mixed, that is, each first audio training sample in the first audio training sample set includes audio and labels in the same language, and the first audio training sample set includes multiple languages. The label can represent the text corresponding to the audio, and the label can also include an identifier for indicating the language, and the language can also be determined based on the text.

[0130] For example, if you want the model to be able to recognize audio in four languages ​​(for example, Chinese, English, Korean, and Japanese) and mixed audio in multiple languages, you can collect the audio and corresponding text in these four languages ​​separately, and obtain four first audio training samples corresponding to the four languages: {REAL, Chinese}, {REAL, English}, {REAL, Korean}, and {REAL, Japanese} as the first audio training sample set. REAL means that the data in the data set is real data, and the content expressed can be the same, but the language is different. The first audio training sample can be related to the business scenario. The following is an introduction using multilingual including four languages ​​as an example.

[0131] For the current first audio training sample, perform the following training process:

[0132] Step S502: extracting acoustic information of the first audio training sample, and determining language information of the first audio training sample based on the acoustic information of the first audio training sample.

[0133] For example, the audio in {REAL, i} can be input into the encoder, and the encoder can be used to extract its acoustic information, where i is used to indicate the corresponding language. The acoustic information is the above-mentioned embedding, and its dimension is (T, dim), where T is a preset value (e.g., 10), and dim is a preset parameter for controlling the model size (e.g., 128).

[0134] The acoustic information of the first audio training sample can be input into the language recognition module to obtain the language information of the first audio training sample output by the language recognition module (ie, the above lid_embed), whose dimension is (T, dim).

[0135] Step S503, determining text information associated with the language of the first audio training sample of the current frame based on the audio recognition result of the first audio training sample of the previous frame, and determining text information of the first audio training sample of the current frame associated with other languages ​​based on a preset value.

[0136] The preset value may be a fake value (for example, all 0s). For example, when the audio in {REAL, i} is input into the encoder, the audio recognition result of the first audio training sample in the previous frame may be input into the prediction network corresponding to language i (i.e., predictor_i), and 0 may be input into other prediction networks. When the current frame is the first frame of audio, a preset identifier (such as SOS (start of sentence)) may be input into the predictor_i corresponding to language i. The audio recognition result of the first audio training sample in the previous frame is the output of the joint network.

[0137] Taking 4 languages ​​as an example, each prediction network can output corresponding text information P-1 to 4 respectively. Each text information can represent relevant information of the corresponding language, and the dimension of each text information is (1, dim).

[0138] Step S504: determining a second score corresponding to each language based on the language information of the first audio training sample and the text information of each current frame first audio training sample.

[0139] See also Figure 3 , the text information of the first audio training sample of the current frame output by each prediction network can also be concatenated (concat) to obtain concatenated text information (which can be called P), whose dimension is (4, dim).

[0140] See also Figure 3 Based on the language information (lid_embed) output by the language recognition module and the concatenated text information (referred to as P), the language feature P_L can be calculated using the cross attention mechanism. Based on the language feature P_L, multilingual speech recognition can be achieved.

[0141] Among them, the dimension of P_L is (4, dim), and the second score can be obtained by normalizing P_L into 4 categories (softmax).

[0142] Each language has a corresponding second score, which can represent the probability that the first audio training sample of the current frame is the language corresponding to the second score. The higher the second score, the greater the probability that the first audio training sample of the current frame is the language corresponding to the second score.

[0143] Step S505 : determining a first audio training sample recognition result based on the acoustic information of the first audio training sample, the second score, and the text information of the first audio training sample of each current frame.

[0144] Among them, the second scores corresponding to each language can be input into the gating module, and the gating module can weight each text information based on the second scores corresponding to each language and output the target text information. The gating module can also output the text information determined by the prediction network associated with the language with the highest second score as the target text information. The embedding output by the encoder and the target text information output by the gating module can be input into the joint network for decoding to determine the first audio training sample recognition result. The first audio training sample recognition result represents the text corresponding to the recognized input audio.

[0145] Step S506: training the initial neural network model based on the second scores corresponding to the languages ​​and the recognition results of the first audio training samples to obtain an initial neural network model for executing a training process for the next first audio training sample.

[0146] Among them, you can:

[0147] Based on the second scores corresponding to each language and the label of the first audio training sample, a first loss function value is calculated; based on the recognition result of the first audio training sample and the label of the first audio training sample, a second loss function value is calculated; based on the first loss function value, the parameters of the encoder, each prediction network, the language recognition module and the gating module of the initial neural network model are adjusted, and based on the second loss function value, the parameters of the encoder, each prediction network, the language recognition module, the gating module and the joint network of the initial neural network model are adjusted; in response to the first loss function value and the second loss function value satisfying the first preset condition, the training is stopped and the initial neural network model for executing the training process for the next first audio training sample is obtained.

[0148] The first loss function value (gate_loss) may be a cross entropy loss function value of classification, and when calculating the first loss function value, the label of the first audio training sample may be an identifier for indicating the language. The second loss function value (rnnt_loss) may be a cross entropy, CTC or other loss function value, and when calculating the second loss function value, the label of the first audio training sample is the text corresponding to the audio.

[0149] The first preset condition may be that the weighted sum of the first loss function value and the second loss function value reaches a preset threshold, thereby obtaining an initial neural network model for executing a training process for the next first audio training sample.

[0150] Step S507: After completing the training process for each first audio training sample, an intermediate neural network model is obtained.

[0151] Therefore, the intermediate neural network model can have the ability to select different prediction networks for language recognition based on audio in different languages.

[0152] This training process is the process of steps S502 - S506. Taking the first audio training samples in four languages as an example, {REAL, Chinese}, {REAL, English}, {REAL, Korean}, and {REAL, Japanese}, steps S502 - S506 can be sequentially executed on the four first audio training samples in any order. After all 4 first audio training samples are executed, an intermediate neural network model is obtained.

[0153] Step S508, based on the intermediate neural network model for training, a trained neural network model is obtained.

[0154] This process can correspond to the second training stage in the formal training phase. Refer to Figure 6 , which shows a flowchart of a multilingual speech recognition method according to an embodiment of the present application. As Figure 6 shown, this step S508 may include:

[0155] Step S601, based on the first audio training sample set, determine the second audio training sample set.

[0156] The first audio training sample set can be expanded to obtain the second audio training sample set. The second audio training sample set includes the audio and labels after mixing multiple languages. The audio in the second audio training sample set can include an audio with only a single language or an audio with multiple languages.

[0157] Among them, it is possible to: extract the audio and labels of multiple languages in the first audio training sample set according to a preset ratio, splice the audio and labels of multiple languages respectively, and determine the audio and labels after mixing multiple languages in the second audio training sample set.

[0158] For example, it is possible to randomly extract {wav_Chinese_1, text_Chinese_1} (indicating a segment of Chinese audio wav_Chinese_1 and the corresponding text text_Chinese_1 in the first audio training sample of Chinese) from {REAL, Chinese} in the first audio training sample set, and randomly extract {wav_English_1, text_English_1} (indicating a segment of English audio wav_English_1 and the corresponding text text_English_1 in the first audio training sample of English) from {REAL, English}. {wav_Chinese_1} can be spliced with {wav_English_1}, and correspondingly {text_Chinese_1} can be spliced with {text_English_1}, and {wav_Chinese - English_1, text_Chinese - English_1} can be obtained, which can be used as the content of the second audio training sample in the second audio training sample set. The first audio training sample set can be expanded based on this method to obtain the second audio training sample set.

[0159] It is also possible to:

[0160] Determine the boundaries of each word in the audio of the first audio training sample set, and extract some words as target words according to a preset ratio; translate the target words into word texts in other languages, and replace the text corresponding to the target words in the labels of the first audio training sample set with the translated word texts; generate audio in other languages ​​based on the translated word texts, and replace the audio corresponding to the target words in the audio of the first audio training sample set with the generated audio in other languages.

[0161] Among them, speech segmentation or speech recognition technology can be used to determine the boundaries of each word in the audio to extract some words as target words, and the target words in the label text in {REAL, i} can be determined based on methods such as named entity recognition, and the target words can be translated and replaced with words in other languages. Based on the position of the target words in the audio, the audio corresponding to the target words can be replaced with the corresponding audio in other languages ​​using methods such as speech synthesis. The first audio training sample set is expanded to obtain the second audio training sample set.

[0162] The training data in multiple languages ​​mixed obtained by the above expansion can be called FAKE data, the first audio training sample can be called REAL data, and the second audio training sample can include REAL data and FAKE data.

[0163] Thereby, the cost of collecting training data can be saved and the model recognition accuracy can be improved.

[0164] Step S602: extracting acoustic information of the second audio training samples in the second audio training sample set, and determining language information of the second audio training samples based on the acoustic information of the second audio training samples.

[0165] The acoustic information may refer to embedding in S502 , and the language information may refer to lid_embed in S502 .

[0166] Step S603: determining text information of the second audio training sample of the current frame associated with each language based on the audio recognition result of the second audio training sample of the previous frame.

[0167] The audio recognition result of the second audio training sample of the previous frame can be input into each prediction network respectively, and each prediction network outputs the text information of the second audio training sample of the current frame associated with each language respectively.

[0168] At this time, the preset value is no longer input into the prediction network. When the current frame is the first frame of audio, the preset identifier (such as SOS (start of sentence)) can be input into predictor_i corresponding to language i. The audio recognition result of the previous frame of the second audio training sample can be the output of the joint network.

[0169] The text information of the second audio training sample of the current frame output by each prediction network can be referred to P-1 to P-4 in step S503.

[0170] Step S604: determining a second audio training sample recognition result based on the acoustic information of the second audio training sample and the text information of the second audio training sample of each current frame.

[0171] Among them, the acoustic information of the second audio training sample can be input into the language recognition module to obtain the language information output by the language recognition module (see the above lid_embed); the text information of the second audio training sample of the current frame output by each prediction network can be spliced ​​to obtain the spliced ​​text information; based on the language information output by the language recognition module and the spliced ​​text information, the language features can be calculated using the cross-attention mechanism (see the above P_L), and after performing 4-category normalization processing on P_L, a score representing the probability that the second audio training sample of the current frame is the corresponding language can be obtained.

[0172] The scores corresponding to each language can be input into the gating module, and the gating module can weight the text information of the second audio training sample of each current frame based on the scores corresponding to each language and then output the target text information. The gating module can also determine the text information of the second audio training sample of the current frame as the target text information output by the prediction network associated with the language with the highest corresponding score. The embedding output by the encoder in step S602 and the target text information output by the gating module can be input into the joint network for decoding to determine the recognition result of the second audio training sample. The second training audio recognition result represents the text corresponding to the recognized input audio.

[0173] Step S605: Calculate a third loss function value based on the second audio training sample recognition result and the label of the second audio training sample.

[0174] The calculation method of the third loss function value can refer to the calculation method of the second loss function value mentioned above.

[0175] Step S606, fix the parameters of the encoder, language recognition module and gating module, adjust the parameters of each prediction network and the joint network of the intermediate neural network model based on the third loss function value, and in response to the third loss function value satisfying the second preset condition, stop training and obtain the trained neural network model.

[0176] The second preset condition may be that the value of the third loss function reaches a preset threshold. At this time, the parameters of the modules in the intermediate neural network model except for the prediction networks and the joint network are fixed, and only the parameters of the prediction networks and the joint network are adjusted. Thus, the model can be trained to have the modeling ability for mixed languages ​​on the one hand, while the joint network and the prediction network can use information from different languages ​​to improve recognition accuracy. The training method of the embodiment of the present application can make the recognition accuracy of the trained neural network model higher.

[0177] Steps S602 to S606 may be performed sequentially on each second audio training sample in the second audio training sample set until the second preset condition is met.

[0178] The trained neural network model can be deployed for inference. Figure 2 :

[0179] In step S202, the audio to be recognized may be input into an encoder to extract acoustic information of the audio to be recognized; and the acoustic information of the audio to be recognized in the current frame may be input into a language recognition module for feature extraction to obtain language information.

[0180] The acoustic information of the audio to be recognized can be found in the above embedding. The language information can be found in the above lid_embed.

[0181] Step S203: determining target text information based on the language information and the multiple text information.

[0182] Among them, the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages.

[0183] The audio recognition result of the previous frame of audio to be recognized can be input into each prediction network respectively to determine the text information associated with each language output by each prediction network. The multiple text information can refer to P-1 to P-4 in step S603.

[0184] The step S203 may include:

[0185] Splicing multiple text information to obtain spliced ​​text information;

[0186] Based on the language information (lid_embed) and the concatenated text information (which can be called P), the cross-attention mechanism is used to calculate the language features, and multi-language speech recognition can be realized, wherein the first score corresponding to each language can be determined based on the language features, and the first score represents the probability that the audio to be recognized in the current frame is the language corresponding to the first score;

[0187] The target text information is determined based on the first scores corresponding to the languages ​​and the text information corresponding to the languages.

[0188] Among them, the language features can refer to the above P_L. When the number of prediction networks is N, the language features can be normalized for N categories to obtain a first score, and the first score can represent the probability that the audio to be recognized in the current frame is the language corresponding to the first score. The first score corresponding to each language can be input into the gating module, and the gating module can weight each text information based on the first score corresponding to each language and output the target text information. The gating module can also output the text information determined by the prediction network associated with the language with the highest first score as the target text information.

[0189] In step S204, an audio recognition result of the audio to be recognized may be determined based on the acoustic information and the target text information.

[0190] The acoustic information of the audio to be recognized in the current frame and the target text information output by the gating module can be input into the joint network to determine the audio recognition result, which can represent the text of the recognized audio to be recognized.

[0191] According to the embodiment of the present application, by obtaining audio to be recognized in one or more languages, extracting acoustic information of the audio to be recognized and performing feature extraction based on the acoustic information to determine language information, determining target text information based on the language information and multiple text information, and determining the audio recognition result based on the acoustic information and the target text information, a multi-language speech recognition task can be achieved. And by determining multiple text information based on the audio recognition result of the previous frame of audio to be recognized, different text information corresponds to different languages, and the language information can be fully utilized to improve the robustness of recognition of different languages, effectively improve the multi-language speech recognition effect, and make the recognition accuracy higher.

[0192] Figure 7 FIG. 2 shows a structural diagram of a multilingual speech recognition device according to an embodiment of the present application. Figure 7 As shown, the device comprises:

[0193] An acquisition module 701 is used to acquire audio to be recognized, where the audio to be recognized includes audio in one or more languages;

[0194] A first determination module 702 is used to extract acoustic information of the audio to be recognized, and perform feature extraction based on the acoustic information to determine language information;

[0195] A second determination module 703 is used to determine target text information based on language information and multiple text information, wherein the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages;

[0196] The third determination module 704 is used to determine the audio recognition result of the audio to be recognized based on the acoustic information and the target text information.

[0197] In a possible implementation, the second determining module 703 is configured to:

[0198] Splicing multiple text information to obtain spliced ​​text information;

[0199] Based on the language information and the concatenated text information, the language features are calculated using the cross-attention mechanism, and the first scores corresponding to the languages ​​are determined based on the language features. The first scores represent the probability that the audio to be recognized in the current frame is of the language corresponding to the first score.

[0200] The target text information is determined based on the first scores corresponding to the languages ​​and the text information corresponding to the languages.

[0201] In one possible implementation, the device is used for a neural network model, which includes an encoder, a gating module, a plurality of prediction networks respectively associated with each language, a language recognition module, and a joint network;

[0202] The encoder is used to determine acoustic information, the gating module is used to determine target text information; multiple prediction networks associated with each language are used to determine multiple text information; the language recognition module is used to determine language information; the joint network is used to determine the audio recognition result based on the acoustic information and the target text information.

[0203] In a possible implementation, the device is used for a trained neural network model, and the device also includes:

[0204] A fourth determination module is used to determine a first audio training sample set, each first audio training sample in the first audio training sample set includes audio and a label in the same language, and the first audio training set includes multiple languages;

[0205] For the current first audio training sample, perform the following training process:

[0206] a fifth determination module, configured to extract acoustic information of the first audio training sample, and determine language information of the first audio training sample based on the acoustic information of the first audio training sample;

[0207] a sixth determination module, configured to determine text information associated with the language of the first audio training sample of the current frame based on the audio recognition result of the first audio training sample of the previous frame, and to determine text information of the first audio training sample of the current frame associated with other languages ​​based on a preset value;

[0208] A seventh determination module, configured to determine a second score corresponding to each language based on the language information of the first audio training sample and the text information of the first audio training sample of each current frame, wherein the second score indicates a probability that the first audio training sample of the current frame is of the language corresponding to the second score;

[0209] an eighth determination module, configured to determine a first audio training sample recognition result based on the acoustic information of the first audio training sample, the second score, and the text information of the first audio training sample of each current frame;

[0210] A ninth determination module, configured to train the initial neural network model based on the second scores corresponding to the languages ​​and the recognition results of the first audio training samples, to obtain an initial neural network model for executing a training process for the next first audio training sample;

[0211] A tenth determination module, configured to obtain an intermediate neural network model after completing the training process for each first audio training sample;

[0212] The training module is used to perform training based on the intermediate neural network model to obtain a trained neural network model.

[0213] In a possible implementation manner, the ninth determining module is configured to:

[0214] Calculating a first loss function value based on the second scores corresponding to each language and the labels of the first audio training samples;

[0215] Calculating a second loss function value based on the first audio training sample recognition result and the label of the first audio training sample;

[0216] Based on the first loss function value, the parameters of the encoder, each prediction network, language recognition module and gating module of the initial neural network model are adjusted; based on the second loss function value, the parameters of the encoder, each prediction network, language recognition module, gating module and joint network of the initial neural network model are adjusted; in response to the first loss function value and the second loss function value satisfying the first preset condition, the training is stopped and the initial neural network model for executing the training process for the next first audio training sample is obtained.

[0217] In a possible implementation, the training module is used to:

[0218] Based on the first audio training sample set, determining a second audio training sample set, wherein the second audio training sample set includes multi-language mixed audio and labels;

[0219] Extracting acoustic information of the second audio training sample in the second audio training sample set, and determining language information of the second audio training sample based on the acoustic information of the second audio training sample;

[0220] Determine text information of the second audio training sample of the current frame associated with each language based on the audio recognition result of the second audio training sample of the previous frame;

[0221] Determining a second audio training sample recognition result based on the acoustic information of the second audio training sample and the text information of the second audio training sample of each current frame;

[0222] Calculating a third loss function value based on the second audio training sample recognition result and the label of the second audio training sample;

[0223] The parameters of the encoder, language recognition module and gating module are fixed, and the parameters of each prediction network and the joint network of the intermediate neural network model are adjusted based on the third loss function value. In response to the third loss function value satisfying the second preset condition, the training is stopped and the trained neural network model is obtained.

[0224] In a possible implementation, determining the second audio training sample set based on the first audio training sample set includes:

[0225] The audio and labels of multiple languages ​​in the first audio training sample set are extracted according to a preset ratio, the audio and labels of multiple languages ​​are spliced ​​respectively, and the audio and labels of multiple languages ​​mixed in the second audio training sample set are determined.

[0226] In a possible implementation, determining the second audio training sample set based on the first audio training sample set includes:

[0227] Determine the boundaries of each word in the audio of the first audio training sample set, and extract some words as target words according to a preset ratio;

[0228] Translate the target word into a word text in another language, and replace the text corresponding to the target word in the label of the first audio training sample set with the translated word text;

[0229] An audio in another language is generated based on the translated word text, and the audio corresponding to the target word in the audio of the first audio training sample set is replaced with the generated audio in the other language.

[0230] In a possible implementation, the encoder, each prediction network and language recognition module in the initial neural network model are obtained through pre-training.

[0231] According to the embodiment of the present application, by obtaining audio to be recognized in one or more languages, extracting acoustic information of the audio to be recognized and performing feature extraction based on the acoustic information to determine language information, determining target text information based on the language information and multiple text information, and determining the audio recognition result based on the acoustic information and the target text information, a multi-language speech recognition task can be achieved. And by determining multiple text information based on the audio recognition result of the previous frame of audio to be recognized, different text information corresponds to different languages, and the language information can be fully utilized to improve the robustness of recognition of different languages, effectively improve the multi-language speech recognition effect, and make the recognition accuracy higher.

[0232] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0233] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0234] The embodiment of the present disclosure also proposes a multilingual speech recognition device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0235] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0236] Figure 8 1 is a block diagram of a device 1900 for multilingual speech recognition according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Figure 8 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0237] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0238] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0239] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0240] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0241] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0242] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0243] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0244] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0245] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0246] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0247] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multilingual speech recognition method, characterized in that: The method comprises: Acquire audio to be recognized, where the audio to be recognized includes audio in one or more languages; Extracting acoustic information of the audio to be recognized, and performing feature extraction based on the acoustic information to determine language information; Based on the language information and the multiple text information, target text information is determined, wherein the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages; the multiple text information is determined by prediction networks associated with each language, respectively, and the target text information is determined based on the text information corresponding to each language and the probability that the current frame of audio to be recognized corresponding to each language is the corresponding language; Based on the acoustic information and the target text information, an audio recognition result of the audio to be recognized is determined.

2. The method according to claim 1, characterized in that The step of determining target text information based on the language information and the plurality of text information includes: Splicing multiple text information to obtain spliced ​​text information; Based on the language information and the concatenated text information, a cross-attention mechanism is used to calculate language features, and a first score corresponding to each language is determined based on the language features, wherein the first score represents a probability that the audio to be recognized in the current frame is of the language corresponding to the first score; The target text information is determined based on the first scores corresponding to the languages ​​and the text information corresponding to the languages.

3. The method according to claim 2, characterized in that The method is used for a neural network model, which includes an encoder, a gating module, a plurality of prediction networks respectively associated with each language, a language recognition module, and a joint network; The encoder is used to determine the acoustic information, the gating module is used to determine the target text information; the multiple prediction networks respectively associated with each language are used to determine multiple text information; the language recognition module is used to determine the language information; the joint network is used to determine the audio recognition result based on the acoustic information and the target text information.

4. The method according to claim 1, characterized in that: The method is used for a trained neural network model, and the method further comprises: Determine a first audio training sample set, wherein each first audio training sample in the first audio training sample set includes audio and a label in the same language, and the first audio training set includes multiple languages; For the current first audio training sample, perform the following training process: extracting acoustic information of the first audio training sample, and determining language information of the first audio training sample based on the acoustic information of the first audio training sample; Determine text information associated with the language of the first audio training sample of the current frame based on the audio recognition result of the first audio training sample of the previous frame, and determine text information of the first audio training sample of the current frame associated with other languages ​​based on a preset value; Determine, based on the language information of the first audio training sample and the text information of the first audio training sample of each current frame, a second score corresponding to each language, wherein the second score indicates the probability that the first audio training sample of the current frame is the language corresponding to the second score; Determine a first audio training sample recognition result based on the acoustic information of the first audio training sample, the second score, and text information of the first audio training sample of each current frame; Training the initial neural network model based on the second scores corresponding to each language and the recognition result of the first audio training sample to obtain an initial neural network model for executing the training process for the next first audio training sample; After completing the training process for each first audio training sample, an intermediate neural network model is obtained; Training is performed based on the intermediate neural network model to obtain a trained neural network model.

5. The method according to claim 4, characterized in that The step of training the initial neural network model based on the second scores corresponding to the languages ​​and the recognition result of the first audio training sample to obtain the initial neural network model for executing the training process for the next first audio training sample includes: Calculating a first loss function value based on the second scores corresponding to each language and the label of the first audio training sample; Calculating a second loss function value based on the first audio training sample recognition result and the label of the first audio training sample; Based on the first loss function value, the parameters of the encoder, each prediction network, language recognition module and gating module of the initial neural network model are adjusted; based on the second loss function value, the parameters of the encoder, each prediction network, language recognition module, gating module and joint network of the initial neural network model are adjusted; in response to the first loss function value and the second loss function value satisfying a first preset condition, the training is stopped and the initial neural network model for executing the training process for the next first audio training sample is obtained.

6. The method according to claim 4, characterized in that The training is performed based on the intermediate neural network model to obtain a trained neural network model, including: Determine a second audio training sample set based on the first audio training sample set, wherein the second audio training sample set includes multi-language mixed audio and labels; Extracting acoustic information of the second audio training sample in the second audio training sample set, and determining language information of the second audio training sample based on the acoustic information of the second audio training sample; Determine text information of the second audio training sample of the current frame associated with each language based on the audio recognition result of the second audio training sample of the previous frame; Determining a second audio training sample recognition result based on the acoustic information of the second audio training sample and the text information of the second audio training sample of each current frame; Calculating a third loss function value based on the second audio training sample recognition result and the label of the second audio training sample; The parameters of the encoder, language recognition module and gating module are fixed, and the parameters of each prediction network and the joint network of the intermediate neural network model are adjusted based on the third loss function value. In response to the third loss function value satisfying the second preset condition, the training is stopped and the trained neural network model is obtained.

7. The method according to claim 6, characterized in that Determining a second audio training sample set based on the first audio training sample set includes: The audio and labels in multiple languages ​​are extracted from the first audio training sample set according to a preset ratio, and the audio and labels in multiple languages ​​are respectively spliced ​​to determine the audio and labels of multiple languages ​​mixed in the second audio training sample set.

8. The method according to claim 7, characterized in that Determining a second audio training sample set based on the first audio training sample set includes: Determine the boundaries of each word in the audio of the first audio training sample set, and extract some words as target words according to a preset ratio; Translate the target word into a word text in another language, and replace the text corresponding to the target word in the label of the first audio training sample set with the translated word text; An audio in another language is generated based on the translated word text, and the audio corresponding to the target word in the audio of the first audio training sample set is replaced with the generated audio in the other language.

9. The method according to claim 4, characterized in that The encoder, prediction networks and language recognition module in the initial neural network model are obtained through pre-training.

10. A multilingual speech recognition device, characterized in that: The device comprises: An acquisition module, used to acquire audio to be recognized, wherein the audio to be recognized includes audio in one or more languages; A first determination module, used to extract acoustic information of the audio to be recognized, and perform feature extraction based on the acoustic information to determine language information; A second determination module is used to determine target text information based on the language information and multiple text information, wherein the multiple text information is determined based on the audio recognition result of the previous frame of audio to be recognized, and different text information corresponds to different languages; the multiple text information is determined by prediction networks associated with each language respectively, and the target text information is determined based on the text information corresponding to each language and the probability that the current frame of audio to be recognized corresponding to each language is the corresponding language; The third determination module is used to determine the audio recognition result of the audio to be recognized based on the acoustic information and the target text information.

11. A multilingual speech recognition device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images