Multilingual speech recognition model training method and related device

Abstract

The invention discloses a multilingual speech recognition model training method and a related device, and relates to the technical field of speech recognition, and the method comprises the steps: constructing a multilingual speech recognition model training model comprising a first training branch and a second training branch in advance, the first training branch and the second training branch are each composed of an encoder and a decoder, the first training branch and the second training branch share a decoder and a partial encoder; and performing multi-task joint training on the first training branch and the second training branch through the voice text supervised training data and the plain text training data to obtain the trained first training branch as a multilingual voice recognition model. According to the scheme, the calculation overhead of model training can be reduced, and the utilization rate of the training data is improved, so that an efficient and accurate multilingual speech recognition model is obtained through training under the condition that the training period is effectively shortened and supervised training data is reduced, and the multilingual speech recognition effect is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a multilingual speech recognition model training method and related devices. Background Art

[0002] Automatic Speech Recognition (ASR) technology can extract text information from spoken language and convert it into corresponding text. Speech recognition technology is essential to connect human language with computer processing capabilities. With the popularity of mobile devices, wearable technology, smart homes, and in-car infotainment systems, speech recognition technology plays an increasingly important role in many application areas. In these devices and systems, voice interaction is a more natural and convenient way of interaction that most people have mastered, making it a more superior usage mode on these devices and systems.

[0003] With the rapid development of deep learning technology and the continuous improvement of computer hardware performance, the recognition effect of speech recognition systems has reached a good level in common languages ​​such as Chinese and English, and has been applied in many smart devices. However, there are more than 7,000 languages ​​in the world, and most of them lack available speech recognition systems. There are large differences in writing systems, pronunciation patterns, transcription methods, data resources, etc. between different languages, which makes it a huge challenge to expand the application scope of speech recognition systems to multiple languages. The current multilingual speech recognition system has poor results when performing multilingual recognition.

[0004] Therefore, how to provide a multilingual speech recognition method to improve the effect of multilingual speech recognition has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] In view of the above problems, the present application provides a multilingual speech recognition model training method and related devices to achieve the purpose of improving the effect of multilingual speech recognition. The specific scheme is as follows:

[0006] The first aspect of the present application provides a multilingual speech recognition model training method, comprising:

[0007] Determine training data, wherein the training data includes voice text supervised training data and plain text training data;

[0008] Obtain a pre-built multilingual speech recognition model training model, wherein the multilingual speech recognition model training model includes a first training branch and a second training branch; wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share the decoder and part of the encoder;

[0009] The first training branch and the second training branch are jointly trained in a multi-task manner using the training data to obtain a trained first training branch as a multilingual speech recognition model.

[0010] In a possible implementation, determining the training data includes:

[0011] Collecting voice text supervised training sample data and plain text training sample data, wherein the voice text supervised training sample data includes voice sample data and recognition text labels corresponding to the voice sample data;

[0012] Performing sequence conversion processing on the speech sample data and the plain text training sample data in the speech text supervised training sample data to obtain training sequence data;

[0013] The training sequence data is subjected to labeling processing to obtain the training data.

[0014] In a possible implementation, performing sequence conversion processing on the speech sample data and the plain text training sample data in the speech text supervised training sample data to obtain training sequence data includes:

[0015] For each speech text supervised training sample data, frame feature extraction is performed on the speech sample data in the speech text supervised training sample data to obtain a speech frame feature sequence corresponding to the speech sample data in the speech text supervised training sample data;

[0016] For each plain text training sample data, the plain text training sample data is processed into text-to-phoneme to obtain a phoneme sequence corresponding to the plain text training sample data.

[0017] In a possible implementation, the first training branch consists of a speech feature encoder, a speech representation encoder, a speech-text shared encoder and a speech recognition decoder, and the second training branch consists of a language-shared text encoder, a text representation encoder, a speech-text shared encoder and a speech recognition decoder; wherein the first training branch and the second training branch share the speech-text shared encoder and the speech recognition decoder.

[0018] In a possible implementation, performing multi-task joint training on the first training branch and the second training branch using the training data includes:

[0019] The training data is used to jointly train the first training branch and the second training branch for a speech recognition task, a language recognition task, and a speech-text alignment task.

[0020] In a possible implementation, the speech text supervised training data includes training sequence data corresponding to the speech sample data, recognition text labels, language labels, and phoneme labels corresponding to each speech frame; the plain text training data includes training sequence data and language labels corresponding to the plain text sample data; then the joint training of the first training branch and the second training branch with the training data for speech recognition tasks, language recognition tasks, and speech-text alignment tasks includes:

[0021] Input the training sequence data corresponding to the speech sample data into the first training branch, and input the training sequence data corresponding to the plain text sample data into the second training branch;

[0022] The loss function of the speech recognition task, the loss function of the language recognition task and the loss function of the speech-text alignment task are constructed as a joint loss function;

[0023] The first training branch and the second training branch are trained based on the joint loss function.

[0024] A second aspect of the present application provides a multilingual speech recognition model training device, comprising:

[0025] A training data determination unit, used to determine training data, wherein the training data includes voice text supervised training data and plain text training data;

[0026] A training model acquisition unit, used to acquire a pre-built multilingual speech recognition model training model, wherein the multilingual speech recognition model training model includes a first training branch and a second training branch; wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share the decoder and part of the encoder;

[0027] The multi-task joint training unit is used to perform multi-task joint training on the first training branch and the second training branch using the training data to obtain the trained first training branch as a multilingual speech recognition model.

[0028] In a possible implementation, the training data determining unit includes:

[0029] A training sample data collection unit, used to collect voice-text supervised training sample data and plain text training sample data, wherein the voice-text supervised training sample data includes voice sample data and recognition text labels corresponding to the voice sample data;

[0030] A sequence conversion processing unit, used for performing sequence conversion processing on the speech sample data and the plain text training sample data in the speech text supervised training sample data to obtain training sequence data;

[0031] The labeling processing unit is used to perform labeling processing on the training sequence data to obtain the training data.

[0032] In a possible implementation, the sequence conversion processing unit is specifically used to:

[0033] For each speech text supervised training sample data, frame feature extraction is performed on the speech sample data in the speech text supervised training sample data to obtain a speech frame feature sequence corresponding to the speech sample data in the speech text supervised training sample data;

[0034] For each plain text training sample data, the plain text training sample data is processed into text-to-phoneme to obtain a phoneme sequence corresponding to the plain text training sample data.

[0035] In a possible implementation, the first training branch consists of a speech feature encoder, a speech representation encoder, a speech-text shared encoder and a speech recognition decoder, and the second training branch consists of a language-shared text encoder, a text representation encoder, a speech-text shared encoder and a speech recognition decoder; wherein the first training branch and the second training branch share the speech-text shared encoder and the speech recognition decoder.

[0036] In a possible implementation, the multi-task joint training unit is specifically used to:

[0037] The training data is used to jointly train the first training branch and the second training branch for a speech recognition task, a language recognition task, and a speech-text alignment task.

[0038] In a possible implementation, the speech text supervised training data includes training sequence data corresponding to the speech sample data, recognition text labels, language labels, and phoneme labels corresponding to each speech frame; the plain text training data includes training sequence data and language labels corresponding to the plain text sample data; then the multi-task joint training unit includes:

[0039] An input unit, used to input the training sequence data corresponding to the speech sample data into the first training branch, and input the training sequence data corresponding to the plain text sample data into the second training branch;

[0040] A joint loss function construction unit, used to construct the loss function of the speech recognition task, the loss function of the language recognition task and the loss function of the speech-text alignment task into a joint loss function;

[0041] A training unit is used to train the first training branch and the second training branch based on the joint loss function.

[0042] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the multilingual speech recognition model training method of the first aspect or any implementation of the first aspect.

[0043] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program so that the electronic device can implement the multilingual speech recognition model training method of the first aspect or any implementation method of the first aspect.

[0046] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the multilingual speech recognition model training method of the first aspect or any implementation of the first aspect.

[0047] A sixth aspect of the present application provides a speech recognition method, comprising:

[0048] Get the speech to be recognized;

[0049] Extracting frame features of the speech to be recognized to obtain a frame feature sequence of the speech to be recognized;

[0050] The frame feature sequence of the speech to be recognized is input into a multilingual speech recognition model, and the multilingual speech recognition model encodes the frame feature sequence of the speech to be recognized to obtain the encoded feature sequence of the speech to be recognized; and decodes the encoded feature sequence of the speech to be recognized to obtain a recognition result. The multilingual speech recognition model is trained based on the multilingual speech recognition model training method as described in the first aspect or any implementation of the first aspect.

[0051] By means of the above technical scheme, the multilingual speech recognition model training method and related devices provided by the present application pre-construct a multilingual speech recognition model training model including a first training branch and a second training branch, wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share a decoder and part of the encoder; through the supervised training data of speech text and the plain text training data, the first training branch and the second training branch are jointly trained with multiple tasks to obtain the trained first training branch as a multilingual speech recognition model. This scheme can reduce the computational overhead of model training and improve the utilization rate of training data, thereby training an efficient and accurate multilingual speech recognition model while effectively reducing the training cycle and reducing supervised training data. Using the multilingual speech recognition model trained by the present application for multilingual speech recognition can improve the effect of multilingual speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0053] Figure 1 A flowchart of a multilingual speech recognition model training method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the structure of a multilingual speech recognition model training model provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a speech feature encoder provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of a speech representation encoder, a text representation encoder, and a speech-text shared encoder provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of a multilingual speech recognition model training device provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0060] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] In order to better understand the technical solution of this application, the existing speech recognition technology is first introduced as follows:

[0063] Automatic Speech Recognition (ASR) technology can extract text information from spoken language and convert it into corresponding text. Speech recognition technology is essential to connect human language with computer processing capabilities. With the popularity of mobile devices, wearable technology, smart homes, and in-car infotainment systems, speech recognition technology plays an increasingly important role in many application areas. In these devices and systems, voice interaction is a more natural and convenient way of interaction that most people have mastered, making it a more superior usage mode on these devices and systems.

[0064] With the rapid development of deep learning technology and the continuous improvement of computer hardware performance, the recognition effect of speech recognition systems has reached a good level in common languages ​​such as Chinese and English, and has been applied in many smart devices. However, there are more than 7,000 languages ​​in the world, and most of them lack available speech recognition systems. There are large differences in writing systems, pronunciation patterns, transcription methods, data resources, etc. between different languages, which makes it a huge challenge to expand the application scope of speech recognition systems to multiple languages.

[0065] The initial solution was to use a model where each language model was built separately, data was trained separately, and applications were deployed and maintained separately, resulting in huge computing overhead and labor costs. The speech recognition system for each language usually relies on a large amount of manually annotated data to obtain excellent recognition results. However, most languages ​​have characteristics such as a small population base, a small scope of use, and constantly changing writing and pronunciation patterns, which makes it difficult to collect data resources and the cost of data annotation high, so that the speech recognition systems built separately for each language have poor results.

[0066] A multilingual speech recognition system is a speech recognition system that supports transcription of multiple languages. It has emerged as an effective means to expand the scope of application of speech recognition systems. The multilingual speech recognition system uses a cross-language unified model framework to learn speech recognition resources in multiple languages. It extracts relevant features and difference features between different languages ​​through a single shared deep learning model, effectively reducing the difficulty of semantic modeling of speech in a specific language, and shows great prospects in covering all languages ​​in the world. Compared with monolingual speech recognition systems, multilingual speech recognition systems not only show more significant performance in the recognition accuracy of low-resource languages, but also greatly simplify the training steps and the difficulty of deployment and maintenance. In addition, the multilingual speech recognition system can realize the cross-reuse of multilingual training data, improve the utilization rate of data resources, and reduce the labor and computing costs of application implementation.

[0067] Therefore, it is particularly necessary to build a speech recognition system that supports speech recognition in multiple languages.

[0068] Existing speech recognition systems can be divided into cascade speech recognition systems and end-to-end speech recognition systems based on model architecture.

[0069] The cascade speech recognition system consists of three subsystems: acoustic model, language model, and pronunciation dictionary. The acoustic model models the acoustic features, and the language model decodes the text in combination with the pronunciation dictionary. This type of system uses different types of data to train different models separately, and requires building specific data sets for different languages, making it difficult to effectively expand to speech recognition systems for a large number of languages.

[0070] The end-to-end speech recognition system jointly models the acoustic model, language model, pronunciation dictionary, etc. into a single unified model. The single model extracts multi-language speech features through a unified audio encoder and predicts the text content of the pronunciation through a unified text decoder and pronunciation dictionary, thereby effectively adapting to speech recognition systems with massive data scale and a large number of languages. This type of system supports multiple algorithms such as encoder-decoder, CTC, and RNN-T. The numerous related studies, excellent recognition performance, and simple system selection have led to its widespread application.

[0071] Multilingual speech recognition systems can be divided into supervised learning speech recognition systems and unsupervised data speech recognition systems based on learning algorithms.

[0072] Multilingual speech recognition systems with supervised learning are usually trained on labeled data (e.g., manually labeled data, pseudo-labeled data labeled by the recognition system, and labeled data generated by the speech synthesis system).

[0073] Multilingual speech recognition systems that use unsupervised data are trained by using unlabeled data (such as unlabeled speech data and unlabeled text data). Compared with the large amount of labeling cost required for labeled data, the acquisition of unlabeled data is more efficient, which is especially important for low-resource data and data in small languages.

[0074] When using unsupervised data for multilingual speech recognition systems to train unlabeled data, commonly used techniques are based on self-training (one of the semi-supervised learning algorithms) or self-supervised learning.

[0075] Among them, the self-training method (one of the semi-supervised learning algorithms) uses the existing recognition system to transcribe the unlabeled audio data, and the pseudo-labels obtained are improved through data cleaning strategies, and then the cleaned pseudo-labeled data and labeled data are used to retrain the speech recognition system. The self-training method is limited by a certain scale of labeled data, and most languages ​​currently do not have an available recognition system, making it difficult to fully apply to low-resource scenarios and small language fields.

[0076] The self-supervised learning method provides a speech recognition technology that effectively utilizes unlabeled data. This method usually uses pure speech data or non-parallel speech and text data to pre-train the speech recognition model, and learns effective speech representation based on a preset learning algorithm (such as autoregression-based methods, contrastive learning-based methods, and mask prediction-based methods). The pre-trained model is fine-tuned with a small amount of labeled data to achieve the same effect as supervised learning using more labeled data.

[0077] Compared with the self-training-based method which requires complex annotation cleaning and repeated iterations, the method based on self-supervised learning is relatively simple, but still requires two stages of pre-training and fine-tuning. In addition, model pre-training in multilingual scenarios still requires training of large-scale unlabeled data.

[0078] An existing multilingual speech recognition system based on self-supervised learning pre-trains the speech recognition model through multilingual unlabeled speech data, which effectively improves the content expression ability of speech representation in multilingual scenarios. Expanding the scale of unlabeled speech data can further effectively improve the multilingual recognition effect of the speech recognition system.

[0079] However, learning large-scale unlabeled speech data requires huge computational costs. For example, SeamlessM4T uses the w2v-BERT method to pre-train a 600 million-parameter speech encoder on a 1 million-hour speech dataset in 143 languages, which greatly prolongs the system's development cycle. Secondly, in the joint modeling of multilingual unlabeled speech data, the differences in pronunciation styles between different languages ​​make it difficult for the shared model to effectively distinguish pronunciation units, causing pronunciation interference between different languages, model capacity dilution and language crosstalk problems. On the other hand, multilingual speech recognition systems based on self-supervised learning use decoders with randomly initialized weights to fine-tune speech recognition tasks, resulting in huge differences in the training adequacy between the encoder and decoder, and the additional design of parameter optimization strategies, which increases the difficulty and uncertainty of model training, making it difficult to effectively enhance the information modeling capabilities of the speech recognition system at the semantic level.

[0080] Considering the above-mentioned defects of the existing multilingual speech recognition system, the effect of multilingual recognition using the existing multilingual speech recognition system is poor.

[0081] In order to solve the above problems, the inventors of this application found that compared with unlabeled speech data, unlabeled text data has a wider range of themes, styles and fields, and storage is simpler and more efficient, which effectively reduces the cost of data annotation, greatly increases the richness of resources, and improves the generalization ability of the recognition system. Based on this, the inventors of this application proposed a method for training a multilingual speech recognition model based on the actual needs of multilingual speech recognition. While effectively reducing the training cycle and reducing the labeled data, it makes full use of unlabeled text data to improve the effect of the multilingual speech recognition model.

[0082] The multilingual speech recognition model proposed in this application can be applied to multilingual voice interaction scenarios, such as smart home, smart subtitles, content review, voice assistant, education and training, car navigation, etc.

[0083] The multilingual speech recognition method proposed in this application is described in detail below with reference to the accompanying drawings.

[0084] Reference Figure 1 , Figure 1A flowchart of a multilingual speech recognition model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a multilingual speech recognition model training method provided in an embodiment of the present application may include the following steps, which are described in detail below.

[0085] S101: Determine training data, where the training data includes voice text supervised training data and plain text training data;

[0086] In the present application, the supervised speech text training data includes training sequence data corresponding to the speech sample data, recognition text labels, language labels, and phoneme labels corresponding to each speech frame; the plain text training data includes training sequence data and language labels corresponding to the plain text sample data;

[0087] S102: Acquire a pre-built multilingual speech recognition model training model, wherein the multilingual speech recognition model training model includes a first training branch and a second training branch; wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share the decoder and part of the encoder;

[0088] In the present application, in terms of network structure, the network structure of the encoder and the structure of the decoder can adopt the structure of any neural network model, and the present application does not impose any limitation on this.

[0089] For ease of understanding, this application provides an implementation of a multilingual speech recognition model training model, as follows:

[0090] In one possible implementation, see Figure 2 , Figure 2 A schematic diagram of the structure of a multilingual speech recognition model training model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the multilingual speech recognition model training model includes a first training branch and a second training branch, wherein the first training branch is composed of a speech feature encoder, a speech representation encoder, a speech-text shared encoder and a speech recognition decoder, and the second training branch is composed of a language-shared text encoder, a text representation encoder, a speech-text shared encoder and a speech recognition decoder; wherein the first training branch and the second training branch share the speech-text shared encoder and the speech recognition decoder;

[0091] Among them, the speech feature encoder consists of three layers of two-dimensional convolutional layers and linear layers. Its structure is as follows Figure 3 shown.

[0092] The language-shared text encoder consists of an embedding layer and a normalization layer, which transforms discrete phoneme sequences into continuous 512-dimensional features.

[0093] The speech representation encoder, text representation encoder and speech-text shared encoder adopt a multi-layer Conformer block structure. The Conformer block structure is as follows Figure 4 As shown. Figure 4 As shown, the Conformer block structure consists of a series of feedforward network layers, self-attention layers, convolutional layers, and normalization layers. In a possible implementation, the speech representation encoder can adopt an 8-layer Conformer block structure, the text representation encoder can adopt a 2-layer Conformer block structure, and the speech-text shared encoder can adopt an 8-layer Conformer block structure.

[0094] The speech recognition decoder can adopt a 2-layer LSTM structure.

[0095] S103: Perform multi-task joint training on the first training branch and the second training branch using the training data to obtain a trained first training branch as a multilingual speech recognition model.

[0096] In the present application, the tasks for performing multi-task joint training on the first training branch and the second training branch using the training data may include a speech recognition task, a language recognition task, and a speech-text alignment task. How to use the training data to perform joint training on the first training branch and the second training branch for the above-mentioned tasks will be described in detail through the following embodiments.

[0097] In this embodiment, a multilingual speech recognition model training model including a first training branch and a second training branch is pre-constructed, wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share a decoder and part of the encoder; through the supervised training data of speech text and the plain text training data, the first training branch and the second training branch are multi-task jointly trained to obtain the trained first training branch as a multilingual speech recognition model. This solution can reduce the computational overhead of model training and improve the utilization rate of training data, thereby effectively reducing the training cycle and reducing supervised training data, and training an efficient and accurate multilingual speech recognition model. Using the multilingual speech recognition model trained by the present application for multilingual speech recognition can improve the effect of multilingual speech recognition.

[0098] In another embodiment of the present application, a specific implementation method for determining training data is described, and the method may include the following steps:

[0099] S201: Collecting voice text supervised training sample data and plain text training sample data, wherein the voice text supervised training sample data includes voice sample data and recognition text labels corresponding to the voice sample data;

[0100] In the present application, the method for obtaining the supervised training sample data of speech text may be as follows: obtaining the initial speech text data, and determining the supervised training sample data of speech text based on the initial speech text data.

[0101] On the one hand, the initial speech text data can be collected from open source data, and on the other hand, it can be obtained by recognizing unlabeled pure speech data through an open source multilingual speech recognition model.

[0102] Based on the initial speech text data, the supervised training sample data of speech text is determined. Specifically, the initial speech text data can be directly determined as the supervised training sample data of speech text, or the initial speech text data can be expanded by data enhancement to obtain the supervised training sample data of speech text. The data enhancement method can be to add noise, reverberation, speed disturbance, volume disturbance, frequency domain mask, etc. to the audio.

[0103] The plain text training sample data may be obtained in the following manner: initial plain text data is obtained, and the plain text training sample data is determined based on the initial plain text data.

[0104] The initial plain text data can be obtained by crawling from the Internet on the one hand, and extracted from open source multilingual speech recognition data, speech translation data and machine translation data on the other hand.

[0105] The plain text training data is determined based on the initial plain text data. Specifically, the initial plain text data can be directly determined as the plain text training sample data, or the initial plain text data can be expanded by data enhancement to obtain the plain text sample training data. The data enhancement method can be phrase insertion, random word deletion, random change of phoneme duration, etc.

[0106] S202: performing sequence conversion processing on the speech sample data and the plain text training sample data in the speech text supervised training sample data to obtain training sequence data;

[0107] In the present application, after collecting the supervised training sample data of speech text and the plain text training sample data, the speech sample data and the plain text training sample data in the supervised training sample data of speech text are sequence converted to obtain the training sequence data. For the speech sample data in the supervised training sample data of speech text, the corresponding training sequence data is the speech frame feature sequence, and for the plain text training sample data, the corresponding training sequence data is the phoneme sequence.

[0108] In a possible implementation, the speech sample data in the speech text supervised training sample data and the plain text training sample data are subjected to sequence conversion processing to obtain training sequence data, including: for each speech text supervised training sample data, frame feature extraction is performed on the speech sample data in the speech text supervised training sample data to obtain a speech frame feature sequence corresponding to the speech sample data in the speech text supervised training sample data; for each plain text training sample data, text-to-phoneme processing is performed on the plain text training sample data to obtain a phoneme sequence corresponding to the plain text training sample data.

[0109] For ease of understanding, a speech frame feature sequence can be expressed as S=[s1, …, s N ], where N is the number of frames.

[0110] For ease of understanding, the phoneme sequence can be expressed as T=[t1, …, t M ], where M is the number of phonemes.

[0111] In the present application, a multilingual shared pronunciation unit dictionary can be pre-constructed. The multilingual shared pronunciation dictionary can be constructed by aggregating open source word pronunciation dictionaries on the one hand, and can predict the pronunciation of out-of-set words through an open source G2P (Grapheme-to-phoneme) model on the other hand. In the present application, the multilingual shared pronunciation unit dictionary includes multiple pronunciation units, and the pronunciation units can be phonemes.

[0112] In a possible implementation, a pre-built multilingual shared pronunciation unit dictionary may be used to perform text-to-phoneme processing on the plain text training sample data.

[0113] S203: performing labeling processing on the training sequence data to obtain the training data.

[0114] The labels of the training sequence data corresponding to the speech sample data in the speech text supervised training sample data include language labels and phoneme labels corresponding to each speech frame, and the labels of the training sequence data corresponding to the plain text training sample data include language labels. In a possible implementation, the phoneme labels corresponding to each speech frame can be determined using a pre-built multilingual shared pronunciation unit dictionary.

[0115] In the present application, in a possible implementation, the using the training data to perform multi-task joint training on the first training branch and the second training branch includes: using the training data to perform joint training on the first training branch and the second training branch for a speech recognition task, a language recognition task, and a speech-text alignment task.

[0116] In this application, the multilingual semantic information modeling capability of the multilingual speech recognition model can be improved by jointly training the speech recognition task, the language recognition task and the speech-text alignment task.

[0117] In another embodiment of the present application, a specific implementation method of using the training data to jointly train the first training branch and the second training branch for a speech recognition task, a language recognition task, and a speech-text alignment task is described. The method may include the following steps:

[0118] S301: inputting the training sequence data corresponding to the speech sample data into the first training branch, and inputting the training sequence data corresponding to the plain text sample data into the second training branch;

[0119] In a possible implementation, when the multilingual speech recognition model is trained each time using the training data, the speech frame feature S=[s1, …, s N ] Input speech feature encoder, where the two-dimensional convolution layer downsamples the input speech frame. Each convolution kernel size is 3, the number of channels is 64, the downsampling rate is 2, and each layer is activated by ReLU function; the linear layer projects the downsampled speech features to the same size as the embedding dimension of the speech representation encoder to generate high-dimensional features U=[u1, …, u N' ], input the high-dimensional features into the speech representation encoder to generate the speech context representation V=[v1, …, v N' ],in, is the length of the speech context representation, and then the speech context representation V=[v1, …, v N' ] is input to the speech-text shared encoder, and finally passes through the speech recognition decoder. The speech recognition decoder represents the output of the speech-text shared encoder as the posterior probability distribution p of the preset multilingual seed word dictionary dimension. That is, the speech recognition decoder outputs the probability of each subword in the multilingual seed word dictionary. The multilingual seed word dictionary here is obtained by integrating subword dictionaries of multiple languages.

[0120] On the other hand, the phoneme sequence T=[t1, …, t M ] Input language-shared text encoder to generate phoneme representation sequence W=[w1, …, w M ], and then input the phoneme representation sequence into the text representation encoder to generate the text context representation Z=[z1, …, z M], and then input the text context representation into the speech-text shared encoder, and finally the output of the speech-text shared encoder is represented by the speech recognition decoder as the posterior probability distribution p of the preset multilingual seed word dictionary dimension. That is, the speech recognition decoder outputs the probability of each subword in the multilingual seed word dictionary.

[0121] In addition, the speech recognition decoder can also output language prediction results, where the language prediction results are used to represent the posterior probabilities of each preset language.

[0122] S302: constructing the loss function of the speech recognition task, the loss function of the language recognition task, and the loss function of the speech-text alignment task into a joint loss function;

[0123] In the present application, the loss function of the speech recognition task is used to characterize the similarity between the text prediction result output by the first training branch and the recognized text label, and the similarity between the text prediction result output by the second training branch and the plain text sample data; the loss function of the language recognition task is used to characterize the similarity between the language prediction result output by the first training branch and the corresponding language label, and the similarity between the language prediction result output by the second training branch and the corresponding language label; the loss function of the speech-text alignment task is used to characterize the similarity between the representation corresponding to each frame in the speech context representation output by the speech representation encoder and the representation of the corresponding phoneme label, and the similarity between the representation corresponding to each position of the mask part in the text context representation output by the text representation encoder and the representation of the phoneme corresponding to the position.

[0124] In one possible implementation, the loss function for the speech recognition task is The speech recognition task loss function of the first training branch And the speech recognition task loss function of the second training branch It consists of two parts, and its mathematical expression is

[0125]

[0126]

[0127]

[0128] Where k is the index of each text modeling unit, K is the size of the preset multilingual text dictionary, and y t-1 is the text annotation at the previous moment, p represents the posterior probability of speech recognition output by the speech recognition decoder, is the model parameter shared by the first training branch and the second training branch, are model parameters unique to the first training branch, are model parameters unique to the second training branch, is the indicator function.

[0129] Loss function for language identification task The language recognition task loss function from the first training branch and the language recognition task loss function of the second training branch It consists of two parts, and its mathematical expression is

[0130]

[0131]

[0132]

[0133] Among them, l is the index of each language, L is the language type, is the label of the language type, p represents the posterior probability of language recognition output by the speech recognition decoder, is the model parameter shared by the first training branch and the second training branch, are model parameters unique to the first training branch, are model parameters unique to the second training branch, is the indicator function.

[0134] Loss function for speech-to-text alignment task is the first alignment loss function and the second alignment loss function Its mathematical expression is:

[0135]

[0136]

[0137]

[0138] Where n is the frame number position, is the length of the speech context representation, h is the index of the phoneme, H is the type of phoneme, exp is the exponential function with natural numbers as the base, cos is the cosine similarity function, and v n is the speech context representation at position n, y n is the phoneme notation at position n, w h represents the feature of phoneme embedding, m represents the phoneme position, Indicates the position of the mask part, z m Represents the context representation of the text at position m, y m is the phoneme annotation at position m, and τ is the scaling factor of the similarity.

[0139] It should be noted that for the speech-text alignment task, by optimizing the first alignment loss function, the speech and text can be aligned to a unified pronunciation space at the phoneme level, and by optimizing the second alignment loss function, the speech and text can be assisted to achieve word-level alignment, thereby improving the recognition accuracy of the multilingual speech recognition model.

[0140] The joint loss function may be a weighted sum of the loss function of the speech recognition task, the loss function of the language recognition task, and the loss function of the speech-text alignment task.

[0141] For ease of understanding, assume that the loss function of the speech recognition task is , the loss function of the language recognition task is , the loss function of the speech-to-text alignment task is , then in a possible implementation, the joint loss function can be expressed as:

[0142]

[0143] in, and They are respectively the weight coefficients of the loss function of the language recognition task and the loss function of the speech-to-text alignment task.

[0144] S303: Training the first training branch and the second training branch based on the joint loss function.

[0145] In the present application, when a preset number of iterations is reached, or the joint loss function converges, the training of the first training branch and the second training branch is completed.

[0146] The above introduces a multilingual speech recognition model training method provided by an embodiment of the present application. The following will introduce a device for executing the above multilingual speech recognition model training method.

[0147] See also Figure 5 , Figure 5 A schematic diagram of the structure of a multilingual speech recognition model training device provided in an embodiment of the present application. Figure 5 As shown, the multilingual speech recognition model training device includes:

[0148] A training data determination unit 11, used to determine training data, wherein the training data includes voice text supervised training data and plain text training data;

[0149] The training model acquisition unit 12 is used to acquire a pre-built multilingual speech recognition model training model, wherein the multilingual speech recognition model training model includes a first training branch and a second training branch; wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share the decoder and part of the encoder;

[0150] The multi-task joint training unit 13 is used to perform multi-task joint training on the first training branch and the second training branch using the training data to obtain the trained first training branch as a multilingual speech recognition model.

[0151] In a possible implementation, the training data determining unit includes:

[0152] A training sample data collection unit, used to collect voice-text supervised training sample data and plain text training sample data, wherein the voice-text supervised training sample data includes voice sample data and recognition text labels corresponding to the voice sample data;

[0153] A sequence conversion processing unit, used for performing sequence conversion processing on the speech sample data and the plain text training sample data in the speech text supervised training sample data to obtain training sequence data;

[0154] The labeling processing unit is used to perform labeling processing on the training sequence data to obtain the training data.

[0155] In a possible implementation, the sequence conversion processing unit is specifically used to:

[0156] For each speech text supervised training sample data, frame feature extraction is performed on the speech sample data in the speech text supervised training sample data to obtain a speech frame feature sequence corresponding to the speech sample data in the speech text supervised training sample data;

[0157] For each plain text training sample data, the plain text training sample data is processed into text-to-phoneme to obtain a phoneme sequence corresponding to the plain text training sample data.

[0158] In a possible implementation, the first training branch consists of a speech feature encoder, a speech representation encoder, a speech-text shared encoder and a speech recognition decoder, and the second training branch consists of a language-shared text encoder, a text representation encoder, a speech-text shared encoder and a speech recognition decoder; wherein the first training branch and the second training branch share the speech-text shared encoder and the speech recognition decoder.

[0159] In a possible implementation, the multi-task joint training unit is specifically used to:

[0160] The training data is used to jointly train the first training branch and the second training branch for a speech recognition task, a language recognition task, and a speech-text alignment task.

[0161] In a possible implementation, the speech text supervised training data includes training sequence data corresponding to the speech sample data, recognition text labels, language labels, and phoneme labels corresponding to each speech frame; the plain text training data includes training sequence data and language labels corresponding to the plain text sample data; then the multi-task joint training unit includes:

[0162] An input unit, used to input the training sequence data corresponding to the speech sample data into the first training branch, and input the training sequence data corresponding to the plain text sample data into the second training branch;

[0163] A joint loss function construction unit, used to construct the loss function of the speech recognition task, the loss function of the language recognition task and the loss function of the speech-text alignment task into a joint loss function;

[0164] A training unit is used to train the first training branch and the second training branch based on the joint loss function.

[0165] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0166] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0167] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0168] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any multilingual speech recognition model training method provided in the embodiment of the present application.

[0169] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any multilingual speech recognition model training method provided in the embodiment of the present application.

[0170] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0171] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0173] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A multilingual speech recognition model training method, characterized in that: include: Determine training data, wherein the training data includes voice text supervised training data and plain text training data; Obtain a pre-built multilingual speech recognition model training model, wherein the multilingual speech recognition model training model includes a first training branch and a second training branch; wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share the decoder and part of the encoder; The first training branch and the second training branch are jointly trained in a multi-task manner using the training data to obtain a trained first training branch as a multilingual speech recognition model.

2. The method according to claim 1, characterized in that The determining of training data comprises: Collecting voice text supervised training sample data and plain text training sample data, wherein the voice text supervised training sample data includes voice sample data and recognition text labels corresponding to the voice sample data; Performing sequence conversion processing on the speech sample data in the speech text supervised training sample data and the plain text training sample data to obtain training sequence data; The training sequence data is subjected to labeling processing to obtain the training data.

3. The method according to claim 2, characterized in that The performing sequence conversion processing on the speech sample data in the speech text supervised training sample data and the plain text training sample data to obtain training sequence data includes: For each speech text supervised training sample data, frame feature extraction is performed on the speech sample data in the speech text supervised training sample data to obtain a speech frame feature sequence corresponding to the speech sample data in the speech text supervised training sample data; For each plain text training sample data, the plain text training sample data is processed into text-to-phoneme to obtain a phoneme sequence corresponding to the plain text training sample data.

4. The method according to claim 1, characterized in that The first training branch consists of a speech feature encoder, a speech representation encoder, a speech-text shared encoder and a speech recognition decoder, and the second training branch consists of a language-shared text encoder, a text representation encoder, a speech-text shared encoder and a speech recognition decoder; wherein the first training branch and the second training branch share the speech-text shared encoder and the speech recognition decoder.

5. The method according to claim 4, characterized in that The performing multi-task joint training on the first training branch and the second training branch by using the training data includes: The training data is used to jointly train the first training branch and the second training branch for a speech recognition task, a language recognition task, and a speech-text alignment task.

6. The method according to claim 5, characterized in that The supervised speech text training data includes training sequence data corresponding to the speech sample data, recognition text labels, language labels, and phoneme labels corresponding to each speech frame; the plain text training data includes training sequence data and language labels corresponding to the plain text sample data; and the joint training of the speech recognition task, the language recognition task, and the speech-text alignment task for the first training branch and the second training branch using the training data includes: Input the training sequence data corresponding to the speech sample data into the first training branch, and input the training sequence data corresponding to the plain text sample data into the second training branch; The loss function of the speech recognition task, the loss function of the language recognition task and the loss function of the speech-text alignment task are constructed as a joint loss function; The first training branch and the second training branch are trained based on the joint loss function.

7. A multilingual speech recognition model training device, characterized in that: include: A training data determination unit, used to determine training data, wherein the training data includes voice text supervised training data and plain text training data; A training model acquisition unit, used to acquire a pre-built multilingual speech recognition model training model, wherein the multilingual speech recognition model training model includes a first training branch and a second training branch; wherein the first training branch and the second training branch are both composed of an encoder and a decoder, and the first training branch and the second training branch share the decoder and part of the encoder; The multi-task joint training unit is used to perform multi-task joint training on the first training branch and the second training branch using the training data to obtain the trained first training branch as a multilingual speech recognition model.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the multilingual speech recognition model training method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the multilingual speech recognition model training method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multilingual speech recognition model training method as described in any one of claims 1 to 6.

Images