A method for constructing and training a multilingual recognition model based on multi-task learning

Abstract

The present invention relates to a model construction method and training method for multi-task language recognition, including: using an unsupervised pre-training method to perform a first pre-training on a wav2vec model; establishing a training sample set containing multi-language sample data according to task requirements including language recognition and data effective recognition; using sample data in the training sample set to perform a second pre-training on the wav2vec model pre-trained for the first time; adaptively fine-tuning the wav2vec model pre-trained for the second time according to the task requirements to construct a final multi-language recognition model; and using sample data in the training sample set to train the multi-language recognition model again, so that the multi-language recognition model can simultaneously recognize the language and voice effectiveness of input voice data. The present invention realizes the simultaneous recognition of two tasks, language recognition and effective voice recognition, and improves the accuracy and recognition efficiency of language recognition.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a method for constructing and training a multi-language recognition model based on multi-task learning. Background Art

[0002] In recent years, speech recognition technology has made significant progress. As one of the important technologies of speech recognition, language recognition plays an important role in many fields such as military, medical, and education. In multilingual speech recognition systems, language recognition, as an upstream task in the speech recognition system, plays an important role in improving the accuracy of multilingual language recognition and the user experience of the system.

[0003] Traditional language recognition methods usually include three processes: speech signal feature extraction, language model construction and decision rules. In this process, commonly used speech features include MFCC, Fbank, spectrogram, i-vector, etc. The language model is usually one or more classifiers. Commonly used classifiers include SVM, decision tree, random forest, etc. The decision rule is related to the selected classifier. It is usually to calculate the probability or confidence of the sample to be classified based on prior information to predict the language category.

[0004] Language recognition methods based on deep learning usually do not require manual feature extraction. Instead, they transform speech into a distinguishable vector space through deep learning models, such as CNN, LSTM, transformer, and other nonlinear feature extractors, thereby defining an optimization objective function and using a gradient descent algorithm to update the model parameters until the category predicted by the model is consistent with the true category. In this type of method, the quality of the model often depends on the quality and quantity of the labeled data. In addition, these models find it difficult to extract deep features of speech, such as the speaker's gender, age, and pronunciation differences between different languages. Summary of the invention

[0005] In view of the above analysis, the present invention aims to provide a method for constructing and training a multi-language recognition model based on multi-task learning, and simultaneously realize multi-task recognition of the language of input data and whether it is a valid voice.

[0006] The technical solution provided by the present invention is:

[0007] The present invention discloses a method for constructing and training a multi-language recognition model based on multi-task learning, comprising the following steps:

[0008] Use unsupervised pre-training to perform the first pre-training of the wav2vec model;

[0009] Establish a training sample set containing multilingual sample data according to the task requirements including language recognition and data effective recognition;

[0010] The wav2vec model pre-trained for the first time is pre-trained again using the sample data in the training sample set;

[0011] According to the task requirements, the wav2vec model pre-trained for the second time is adaptively fine-tuned to construct the final multilingual recognition model;

[0012] The multilingual recognition model is trained again using sample data in the training sample set, so that the multilingual recognition model can simultaneously recognize the language and voice validity of the input speech data.

[0013] Furthermore, the data source of the training sample set includes call recording voice data; the call recording voice data includes valid voice data in multiple languages; after the valid voice data is processed including data resampling, silence removal and data cutting, sample data of valid voice is obtained; and the sample data is marked with language category.

[0014] Furthermore, the training sample set also includes invalid voice data whose audio range and features are different from those of valid voice data; after the invalid voice data is processed including data resampling, silence removal and data cutting, sample data of invalid voice is obtained, and the data is marked as invalid.

[0015] Furthermore, the processing of the valid voice data or invalid voice data includes:

[0016] 1) After the valid voice data or invalid voice data of multiple languages ​​are initially resampled, the sampling rate, encoding, precision and header file format of the voice data are unified;

[0017] 2) Split the voice data with a channel number greater than 1 into mono voice data;

[0018] 3) Perform speech detection on the monophonic voice data, divide the voice data into multiple voice segments of fixed duration, and judge whether the voice segment is a silent segment according to the energy of each sub-band within the frequency band of human voice. If yes, remove the segment, otherwise keep it, and obtain the voice data without silence;

[0019] 4) According to the threshold range of voice cutting, the voice data after removing silence is cut into voice data segments of fixed length;

[0020] 5) After resampling the cut voice data segments, sample data of valid or invalid voice input for model training and recognition are obtained.

[0021] Furthermore, the amount of speech data of a language with a small amount of speech data is increased by training sample enhancement; the training sample enhancement increases the amount of speech data by performing variable speed disturbance on the speech data at different speech speed multiples.

[0022] Furthermore, the multilingual recognition model includes a feature extraction layer, a context encoding layer and an output layer; the feature extraction layer adopts the convolutional network in the wav2vec model pre-trained for the second time; it is used to extract shallow features at the speech frame level of the input speech sample; the context encoding layer uses the transformer network based on the self-attention mechanism in the wav2vec model pre-trained for the second time, which is used to extract weights and features between each speech frame; based on the language recognition and data effective recognition tasks, the output layer and loss function of the wav2vec model are adjusted, and the extracted weights and features are transformed to the label dimension of the corresponding task using a fully connected network, and the recognition results of the two tasks are output at the same time.

[0023] Further, according to the task requirements, the adjusted output layer obtains the first i The output of the language task belongs to samples Y i And the output of the data effective recognition task Z i ;in,

[0024]

[0025]

[0026] Among them, P(X i , h y i ) is the normalized output of language category probability, P(X i , h z i ) Data validity probability normalized output, X i The first input of the multilingual recognition model i Sample data;

[0027] h y i It is the hidden layer feature obtained after the convolutional network and the self-attention network belonging to the language task; h y i =( h y i1 , h y i2,…, h y ij ,…, h y iT );

[0028] h z i For data effective recognition tasks, hidden features are obtained after passing through convolutional networks and self-attention networks; h y i =( h y i1 , h y i2 ,…, h y ij ,…, h y iT ); j =1,2,…,T; T is the number of frames after the convolution operation in the multilingual recognition model.

[0029] Furthermore, the loss function in the multilingual recognition model is L fine_tuning =(1- α ) L y + αL z ;

[0030] in, α A weighting factor to balance the outputs of the two tasks: language identification and data validity identification; L y is the softmax cross loss for the language recognition task, ; L z The sigmoid cross loss is used for data-efficient recognition tasks. ; y i , z i are the real category of the language and the real category of the effective sound, N is the total number of samples.

[0031] Furthermore, the loss function in the multilingual recognition model is L fine_tuning =(1- α ) L y + αLz ;

[0032] in, α A weighting factor to balance the outputs of the two tasks: language identification and data validity identification; L y is the softmax cross loss for the language recognition task, ; L z The sigmoid cross loss is used for data-efficient recognition tasks. ; y i , z i are the real category of the language and the real category of the effective sound, N is the total number of samples;

[0033] in, ; β The weighting factor for scaling the loss of different categories of languages; Reject Invalid data mark.

[0034] Furthermore, the α and β Take 0.2 and 1.5 respectively.

[0035] The present invention can achieve at least one of the following beneficial effects:

[0036] The multilingual recognition model constructed by the present invention directly inputs voice data, can judge invalid sounds while recognizing the language, and can simultaneously meet the two tasks of valid voice detection and language recognition in actual scenarios, saving time and space costs.

[0037] The present invention adopts a three-time training method, in which the model generated after two pre-trainings is used as the input of the initial model of the multilingual recognition model, and the parameters in the model except the output layer of the multilingual recognition model are initialized, so that the starting point of the fine-tuning task is at a position closer to the optimal point, so that the recognition effect of the two tasks of effective voice detection and language recognition after the third training is better and the training time is short.

[0038] A small amount of language labeled data is used in the training sample set, which reduces the manpower and time costs of obtaining a large amount of labeled data. In addition, labeled invalid speech data is introduced to improve the model's generalization ability for language recognition. In addition, the amount of language data with a small number of samples is increased through data enhancement with speech rate perturbation, so as to improve the model's training accuracy, recall rate and F1 value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0040] Figure 1 is a flow chart of a method for constructing and training a language recognition model in an embodiment of the present invention;

[0041] Figure 2 This is a structural diagram of the wav2vec pre-training task model in an embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of a multilingual recognition model in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used to illustrate the principles of the present invention together with the embodiments of the present invention.

[0044] In this embodiment, a method for constructing and training a language recognition model based on multi-task learning is described. Figure 1 As shown, the following steps are included:

[0045] Step S1, using an unsupervised pre-training method to perform a first pre-training on the wav2vec model; obtaining a representation of the speech data;

[0046] Specifically, this embodiment adopts the open source pre-training model such as the wav2vec model as the initial model, and uses the mask unsupervised pre-training method. Its pre-training task is to predict these fragments to obtain an enhanced representation of the speech context information, and use this as the initial parameters to fine-tune the parameters of downstream tasks, so that the downstream tasks have better performance.

[0047] Figure 2 The pre-trained model structure using wav2vec is given. As can be seen from the figure,

[0048] First, the original speech is input into the Feature encoder and then goes through 7 layers of convolution for feature extraction. The output of each layer is used as the input of the next layer. The step size of each layer is (5, 2, 2, 2, 2, 2, 2), and the convolution kernel width is (10, 3, 3, 3, 3, 2, 2). For example, if a (1, L)-dimensional speech is input, where L is the length of the speech, a 3D vector with a dimension of (1, L / 320, 512) is generated after feature encoding.

[0049] Secondly, after the previous step, we get a speech feature vector with a fixed dimension of 512. Then, we use a 12-layer block to calculate the "attention" between speech features. Each block is a transformer structure with 768 hidden units. Multi-head self-attention is used to calculate the attention weights between the above L / 320 feature sequences to obtain the deep context features of the entire speech. After encoding a speech through a 12-layer transformer, we get a vector with a dimension of (L / 320, 1, 768).

[0050] In the pre-training stage, in order to construct the prediction target, wav2vec masks a fixed-length feature sequence in the features of the CNN convolution output by means of a mask, and the goal of the training task is to predict these masked feature sequences. In order to calculate the masked features, wav2vec introduces a quantization module to discretize the representation of the context encoding output layer into a vector close to the one-hot vector, so as to quantize the output features and calculate the loss with the ground-truth. Wav2vec provides two quantization methods, Gumbel-Softmax and K-means clustering. The pre-training task of the present invention uses the former method. The loss function of the pre-training task is:

[0051] L pre =L m +α 1 L d in ，L m To make the contrast loss for mask prediction, L d For diversity loss, α 1 is set to 0.1. Specifically,

[0052] L m , c t For the current moment t The transformer network output is q t is the quantized output of the latent feature (feature encoder after multi-layer convolution). For the above two outputs, the cosine similarity is calculated, that is, sim(a,b)=a T b / ||a|| ||b||. Here, wav2vec introduces negative sampling technology: at the current moment t , the model should contain qt Including κ +1 quantization candidate identified q t ,in Q t Indicates this κ +1 quantization candidate, , κ is the number of interference items uniformly sampled from other masking items.

[0053] L d , G is the number of codebooks, which is 2, V is the number of entries in each codebook, which is 320, and the dimension of the codebook is 128; is the average gumbel softmax probability of a group of speech segments on each codebook entry; specifically ; is the non-negative temperature of the gumbel softmax, n =-log(-log( u )), u Obey the uniform distribution U(0,1), l g,v For each entry in the codebook, l g,v ∈R (G×V) .

[0054] The training data for the first pre-training of the wav2vec model is large-scale speech data; the large-scale speech data can be speech data other than the target language to be recognized, and does not need to be marked according to task requirements, and will not increase additional workload.

[0055] Use the processed speech to perform pre-training. The optimization goal of pre-training is as follows: L pre Less than the preset maximum value or the pre-trained learning rate is less than the preset maximum learning rate value.

[0056] Step S2: establishing a training sample set including multilingual sample data according to task requirements including language recognition and effective voice recognition;

[0057] In the multi-task learning of this embodiment, the language recognition task and the effective voice recognition task are each formalized as a multi-classification task, that is, the input is voice data, and the output is the language category and whether the voice is effective. Different from the general language recognition task, the task of this embodiment is to determine the language category label while predicting whether the voice is effective. By introducing the effective voice recognition task, the premise of judging the language category of a voice is that the voice must be effective voice data. However, it is difficult to ensure that all inputs are effective sounds in the actual intelligent voice recognition system, and often contains a large number of invalid sounds. The introduction of the effective voice recognition task has practical significance for language recognition. In addition, compared with effective voice, the audio range and characteristics of invalid sound are somewhat different from those of voice data, which can improve the generalization ability of language recognition. Therefore, the sample data in the training sample set includes sample data of effective voice and invalid voice data.

[0058] The data source of the training sample set is call recording voice data, including valid voice data in multiple languages; the valid voice data is processed including data resampling, silence removal and data cutting to obtain sample data of valid voice; and the sample data is marked with language category;

[0059] The invalid speech data in the training sample set is audio data whose audio range and features are different from those of the valid speech data, such as noise data or machine-synthesized voice prompts. After the invalid speech data is processed in the same way as the valid speech data, including data resampling, silence removal, and data cutting, the invalid speech sample data is obtained and the data is marked as invalid. In this embodiment, the invalid speech sample data is marked with the label "Reject" to indicate the language category of the invalid speech.

[0060] Specifically, valid speech data in multiple languages, involving 15 languages, including Russian, Hindi, Bengali, German, Japanese, Chinese, French, Persian, Tamil, Thai, English, Spanish, Vietnamese, Arabic, Korean and some invalid speech data, are randomly divided into training sets, development sets and test sets in the data set in Table 1 according to the different amounts of data in each category, to cross-validate the performance of the model.

[0061] More specifically, in order to achieve better training effect and match the trained model, the processing of the valid voice data or invalid voice data in this embodiment includes the following steps:

[0062] 1) After the valid voice data and invalid voice data of multiple languages ​​are initially resampled, the sampling rate, encoding, precision and header file format of the voice data are unified;

[0063] Specifically, through the initial resampling, the format of all voices is converted into a format with a sampling rate of 8000, a channel number of 1, PCM encoding, and an accuracy of 16 bits, and a voice header file is added to each voice; the header file is a piece of data located at the beginning of the voice file, which is used to describe the main data, occupies a total of 44 bytes, and includes voice format information such as sampling rate, number of channels, and encoding.

[0064] 2) Split the voice data with a channel number greater than 1 into mono voice data.

[0065] 3) Perform speech detection on the monophonic speech data, divide the speech data into multiple speech segments of fixed duration, and judge whether the speech segment is a silent segment based on the energy of each sub-band within the frequency band of human voice. If yes, remove the segment, otherwise keep it; obtain the speech data with silence removed.

[0066] Specifically, this embodiment uses webbrtc voice endpoint detection to detect voice segments;

[0067] First, the input speech is segmented into 20ms intervals to obtain a series of speech segments.

[0068] Secondly, each voice segment will be tested to see if it is silent. If so, the segment will be removed, otherwise it will be retained.

[0069] Among them, the Very Aggressive mode in the VAD of WebRTC is used for detection, and the spectrum of the input voice segment is divided into six sub-bands (80Hz~250Hz, 250Hz~500Hz, 500Hz~1K, 1K~2K, 2K~3K, 3K~4K); and the energy of these six sub-bands, that is, the features, is calculated respectively, and the probability density function of the Gaussian model is used for calculation to obtain the log-likelihood ratio function of the six sub-bands, that is, the probability of silence and speech.

[0070] The log-likelihood ratio of each sub-band is used as the condition for local judgment. According to the spectral range of human voice (approximately 80Hz to 1K), weight optimization is performed and (0.25, 0.25, 0.25, 0.08, 0.08, 0.08) are used as the weights of the above six sub-bands respectively. The weighted sum of the six sub-bands is taken as the global feature.

[0071] When judging whether it is silence, first judge locally, that is, judge whether the log likelihood ratio of each subband exceeds the judgment threshold. The threshold is divided into two groups, local and global, using 94 and 1100 respectively; when one subband exceeds the threshold, it is judged to contain speech; when all locals are judged to not contain speech, then judge globally, that is, judge whether the weighted sum of the six subbands exceeds the judgment threshold. If it exceeds the threshold, it is judged to contain speech, otherwise it is judged to be silence.

[0072] 4) According to the threshold range of voice cutting, the voice data after removing silence is cut into voice data segments of fixed length;

[0073] The threshold range of speech cutting includes a minimum length min_len and a maximum length max_len; the speech is cut into a length within a fixed length range according to min_len and max_len; the minimum speech min_len is used to filter out short speech, thereby removing some noise data in the training set and accelerating the convergence of the model; since the model cannot process too long data, max_len is set to intercept too long speech, thereby improving the training efficiency. In the present invention, min_len and max_len are respectively taken as 1 second and 30 seconds. As an experience, this scheme believes that the speech data below 1 second is difficult to determine its language category due to the limitation of the expression content and the speaker's speaking speed, while the speech data within 30 seconds is sufficient to determine the language involved in the speech content.

[0074] 5) After resampling the cut voice data segments, sample data of valid and invalid voices are obtained as input for model training and recognition.

[0075] The resampling is performed again, and the cut speech is uniformly converted into a format with a sampling rate of 16000 and a sampling accuracy of 16 bits as sample data for model training and recognition.

[0076] The processing of the sample data of this embodiment is performed by resampling twice. A relatively low sampling rate is used in the initial resampling, so that the format of the speech is unified, the amount of data during channel splitting, silence removal and speech cutting is reduced, the processing speed can be accelerated, and the requirements for processing hardware are reduced. After the above processing is completed, resampling is performed again at a high sampling rate so that the sampling rate and sampling accuracy of the sample data meet the requirements of model training.

[0077] For a certain language, due to too little sample data, the accuracy, recall rate and F1 will be poor during the training process, that is, the recognition effect will deteriorate. In order to improve the recognition effect of this language, this embodiment increases the amount of language data with a small amount of voice data by training sample enhancement.

[0078] Specifically, for the speech data of languages ​​with small data volume, variable speed disturbance is performed with each set speech multiple to increase the amount of speech data, so as to alleviate the overfitting caused by sample imbalance and improve the model accuracy.

[0079] Preferably, speed-changing disturbances of 0.9, 1.1, and 1.2 times are performed respectively, and the disturbance method is operated using the sox tool; the speed-changing operation increases the data volume of the language by 3 times; and then the effective voice data is processed by the above-mentioned processing method, which greatly increases the sample data volume of the language, so as to alleviate the overfitting and poor model accuracy problems caused by sample imbalance.

[0080] Step S3, using the sample data in the training sample set to perform a second pre-training on the wav2vec model that has been pre-trained for the first time, to obtain a general representation for the multilingual sample data;

[0081] Specifically, the model structure and loss function are not changed during the second pre-training process, and the training is terminated after the loss reaches a relatively balanced state;

[0082] Preferably, the learning rate of the first pre-training can be L Rpre Add 0.5 during the learning process until the loss L pre The value of is again less than the preset loss L pre The maximum value ends the pre-training; in the second pre-training process, due to the difference between the sample data in the training sample set and the first pre-training data, the loss function may be L pre It gets bigger, but it will gradually get smaller after a few rounds. By continuing training, the model parameters can be closer to the characteristics of the data, which is beneficial to improve the accuracy of the fine-tuning task.

[0083] The second pre-training preferably ends after more than 10,000 iterations.

[0084] Step S4, adaptively fine-tuning the second pre-trained wav2vec model according to the task requirements to construct a final multilingual recognition model; training the multilingual recognition model again using the sample data in the training sample set, so that the multilingual recognition model can simultaneously recognize the language and validity of the input speech data.

[0085] Specifically, after adaptive fine-tuning, the final multilingual recognition model is constructed, including a feature extraction layer, a context encoding layer and an output layer; the feature extraction layer adopts the convolutional network in the wav2vec model pre-trained for the second time; it is used to extract shallow features at the speech frame level of the input speech sample; the context encoding layer uses the transformer network based on the self-attention mechanism in the wav2vec model pre-trained for the second time, which is used to extract weights and features between each speech frame; based on the language recognition and effective speech recognition tasks, the output layer and loss function of the wav2vec model are adjusted, and the extracted weights and features are transformed to the label dimension of the corresponding task using a fully connected network, and the recognition results of the two tasks are output at the same time.

[0086] In order to train the language recognition task and the effective voice recognition task, the output layer of the model in this embodiment includes a softmax layer and a sigmoid layer to predict the language category and the effective voice category respectively. i Sample data, predict its category language category Y i and effective tone categories Z i .

[0087] The fine-tuned model structure is as follows Figure 3 As shown, the voice data i Samples X i =( x i1 , x i2 , …, x iL ) as input, L is the length of the speech (L is related to the sampling rate of the speech, and the sampling rate of the training samples in this scheme is 16K), and the hidden layer features are obtained after the convolutional network and the self-attention network. h i =( h i1 , h i2 ,…, h ij ,…, h iT ), where T is the number of frames after 7 layers of convolution operations, preferably T=L / 320; after the convolution network and the self-attention network, the output Y belonging to the language task is obtained through the improved output layer i and the effective voice category output Z i :

[0088] In the model, the hidden features of the language task are obtained after the convolutional network and the self-attention network. ;

[0089] The hidden features of the data effective recognition task are obtained after the convolutional network and the self-attention network. ;

[0090] Outputs belonging to language tasks:

[0091] Y i =argmax(P(X i ,h i y ))

[0092] ;

[0093] Outputs belonging to the data valid identification task:

[0094] Z i =argmax(P(X i ,h i z ))

[0095] ;

[0096] Among them, P(X i , h y i ) is the normalized output of language category probability, P(X i , h z i ) Data validity probability normalized output, X i The first input of the multilingual recognition model i Sample data;

[0097] h y i It is the hidden layer feature obtained after the convolutional network and the self-attention network belonging to the language task; h y i =( h y i1 , h y i2 ,…, h y ij ,…, h y iT );

[0098] h z iFor data effective recognition tasks, hidden features are obtained after passing through convolutional networks and self-attention networks; h y i =( h y i1 , h y i2 ,…, h y ij ,…, h y iT ); j =1,2,…,T; T is the number of frames after the convolution operation in the multilingual recognition model.

[0099] The softmax in this embodiment uses the log_softmax function, that is, a log function is passed after the softmax to alleviate the overflow and underflow problems in the calculation of the softmax function.

[0100] H i The calculation is as follows: ; It means that the features of the speech are averaged at the frame level, that is, the features of all frames of a speech are averaged as the input of the probability conversion function to obtain the target output category.

[0101] In addition, since there are only two categories of effective speech recognition, "valid" and "invalid", the sigmoid function is used as the activation function of the output layer.

[0102] Introducing weighting factors into the loss function in multilingual recognition models α To balance the output of the two tasks of language recognition and data effective recognition, use the weighting factor β To scale the loss of different categories of language.

[0103] Specifically, the loss function of the entire fine-tuning task is as follows:

[0104] L fine_tuning =(1- α ) L y + αL z ,in, L y is the softmax cross loss of the language, L z is the effective sigmoid cross loss, specifically:

[0105] ,

[0106] therefore,

[0107] ;

[0108] in, y i , z i are the real category of the language and the real category of the effective sound, Y i , Z i They are respectively the language category and the true category of the effective sound output by the model prediction.

[0109] N is the total number of samples.

[0110] In practice, different speech after language recognition is usually input into different transcription recognition engines for processing, which requires a higher accuracy rate for language recognition;

[0111] To this end, we further use the factor β To scale the loss of different categories of language, that is:

[0112] ,in, .

[0113] Specifically, the multilingual recognition model of this embodiment is fine-tuned based on the fairseq open source framework. During the entire fine-tuning process, the main structure of the model is still composed of 7 layers of convolution and 12 layers of Transformer, and the model parameters are not adjusted.

[0114] During the training process, the training samples are input into fairseq for training. The effect is relatively stable when α and β in the loss function are 0.2 and 1.5 respectively. The change of loss is recorded until the loss change is within 0.001 and the training is stopped.

[0115] The statistical results of language recognition training using data from 15 languages ​​are shown in Table 1.

[0116] The data in the table include data of 15 languages, and each language is randomly divided into three parts: training set, development set, and test set. In the experiment of data set 1, except for the languages ​​with *, which have less data and result in poor accuracy, recall rate and F1, the cross-validation of other languages ​​has achieved good recognition results; in data set 2, for these categories with fewer samples, the speech speed perturbation was performed at 0.9, 1.1, and 1.2 times the speed, and it is obvious that the data enhancement after speech speed perturbation has achieved good results in terms of accuracy, recall rate and F1.

[0117] Table 1 Implementation results of language recognition device

[0118]

[0119] In summary, the multilingual recognition model constructed in this embodiment directly inputs voice data and performs effective speech recognition while recognizing the language, which can simultaneously meet the two tasks of effective speech recognition and language recognition in actual scenarios, saving time and space costs.

[0120] A three-time training method is adopted, in which the model generated after two pre-trainings is used as the input of the initial model of the multilingual recognition model, and the parameters of the model except the output layer of the multilingual recognition model are initialized, so that the starting point of the fine-tuning task is at a position closer to the optimal point, so that after the third training, the recognition effects of the two tasks of language recognition and effective speech recognition are better, and the training time is short.

[0121] A small amount of language labeled data is used in the training sample set, which reduces the manpower and time costs of obtaining a large amount of labeled data. In addition, labeled invalid speech data is introduced to improve the model's generalization ability for language recognition. In addition, the amount of language data with a small number of samples is increased through data enhancement with speech rate perturbation, so as to improve the model's training accuracy, recall rate and F1 value.

[0122] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for constructing and training a multi-language recognition model based on multi-task learning, characterized in that: The following steps are involved: Use unsupervised pre-training to perform the first pre-training of the wav2vec model; Establish a training sample set containing multilingual sample data according to the task requirements including language recognition and data effective recognition; The data source of the training sample set includes call recording voice data, and the call recording voice data includes valid voice data in multiple languages; and also includes invalid voice data whose audio range and characteristics are different from those of valid voice data; The wav2vec model pre-trained for the first time is pre-trained again using the sample data in the training sample set to obtain a general representation for multilingual sample data; During the second pre-training process, the model structure and loss function are not changed until the loss reaches a relatively balanced state and the training ends; According to the task requirements, the wav2vec model pre-trained for the second time is adaptively fine-tuned to construct the final multilingual recognition model; After adaptive fine-tuning, the final multilingual recognition model is constructed, including a feature extraction layer, a context encoding layer and an output layer; the feature extraction layer uses the convolutional network in the second pre-trained wav2vec model; it is used to extract shallow features at the speech frame level of the input speech sample; the context encoding layer uses the transformer network based on the self-attention mechanism in the second pre-trained wav2vec model to extract the weights and features between each speech frame; based on the language recognition and effective voice recognition tasks, the output layer and loss function of the wav2vec model are adjusted, and the extracted weights and features are transformed to the label dimension of the corresponding task using a fully connected network, and the recognition results of the two tasks are output at the same time; the output layer includes a softmax layer and a sigmoid layer to predict the language category and the effective sound category respectively; The multilingual recognition model is trained again using the sample data in the training sample set, so that the multilingual recognition model can simultaneously recognize the language and voice validity of the input speech data.

2. The construction and training method according to claim 1, characterized in that: After the effective voice data is processed including data resampling, silence removal and data cutting, sample data of effective voice is obtained; and the sample data is marked with language category.

3. The construction and training method according to claim 1, characterized in that: After the invalid voice data is processed including data resampling, silence removal and data cutting, sample data of the invalid voice is obtained, and the data is marked as invalid.

4. The construction and training method according to claim 2 or 3, characterized in that: The processing of the valid voice data or invalid voice data includes: 1) After the valid voice data or invalid voice data of multiple languages ​​are initially resampled, the sampling rate, encoding, precision and header file format of the voice data are unified; 2) Splitting the voice data with a channel number greater than 1 into mono voice data; 3) Performing speech detection on the monophonic voice data, dividing the voice data into multiple voice segments of fixed duration, judging whether the voice segment is a silent segment according to the energy of each sub-band within the frequency band of human voice, if yes, removing the segment, otherwise retaining it, and obtaining the voice data without silence; 4) according to the threshold range of voice cutting, the voice data after removing the silence is cut into voice data segments of fixed length; 5) After the cut voice data segments are resampled again, sample data of valid voice or invalid voice as input for model training and recognition is obtained.

5. The construction and training method according to claim 2, characterized in that: The amount of speech data of a language with a small amount of speech data is increased by training sample enhancement; the training sample enhancement increases the amount of speech data by performing variable speed disturbance on the speech data at different speech speed multiples.

6. The construction and training method according to claim 1, characterized in that: According to the task requirements, the adjusted output layer obtains the output Y of the i-th sample belonging to the language task i and the output Z of the data effective recognition task i ; in, in, is the normalized output of language category probability, Data validity probability normalized output, X i The i-th sample data input to the multilingual recognition model; It is the hidden layer feature obtained after the convolutional network and the self-attention network belonging to the language task; For data effective recognition tasks, hidden features are obtained after passing through convolutional networks and self-attention networks; T is the number of frames after the convolution operation in the multilingual recognition model.

7. The construction and training method according to claim 6, characterized in that: The loss function L in the multilingual recognition model fine_tuning =(1-α)L y +αL z ; Among them, α is the weighting factor for balancing the output of the two tasks of language recognition and data effective recognition; L y is the softmax cross loss for the language recognition task, L z The sigmoid cross loss is used for data-efficient recognition tasks. y i 、z i are the true category of the language and the true category of the effective sound, and N is the total number of samples.

8. The construction and training method according to claim 6, characterized in that: The loss function L in the multilingual recognition model fine_tuning =(1-α)L y +αL z ; Among them, α is the weighting factor for balancing the output of the two tasks of language recognition and data effective recognition; L y is the softmax cross loss for the language recognition task, L z The sigmoid cross loss is used for data-efficient recognition tasks. y i 、z i are the true categories of the language and the true categories of the effective sounds, respectively, and N is the total number of samples; in, β is the weighting factor for scaling the loss of different language categories; Reject is a marker for invalid data.

9. The construction and training method according to claim 8, characterized in that: The α and β are set to 0.2 and 1.5 respectively.

Images