Speech language category identification method

Abstract

The invention relates to a speech language category recognition method, which comprises the steps of providing a target speech audio, and loading the target speech audio into a pre-constructed language recognition model so as to perform language recognition processing on the target speech audio by using the language recognition model, when the language recognition model carries out language recognition processing on the target voice audio, the language recognition processing at least comprises voice feature extraction processing, language embedding extraction processing and recognition classification processing which are executed in sequence, and after the voice feature extraction processing is executed, corresponding voice feature information is generated based on the target voice audio; after language embedding extraction processing is executed, corresponding language embedding information is generated based on the voice feature information; and after recognition and classification processing is executed, mapping the language embedded information into language classification information representing a language corresponding to the target voice audio. According to the invention, efficient recognition of multiple languages can be realized, and the accuracy and recognition efficiency of language recognition are improved.

Description

Technical Field

[0001] The present invention relates to a recognition method, in particular to a recognition method for speech and language categories. Background Art

[0002] Language recognition refers to the process of automatically identifying a given speech signal to determine the language category of the given speech signal. Language recognition plays a vital role in various applications such as multilingual speech recognition, machine translation, and call routing in customer service centers. In these scenarios, machine learning technology is usually used to train with large data sets to distinguish different languages ​​with high accuracy. With the increasing trend of internationalization, language exchanges between countries have gradually increased, and language recognition technology has become an indispensable part of modern speech processing systems.

[0003] Similar to other speech signal processing classification tasks, early language recognition systems relied on traditional manual features such as MFCC and FBANK. These features mainly describe the acoustic features of speech, cannot effectively capture the linguistic features in the sound, and lack flexibility and cannot adapt to complex speech data. Since then, based on traditional acoustic features, machine learning algorithms such as GMM, SVM and decision trees can be used for speech classification, but this classification method is easily affected by the environment and has poor generalization ability.

[0004] With the development of deep learning, deep neural networks have shown excellent nonlinear feature extraction capabilities in various speech signal processing tasks. Although deep neural networks have achieved good performance in the field of language recognition, they still rely on traditional manually extracted features, and most of these language recognition models based on acoustic features do not support end-to-end training, which to some extent limits the optimization of language recognition systems.

[0005] In recent years, self-supervised learning has emerged as a promising alternative in feature extraction for language recognition tasks. Unlike traditional methods, self-supervised learning uses a large amount of unlabeled data for pre-training and fine-tunes on labeled task-specific data. It can learn speech representations without a large amount of manual annotation. Such models can capture rich and complex features, surpassing the limitations of manual features and providing a more robust and generalizable understanding of data.

[0006] It should be noted that self-supervised learning involves a huge number of parameters, which increases the complexity of training, and relatively simple classification strategies cannot well capture the speech features extracted by self-supervised learning. In addition, the pre-training data and the data used in downstream tasks (i.e., language recognition tasks) may have domain shift problems. For example, the pre-training data may not contain the language that the downstream task needs to recognize, which in turn affects the performance of language recognition.

[0007] In addition, the use of traditional manual features and conventional classification algorithms can show performance that meets industrial needs when the training data is balanced. However, when dealing with unbalanced training data, the accuracy and generalization ability of the model drop sharply, which is a severe challenge in the face of low-resource language recognition.

[0008] The application document with publication number CN113889090A records a multi-language recognition model with multi-task learning. The application uses unsupervised pre-training and adaptive pre-training methods to train the multi-language model. However, the wav2vec model used in the application has a large number of parameters, which increases the complexity of training; and when the language samples in the application are unbalanced, the classification effect will be affected.

[0009] The application document with publication number CN112133291A records a language recognition model obtained by training a generative network, a neural network and a discriminative network based on discriminant probability and predicted language labels. However, the application requires additional data preprocessing steps, and when the model faces actual scenarios with different distribution from the training data, the generalization ability may be challenged.

[0010] In summary, there are still some deficiencies in the recognition of speech and language categories, which makes it difficult to meet current application needs. Summary of the invention

[0011] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for recognizing speech language categories, which can efficiently recognize multiple languages ​​and improve the accuracy and efficiency of language recognition.

[0012] According to the technical solution provided by the present invention, a method for recognizing speech and language categories comprises:

[0013] Provide target speech audio, and load the target speech audio into a pre-built language recognition model, so as to perform language recognition processing on the target speech audio using the language recognition model, wherein:

[0014] When the language recognition model performs language recognition processing on the target speech audio, it at least includes speech feature extraction processing, language embedding extraction processing and recognition classification processing performed in sequence, wherein:

[0015] After performing speech feature extraction processing, corresponding speech feature information is generated based on the target speech audio;

[0016] After executing the language embedding extraction process, generating corresponding language embedding information based on the speech feature information;

[0017] After performing the recognition and classification processing, the language embedding information is mapped to language classification information representing the language corresponding to the target speech audio, wherein the language classification information is a reference language in the reference language set used when constructing the language recognition model.

[0018] The language recognition model includes a speech feature extraction module, a language embedding extraction module and a recognition classification module connected in sequence, wherein:

[0019] When performing language recognition processing on the target speech audio, the speech feature extraction module is used to perform speech feature extraction processing, and the generated speech feature information is loaded into the language embedding extraction module;

[0020] Using the language embedding extraction module to perform language embedding extraction processing, and loading the generated corresponding language embedding information into the recognition and classification module;

[0021] The recognition and classification module is used to perform recognition and classification processing and generate language classification information.

[0022] The speech feature extraction module is generated based on the wav2vec2.0 model, and the language embedding extraction module is generated based on the x-vector model.

[0023] When building a language recognition model, the construction method includes:

[0024] Build and generate speech feature extraction module based on wav2vec2.0 model;

[0025] After the speech feature extraction module is generated, a language recognition basic model is constructed, wherein when the language recognition basic model is constructed, the speech feature extraction module is connected to the language embedding extraction basic module, and the language embedding extraction basic module is connected to the recognition classification basic module;

[0026] A model training sample set is constructed, and the model training sample set is used to train the language recognition basic model until the language recognition basic model training reaches a target training state, and then the language recognition basic model that reaches the target training state is configured as the language recognition model, wherein:

[0027] The model training sample set includes a number of model training samples. When the model training sample set is used to train the language recognition basic model, the network parameters of the speech feature extraction module are frozen, and the speech feature extraction module is used to extract the speech feature information of the model training samples, and the extracted speech feature information is loaded into the language embedding extraction basic module, wherein:

[0028] The model training sample includes a training speech audio and label information corresponding to the training speech audio, wherein the label information is a reference language corresponding to the training speech audio;

[0029] The training speech audio and the target speech audio have the same speech feature parameters, where the speech feature parameters include speech channels, sampling rate and / or bit depth.

[0030] When building a speech feature extraction module, include:

[0031] Construct a speech classification module based on the wav2vec2.0 model, and perform pre-training and fine-tuning training on the speech classification module in turn. After the fine-tuning training, generate a speech feature extraction module based on the speech classification module, wherein:

[0032] When pre-training the speech classification module, a pre-training sample set is constructed, and the speech classification module is pre-trained using the pre-training sample set based on a self-supervised learning training method, and when the pre-training reaches a pre-training target state, a speech classification post-pre-training module is generated, wherein the pre-training sample set includes a plurality of pre-training samples, and the pre-training samples are unlabeled speech audio;

[0033] During fine-tuning training, a fine-tuning training sample set is constructed, and the speech classification pre-trained module is trained using the fine-tuning training sample set. When the speech classification pre-trained module is fine-tuned to reach a target fine-tuning state, the speech classification pre-trained module that has reached the target fine-tuning state is configured as a speech feature extraction module, wherein:

[0034] The fine-tuning training sample set includes fine-tuning training samples in multiple languages, and the fine-tuning training samples are unlabeled speech audio.

[0035] When pre-training the speech classification module, the loss function used is:

[0036]

[0037] Among them, L pretrain is the pre-training loss function value, z i is the potential representation feature of the i-th time step generated based on the current pre-training sample, c i is the potential representation feature z based on the i-th time step i The generated context features, c j It is the context feature generated based on the jth negative sample in the negative sample set, M is the target time step set of the current pre-training sample, N is the negative sample set of the current pre-training sample, and sim is the cosine similarity calculation.

[0038] When training the basic language recognition model, the batch loss function used is:

[0039]

[0040] Among them, Lc is the batch loss function value during model training, B is the batch size, is the correct category y of the i-th model training sample i The predicted probability of It represents the farthest positive sample distance between the i-th model training sample and other model training samples in the same batch. It represents the nearest negative sample distance between the i-th model training sample and other model training samples in the same batch, and δ is the distance boundary threshold.

[0041] The language embedding extraction module includes a language local feature extraction layer, a statistical pooling layer and a language level feature extraction layer connected in sequence, wherein:

[0042] Connecting with the speech feature extraction module through the language local feature extraction layer;

[0043] It is connected to the recognition and classification module through the language-level feature extraction layer.

[0044] The recognition and classification module includes a fully connected layer and a softmax layer, wherein:

[0045] When mapping language embedding information to language classification information, the language embedding information is processed in sequence using the fully connected layer and the softmax layer, and the language classification information is output through the softmax layer.

[0046] A speech language recognition system includes a language recognition device, and the above-mentioned language recognition model is deployed in the language recognition device, wherein:

[0047] The language recognition model is used to perform language recognition processing on the target speech audio, and after the language recognition processing, language classification information corresponding to the target speech audio is generated.

[0048] The advantages of the present invention are as follows: when constructing a language recognition model, a speech feature extraction module after pre-training and adaptive fine-tuning training is used as a feature extractor in the language recognition model, so that the language recognition model can extract features that are closer to downstream tasks; at the same time, the speech feature extraction module is connected to a language embedding extraction module based on an x-vector model, which can fully capture the complex relationship between features, so that the recognition effect of language recognition is better.

[0049] During the construction of the language recognition model, the parameter update of the speech feature extraction module based on the wav2vec2.0 model was frozen, which reduced the trainable parameters and the time cost of training. That is, the training time for building the language recognition model was shorter.

[0050] The language recognition model does not rely on a large amount of language labeled data, reducing the manpower and time cost of obtaining labeled data; the self-supervised model learns the common features in speech, improving the model's generalization ability for language recognition, and uses data enhancement processing to perturb the training data to improve the model's precision, recall, F1 value and other training indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The figure is a flow chart of an embodiment of the present invention for speech language category recognition.

[0052] Figure 2 A schematic diagram of an embodiment of the language recognition model constructed for the present invention.

[0053] Figure 3 A schematic diagram of an embodiment of the language recognition model of the present invention.

[0054] Figure 4 A schematic diagram of an embodiment of the benchmark language set of the present invention.

[0055] Explanation of the accompanying drawings: 1-speech feature extraction module, 2-language embedding extraction module, 3-recognition and classification module. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0057] In order to efficiently recognize multiple languages ​​and improve the accuracy and efficiency of language recognition, the present invention provides a method for recognizing speech language categories. Specifically, the recognition method includes:

[0058] Provide target speech audio, and load the target speech audio into a pre-built language recognition model, so as to perform language recognition processing on the target speech audio using the language recognition model, wherein:

[0059] When the language recognition model performs language recognition processing on the target speech audio, it at least includes speech feature extraction processing, language embedding extraction processing and recognition classification processing performed in sequence, wherein:

[0060] After performing speech feature extraction processing, corresponding speech feature information is generated based on the target speech audio;

[0061] After executing the language embedding extraction process, generating corresponding language embedding information based on the speech feature information;

[0062] After performing the recognition and classification processing, the language embedding information is mapped to language classification information representing the language corresponding to the target speech audio, wherein the language classification information is a reference language in the reference language set used when constructing the language recognition model.

[0063] It should be noted that speech language category recognition is to identify the language type of the speech, such as recognizing the speech type of the speech as Chinese, English, French, etc. The meaning of the speech language category is consistent with the existing technology. Figure 1 An embodiment of the present invention for speech language category recognition is shown in the figure. As can be seen from the figure, when performing speech language category recognition, it is necessary to construct a language recognition model. Thereafter, the constructed language recognition model can be used to perform language category recognition on the target speech audio, wherein the target speech audio is the speech to be subjected to language category recognition.

[0064] Figure 1 Also shown is an embodiment of using a language recognition model to perform language recognition on a target speech audio. As can be seen from the figure, when performing language category recognition, the language recognition model is used to perform language recognition processing on the target speech audio, wherein the language recognition processing should at least include speech feature extraction processing, language embedding extraction processing, and recognition classification processing. Specifically, after performing speech feature extraction processing on the target speech audio, speech feature information corresponding to the target speech audio can be generated. Thereafter, language embedding extraction processing is performed on the speech feature information to generate corresponding language embedding information. After recognition and classification processing of the language embedding information, language classification information can be generated.

[0065] It should be noted that based on the generated language classification information, the language category of the target speech audio can be determined, that is, the language category of the target speech audio is recognized, wherein, when constructing the language recognition model, a reference language set should be configured, such as using the speech audio of each reference language in the configured reference language set for training. In one embodiment of the present invention, when the language category of the target speech audio is recognized, the language category of the target speech audio will be recognized as a reference language in the reference language set. In specific implementation, the reference languages ​​in the reference language set may include Chinese, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Fulah and / or Oromo. At this time, after the language category of the target speech audio is recognized, the language category of the target speech audio can be recognized as one of Chinese, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Fulah and / or Oromo.

[0066] During specific implementation, for the target speech audio loaded into the language recognition model, the speech feature parameters of the target speech audio should be configured, wherein the speech feature parameters include speech channels, sampling rate and / or bit depth. For example, the speech channel of the target speech audio should be mono, the sampling rate can be 16K, and the bit depth can be 16bit. The speech feature parameters of the target speech audio can be set according to the actual language recognition parameters, and examples will not be given one by one here.

[0067] In one embodiment of the present invention, the language recognition model comprises a speech feature extraction module 1, a language embedding extraction module 2 and a recognition classification module 3 connected in sequence, wherein:

[0068] When performing language recognition processing on the target speech audio, the speech feature extraction module 1 is used to perform speech feature extraction processing, and the generated speech feature information is loaded into the language embedding extraction module 2;

[0069] The language embedding extraction module 2 is used to perform language embedding extraction processing, and the generated corresponding language embedding information is loaded into the recognition and classification module 3;

[0070] The recognition and classification module 3 is used to perform recognition and classification processing and generate language classification information.

[0071] Figure 2 and Figure 3 An embodiment of a language recognition model is shown in the figure. As can be seen from the figure, the language recognition model may include a speech feature extraction module 1, a language embedding extraction module 2 and a recognition and classification module 3, wherein the speech feature extraction module 1 can perform the above-mentioned speech feature extraction processing in the language recognition processing, the language embedding extraction module 2 can perform the above-mentioned language embedding extraction processing in the language recognition processing, and the recognition and classification module 3 can perform the above-mentioned recognition and classification processing in the language recognition processing.

[0072] In one embodiment of the present invention, the speech feature extraction module 1 is generated based on the wav2vec2.0 model, and the language embedding extraction module 2 is generated based on the x-vector model. Of course, the speech feature extraction module 1 and the language embedding extraction module 2 can also adopt other forms. The following takes the generation of the speech feature extraction module 1 based on the wav2vec2.0 model and the generation of the language embedding extraction module 2 based on the x-vector model as an example to specifically explain the method of constructing the language recognition model of the present invention.

[0073] In one embodiment of the present invention, when constructing a language recognition model, the construction method includes:

[0074] Based on the wav2vec2.0 model, the speech feature extraction module 1 is constructed;

[0075] After the speech feature extraction module 1 is generated, a language recognition basic model is constructed, wherein when the language recognition basic model is constructed, the speech feature extraction module is connected to the language embedding extraction basic module, and the language embedding extraction basic module is connected to the recognition classification basic module;

[0076] A model training sample set is constructed, and the model training sample set is used to train the language recognition basic model until the language recognition basic model training reaches a target training state, and then the language recognition basic model that reaches the target training state is configured as the language recognition model, wherein:

[0077] The model training sample set includes a number of model training samples. When the model training sample set is used to train the language recognition basic model, the network parameters of the speech feature extraction module are frozen, and the speech feature extraction module is used to extract the speech feature information of the model training samples, and the extracted speech feature information is loaded into the language embedding extraction basic module, wherein:

[0078] The model training sample includes a training speech audio and label information corresponding to the training speech audio, wherein the label information is a reference language corresponding to the training speech audio;

[0079] The training speech audio and the target speech audio have the same speech feature parameters, where the speech feature parameters include speech channels, sampling rate and / or bit depth.

[0080] In specific implementation, when constructing a language recognition model, the speech feature extraction module 1 should be constructed first, and then the language recognition basic model should be constructed. The language recognition basic model can be used to construct the required language recognition model. The process of constructing the speech feature extraction module 1 is specifically described below.

[0081] In one embodiment of the present invention, when constructing the speech feature extraction module 1, it includes:

[0082] A speech classification module based on the wav2vec2.0 model is constructed, and the speech classification module is pre-trained and fine-tuned in sequence. After the fine-tuning training, a speech feature extraction module 1 is generated based on the speech classification module, wherein:

[0083] When pre-training the speech classification module, a pre-training sample set is constructed, and the speech classification module is pre-trained using the pre-training sample set based on a self-supervised learning training method, and when the pre-training reaches a pre-training target state, a speech classification post-pre-training module is generated, wherein the pre-training sample set includes a plurality of pre-training samples, and the pre-training samples are unlabeled speech audio;

[0084] During fine-tuning training, a fine-tuning training sample set is constructed, and the speech classification pre-trained module is trained using the fine-tuning training sample set. When the speech classification pre-trained module is fine-tuned to reach a target fine-tuning state, the speech classification pre-trained module that has reached the target fine-tuning state is configured as a speech feature extraction module, wherein:

[0085] The fine-tuning training sample set includes fine-tuning training samples in multiple languages, and the fine-tuning training samples are unlabeled speech audio.

[0086] From the above description, it can be seen that the wav2vec2.0 model can be an existing commonly used recognition model, and the situation of the wav2vec2.0 model can be consistent with the existing one. For example, the existing wav2vec2.0 model can be used to directly build a speech classification module, that is, the network parameters of the speech classification module can be the default network parameters of the wav2vec2.0 model. The constructed speech classification module should be pre-trained and fine-tuned in turn, such as Figure 2 As shown, a speech feature extraction module 1 can be generated after fine-tuning training. The specific method and process of pre-training and fine-tuning training are illustrated below.

[0087] When pre-training the speech classification module, it is necessary to construct a pre-training sample set, wherein the pre-training sample set includes pre-training samples, and each pre-training sample is an unlabeled speech audio. In specific implementation, speech audio can be obtained from the Internet, public speech data sets, etc. through existing technologies, and used as pre-training samples. It can be seen that since there is no need to produce label data, no additional workload is added when constructing the pre-training sample set. After constructing the pre-training sample set, the pre-training sample set can be used to train the speech classification module, and the specific training method can be consistent with the existing technology.

[0088] Since the speech classification module is built based on the wav2vec2.0 model, the speech classification module has the characteristics of the wav2vec2.0 model. For example, for any pre-trained sample, the speech classification module first uses a convolutional feature extractor composed of 7 layers of convolution to extract features of the pre-trained sample, and outputs the potential representation information of the speech. Among them, in the convolutional feature extractor, the output of each layer of convolution is used as the input of the next layer, and the step size of each layer of convolution is (5, 2, 2, 2, 2, 2, and the convolution kernel sizes are (10, 3, 3, 3, 3, 2, 2), respectively. Finally, a three-dimensional vector with 512 channels is output, that is, the potential representation information is a three-dimensional vector with 512 channels.

[0089] The potential representation information output by the convolutional feature extractor is mapped to 1024 dimensions by the fully connected layer; thereafter, after relative position encoding, it is input into the context feature encoder composed of 24 layers of transformers. In the context feature encoder, each layer of transformer contains a multi-head attention mechanism and a feedforward neural network to capture the global context information in the speech signal; after encoding by the context feature encoder, a three-dimensional vector with 1024 channels can be obtained, and the three-dimensional vector with 1024 channels is the context information.

[0090] In order to predict the target, the wav2vec2.0 model randomly and continuously masks the features of a preset number of time steps output by the convolutional feature extractor through a masking strategy, and reconstructs these masked features from the remaining contextual information, forcing the wav2vec2.0 model to pay more attention to global and local contextual information, thereby learning rich feature representations without display labels. The preset number of time step features can be 10; time step means that the pre-training samples will be divided into small time windows by the convolutional feature extractor, each time window corresponds to a time step, and features will be extracted at each time step to generate a set of feature vectors, thereby generating the corresponding potential representation information.

[0091] In one embodiment of the present invention, when pre-training the speech classification module, the loss function used is:

[0092]

[0093] Among them, L pretrain is the pre-training loss function value, z i is the potential representation feature of the i-th time step generated based on the current pre-training sample, c i is the potential representation feature z based on the i-th time step i The generated context features, c j It is the context feature generated based on the jth negative sample in the negative sample set, M is the target time step set of the current pre-training sample, N is the negative sample set of the current pre-training sample, and sim is the cosine similarity calculation.

[0094] From the above wav2vec2.0 model description, we can see that the potential representation feature z of the i-th time step is i Generate a feature vector for the i-th time step, which is mapped to a feature with 1024 channels through a fully connected layer before relative position encoding. i is the potential representation feature z at the i-th time step i After the relative position encoding, the context feature encoder performs feature capture, and the features generated by the mask strategy (mask), the context feature ci It is a three-dimensional vector with 1024 channels.

[0095] The target time step set M consists of the time steps selected from all pre-trained samples in a batch, which correspond to the positive sample features c that we want the wav2vec2.0 model to learn. i For the potential representation feature z at the i-th time step i , construct a negative sample set N, which contains the potential representation feature z of the i-th time step i The corresponding negative sample features are usually composed of the potential representation features z of other pre-training sample data in the same batch and the i-th time step i The potential representation features corresponding to different time steps, thus, each time step can determine the corresponding positive and negative samples.

[0096] In order to effectively train the speech classification module based on the wav2vec2.0 model, the above pre-training loss can be sampled. After a round of training of the speech classification module using the pre-training sample set, the corresponding pre-training loss function value L is calculated based on the above loss function. pretrain Of course, when pre-training the speech classification module, some pre-training configuration parameters should also be set to update the network parameters of the speech classification module. In specific implementation, during the iterative pre-training process, when the pre-training loss function value L pretrain When it tends to be stable, the pre-training of the speech classification module can be terminated, and at this time, a speech classification pre-trained module can be generated.

[0097] From the above description, we can see that pre-training is Figure 2 A large amount of unlabeled data is used for self-supervised pre-training. The unlabeled data refers to the unlabeled speech audio of the pre-training samples mentioned above. The pre-training reaches the pre-training target state, specifically the training loss function value L pretrain Of course, when pre-training the speech classification module, other loss functions can be used and corresponding pre-training target states can be set. The specific selection can be made according to the needs, and no examples will be given here.

[0098] After the speech classification pre-trained module is obtained through the above pre-training, fine-tuning training should be performed on the speech classification pre-trained module. When fine-tuning training is performed, a fine-tuning training sample set needs to be constructed. The fine-tuning training sample set includes fine-tuning training samples of multiple languages, wherein the language corresponding to the fine-tuning training samples in the fine-tuning training sample set should be consistent with the benchmark language of the above benchmark language set; in addition, the fine-tuning training samples are unlabeled speech audio.

[0099] When constructing the required fine-tuning training sample set, a feasible method is to determine the open source language recognition dataset Fleurs and the open source language recognition dataset Slingua as the source datasets for fine-tuning training samples. After that, a piece of data in the training data part of the open source language recognition dataset Fleurs and the training data part of the open source language recognition dataset Slingua can be used as the corresponding fine-tuning training source sample. For example, a piece of voice data in the training data part of the open source language recognition dataset Fleurs can be used as a fine-tuning training source sample. Similarly, a piece of voice data in the training data part of the open source language recognition dataset Slingua can be used as another fine-tuning training source sample. In specific implementation, all the data in the training data part of the open source language recognition dataset Fleurs and the open source language recognition dataset Slingua can be selected as the corresponding fine-tuning training source samples, or the data in the selected part can be used as the fine-tuning training source samples. It should be noted that when selecting to generate the fine-tuning training source sample, all the reference languages ​​in the above-mentioned reference language set should be included.

[0100] The fine-tuning training source samples generated by the above extraction should be processed into speech feature parameters with the above target speech audio, such as being processed into mono, 16k sampling rate, 16-bit bit depth speech audio. At this time, after processing, each fine-tuning training source sample can generate a corresponding fine-tuning training source preprocessing sample.

[0101] In order to generate corresponding fine-tuning training samples from fine-tuning training source preprocessing samples, data enhancement processing should also be performed on the fine-tuning training source preprocessing samples. In specific implementation, the data enhancement processing method for fine-tuning training source preprocessing samples derived from the open source language recognition dataset Fleurs will be different from the data enhancement processing method for fine-tuning training source preprocessing samples derived from the open source language recognition dataset Slingua. The specific distance description is given below.

[0102] When the fine-tuning training source preprocessing sample comes from the open source language recognition dataset Fleurs, when performing data enhancement processing on each open source language recognition dataset Fleurs, specifically: the fine-tuning training source preprocessing sample is segmented into 4-second segments, and then the segmented speech segments are randomly spliced ​​at both ends or three segments, and then data enhancement such as speed change, noise, music, reverberation, etc. is performed, so that a corresponding fine-tuning training sample can be generated after the data enhancement.

[0103] When the fine-tuning training source preprocessed samples come from the open source language recognition dataset Slingua, when performing data augmentation processing on each open source language recognition dataset Slingua, specifically: the fine-tuning training source preprocessed samples are divided into 15-second segments, and then 1.2 times and 0.8 times speed perturbations are performed, so that a corresponding fine-tuning training sample can be generated after the speed perturbation.

[0104] In specific implementation, the Kaldi tool can be used to perform the above data enhancement processing, where the parameter setting is to start the enhancement from the beginning of the fine-tuning training source preprocessing sample, and the signal-to-noise ratio selection range is randomly selected from the interval of 0-10. In addition, the offline source of noise, music, and reverberation noise can use the open source Musan offline source or the RIRs offline source. Of course, other methods can also be used for data enhancement processing, which will not be explained one by one here.

[0105] From the above description, it can be seen that when sampling the above method to make fine-tuning training samples, it is not necessary to make labels for each fine-tuning training sample. Therefore, fine-tuning training is Figure 2 Use the constructed language recognition unlabeled data for adaptive fine-tuning.

[0106] Based on the speech classification pre-training module, the corresponding weights can be obtained When fine-tuning training, the weights Initialize and continue self-supervised learning. The goal of the fine-tuning training phase is to make the speech classification pre-trained module better adapt to the data distribution of the language recognition task. Specifically, in the fine-tuning phase, the initial parameters of the speech classification pre-trained module are θ finetune (0), the optimized loss function can be consistent with the loss function used in the pre-training stage. The loss function used can refer to the above description. It can be seen that during fine-tuning training, the weight θ of the speech classification pre-training module can be further fine-tuned. finetune , in order to show better performance on multilingual recognition tasks. Specifically, the goal of fine-tuning training can be expressed as:

[0107]

[0108] in, is the weight of the speech classification pre-training module when fine-tuning training reaches the target fine-tuning state, L finetune It is the loss function used in the fine-tuning training stage, and θ is the weight of the module after speech classification pre-training.

[0109] From the above description, we can see that the loss function L finetune Please refer to the description of the pre-training loss function above. In specific implementation, when fine-tuning the training, the loss function L finetuneThe training state tends to be stable, as the fine-tuning training reaches the target fine-tuning state. At this time, for the speech classification pre-training module that has reached the target fine-tuning state, the mask and quantization modules in the speech classification pre-training module are turned off, and this configuration is used as the speech feature extraction module 1, so that the speech feature extraction module 1 is only used for feature extraction. In specific implementation, existing commonly used technical means can be used to freeze the parameters of the speech classification pre-training module, and to turn off the mask and quantization modules, which is based on whether the configuration can form the speech feature extraction module 1.

[0110] After fine-tuning the training and obtaining the speech feature extraction module 1, a language recognition basic model can be constructed, wherein the architecture of the language recognition basic model should be consistent with the above-mentioned language recognition model. Therefore, it can be seen from the above description that the language recognition basic model should also include the speech feature extraction module 1 and the corresponding language embedding extraction basic module and recognition classification basic module. After the language recognition basic model is trained to obtain the language recognition model, the language embedding extraction basic module can generate the corresponding language embedding extraction module 2. At the same time, the recognition classification basic module can generate the corresponding recognition classification module 3. Figure 2 and Figure 3 Detailed description of the embodiment of the basic model of language recognition and the speech feature extraction module 1 corresponding to the language recognition model is shown in FIG.

[0111] In one embodiment of the present invention, the language embedding extraction module 2 comprises a language local feature extraction layer, a statistical pooling layer and a language level feature extraction layer connected in sequence, wherein:

[0112] Connecting with the speech feature extraction module through the language local feature extraction layer;

[0113] It is connected to the recognition and classification module through the language-level feature extraction layer.

[0114] Figure 2 and Figure 3 An embodiment of the language embedding extraction module 2 is also shown in the figure. As can be seen from the figure, the language embedding extraction module 2 based on the x-vector model may include a language local feature extraction layer, a statistical pooling layer, and a language level feature extraction layer. It can be understood that when the language embedding extraction module 2 is based on the x-vector model, the language local feature extraction layer, the statistical pooling layer, and the language level feature extraction layer can adopt the forms commonly used in the existing x-vector model, such as Figure 3In the figure, it is shown that the language local feature extraction layer can be formed by 5 TDNN layers connected in series. The 5-layer TDNN extracts local features from the frame level of the speech signal to capture the short-term information of the speech. The statistical pooling layer can adopt the existing commonly used form to globally aggregate the local features of the entire speech segment, calculate the mean and standard deviation of the frame-level features, and thus generate a global speech embedding. Language level feature extraction layer: It is composed of a fully connected layer, a ReLU activation function layer, and a batch normalization layer. The language level feature extraction layer can extract the global speech embedding generated by the statistical pooling layer to generate language embedding information and load it into the recognition and classification module 3.

[0115] In one embodiment of the present invention, the recognition and classification module 3 includes a fully connected layer and a softmax layer, wherein:

[0116] When mapping language embedding information to language classification information, the language embedding information is processed in sequence using the fully connected layer and the softmax layer, and the language classification information is output through the softmax layer.

[0117] Figure 3 An embodiment of the identification and classification module 3 is shown in FIG. Figure 3 In the above example, Linear is a fully connected layer. Of course, the recognition and classification module 3 can also adopt other forms, which can be selected according to needs. In specific implementation, the language embedding extraction module 2 can fully aggregate the speech feature information extracted by the speech feature extraction module 1, and then extract the language embedding information of fixed length, and the recognition and classification module 3 can map the language embedding information into the language classification result.

[0118] The above gives an embodiment of constructing a basic model for language recognition. After that, a model training sample set should be constructed. Specifically, when constructing the model training sample set, it can be based on the above-mentioned fine-tuning training sample set. It can be seen from the above description that there is no label information for the fine-tuning training samples in the fine-tuning training sample set. In order to meet the needs of model training, the label information of each fine-tuning training sample should be determined by the commonly used technical means in the technical field. The label information is the reference language corresponding to the fine-tuning training sample. After that, a model training sample can be formed according to the fine-tuning training sample and the corresponding language label, and then the model training sample set can be constructed.

[0119] From the above description, it can be seen that when the language recognition basic model is trained using the model training sample set, the network parameters of the speech feature extraction module 1 are frozen, that is, the network parameters of the speech feature extraction module 1 are stopped from being updated. The model training is Figure 2The method of using the model training sample set to train the language recognition basic model can be consistent with the prior art, that is, the language recognition basic model is iteratively trained using the model training sample set. After each epoch of training, the model parameters are updated according to the loss function value. Here, the corresponding parameters of the language embedding extraction basic module and the recognition classification basic module are updated until the language recognition basic model is trained to reach the target training state. Thereafter, the language recognition basic model that has reached the target training state is configured as the language recognition model.

[0120] In one embodiment of the present invention, when the language recognition basic model is trained, the batch loss function used is:

[0121]

[0122] Among them, L c is the batch loss function value during model training, B is the batch size, is the correct category y of the i-th model training sample i The predicted probability of It represents the farthest positive sample distance between the i-th model training sample and other model training samples in the same batch. It represents the nearest negative sample distance between the i-th model training sample and other model training samples in the same batch, and δ is the distance boundary threshold.

[0123] When the batch size is B during model training, that is, B model training samples are loaded into the language recognition basic model at a time. For the B model training samples in the same batch, the cosine distance between every two model training samples is calculated, and the similarity between each pair of model training samples is evaluated. When calculating the cosine distance between two model training samples, the language embedding information corresponding to each model training sample is determined, and the cosine distance is calculated based on the language embedding information. The method and process for calculating the cosine distance can be consistent with the prior art and will not be repeated here. A mask matrix is ​​created using the language label of each model training sample to distinguish between positive and negative samples corresponding to two model training samples in the same batch.

[0124] The following is a description of the process of calculating the above loss function value, specifically:

[0125] Assume that there is a batch of model training samples, and the output dimension obtained by the language recognition basic model is (32, 512), where 32 is the batch size and 512 is the feature dimension. In this case, the batch size B is 32. Calculate the batch loss function value L c The steps are as follows:

[0126] 1) Calculate the distance matrix: Calculate the cosine distance between every two model training samples in the same batch to obtain a distance matrix cdist of size (32, 32), where cdist[i][j] represents the cosine distance between the i-th model training sample and the j-th model training sample. That is, for any model training sample, the corresponding distance matrix cdist between the current model training sample and other model training samples can be calculated.

[0127] It should be noted that the diagonal elements in the distance matrix cdist are the cosine distances between the i-th model training samples and the i-th model training samples. At this time, the cosine distance should be 0, and the diagonal elements in the distance matrix cdist should be 0.

[0128] 2) Create a mask matrix: Create a mask matrix mask_pos according to the language label of each model training sample. The mask matrix mask_pos is a matrix of size (32, 32) used to indicate which samples are of the same type (positive samples, in this case, the language labels are the same), where a mask value of 1 indicates that they are of the same type, and a mask value of 0 indicates that they are of different types (different language labels). For example, assuming that the 0th model training sample and the 1st model training sample are of the same type, then the values ​​of mask_pos[0][1] and mask_pos[1][0] are both 1, and mask_pos[0][1] and mask_pos[1][0] are the values ​​of the corresponding positions in the corresponding mask matrix. That is, for any model training sample, the mask matrix mask_pos corresponding to the current model training sample and other model training samples can be calculated.

[0129] It should be noted that the diagonal elements in the mask matrix mask_pos are the mask values ​​between the i-th model training sample and the i-th model training sample. At this time, due to the same class of language labels, the diagonal elements in the mask matrix mask_pos should be 1.

[0130] 3) Calculate the farthest positive sample distance: Use the mask matrix mask_pos to calculate the farthest positive sample distance of each model training sample, that is, the sample distance farthest from the current model training sample among all similar model training samples. The calculated sample distance result is a vector of size 32.

[0131] It should be noted that when calculating the distance of the farthest positive sample, the distance matrix cdist can be multiplied by the mask matrix mask_pos, where the multiplication of elements specifically refers to the multiplication of each corresponding matrix element. Thereafter, the maximum value of each row is determined in the result matrix of the multiplication. After determining the maximum value of each row, the maximum value of all row maximum values ​​is determined, which is the farthest positive sample distance between the i-th model training sample and other model training samples in the same batch.

[0132] 4) Calculate the nearest negative sample distance: Use the mask matrix mask_pos to calculate the nearest negative sample distance of each model training sample, that is, the sample distance closest to the current model training sample among all model training samples of different classes; the calculated sample distance result is also a vector of size 32.

[0133] It should be noted that when calculating the distance of the nearest negative sample, each element in the mask matrix mask_pos is first multiplied by an infinite value to obtain the mask calculation matrix. After that, the distance matrix cdist can be multiplied with the mask calculation matrix. The meaning of the multiplication operation here can refer to the above description and will not be repeated here. After that, in the result matrix of the multiplication, the minimum value of each row is determined. After determining the minimum value of each row, the minimum value of all row minimum values ​​is determined, which is the nearest negative sample distance between the i-th model training sample and other model training samples in the same batch.

[0134] In specific implementation, the farthest positive sample distance Used to evaluate the similarity between similar samples, the nearest negative sample distance Used to evaluate the similarity between samples of different categories. For the distance boundary threshold δ, it can be specifically set to 0.2, that is, to ensure that the distance between positive samples and negative samples is at least 0.2, to encourage the formation of a significant distance difference between samples of the same category and samples of different categories, and enhance the ability to distinguish features of different categories.

[0135] From the above batch loss function, it can be seen that the language classification loss and vector loss can be optimized at the same time to improve the accuracy of language classification and the distinguishability of embedding. It should be noted that when training the model, after training the basic model of language recognition using the model training samples of each batch, the network parameters of the basic model of language recognition can be updated. When the batch loss function value tends to be stable, it can be considered that the basic model of language recognition has reached the target training state. Of course, the target training state can also be other situations, which can be selected according to needs.

[0136] Specifically, after obtaining the language recognition model, the language category of the target speech audio can be recognized. In order to prove the effectiveness of the language recognition model of the present invention in identifying the language category of the target speech audio, the following can be compared with the existing language recognition model, specifically:

[0137] Table 1 Multilingual recognition indicators under different methods

[0138]

[0139] Specifically, for the above embodiment in which the benchmark language set includes 15 benchmark languages, in specific implementation, Fulah and Oromo can be used for language recognition when the training data is unbalanced, that is, the other 13 languages ​​can be used as mandatory benchmark languages. Among the mandatory 13 benchmark languages, the quantity distribution in the model training sample set can be as follows: Figure 4 In addition, Figure 4 The quantity distribution of other cases is also shown in .

[0140] The Fleurs validation set mentioned above can generally be constructed based on the validation data in the open source language recognition dataset Fleurs. When generating the validation dataset based on the open source language recognition dataset Fleurs, it is also necessary to sample the same data processing as the training dataset in the open source language recognition dataset Fleurs, such as the processing into mono, 16K sampling rate, 16bit bit depth, and then perform data enhancement processing. For details, please refer to the above instructions. For the specific construction method of the Fleurs test set and the Slingua validation set, please refer to the Fleurs validation set construction instructions here, which will not be repeated here.

[0141] Table 1 above shows an embodiment of comparing different language recognition methods using 13 benchmark languages. "Our method" in Table 1 is the language recognition model of the present invention. As can be seen from Table 1, compared with other mainstream language recognition methods, the language recognition model of the present invention has achieved the best results on the three open source test sets.

[0142] In Table 1, XLS-R+Linear represents the weight of the open source pre-trained model XLS-R based on the wav2vec2.0 structure, and then connects the fully connected layer and the Softmax activation function for classification. FBANK+x-vector represents the extraction of the FBANK features of speech and then connects the x-vector structure based on the Time Delay Neural Network (TDNN) for classification. FBANK+F-TDNN represents the extraction of the FBANK features of speech and then connects the x-vector structure based on the Factorized Time-Delay Neural Network (F-TDNN) for classification. FBANK+ECAPA represents the extraction of the FBANK features of speech and then connects the x-vector structure based on the Emphasized Channel Attention-based Time-Delay Neural Network (ECAPA-TDNN) for classification. XLS-R+x-vector represents freezing the weight of the open source XLS-R part and only updating the weight parameters of the x-vector part to classify the language.

[0143] It can be understood that, except for the language recognition model of the present invention, the models corresponding to other methods in Table 1 are all language recognition model methods existing in the prior art. The language recognition situation of the model is well known to those skilled in the art and will not be described in detail here.

[0144] In the specific implementation, we can continue to explore the balance of training data. At this time, 20,000 data are selected from the above 13 languages, and then two languages ​​with less data, Fulah and Oromo, are added to construct an experiment for recognizing 15 languages. Among them, the training set, validation set and test set of Fulah have 3236, 273 and 660 data respectively, and the training set, validation set and test set of Oromo have 1701, 19 and 41 data respectively. The experimental results are shown in Table 2.

[0145] Table 2 Comparison of mainstream methods when training data is unbalanced.

[0146]

[0147] It can be seen from Table 2 that, compared with other mainstream methods, the language recognition model of the present invention can also achieve the best results on the Fleurs validation set and test set respectively when the training data is unbalanced, which proves the effectiveness of the proposed model.

[0148] From the above description, a speech language recognition system can be obtained. In one embodiment of the present invention, a language recognition device is included, and the above-mentioned language recognition model is deployed in the language recognition device, wherein:

[0149] The language recognition model is used to perform language recognition processing on the target speech audio, and after the language recognition processing, language classification information corresponding to the target speech audio is generated.

[0150] Specifically, the language recognition device can be an existing commonly used computer device, and the existing technical means can be used to deploy the above-mentioned language recognition model in the language recognition device. After that, the language category recognition and classification of the target speech audio can be realized. The specific situation of language category recognition and classification can be referred to the above description, which will not be repeated here.

[0151] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited to this, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of this patent. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A method for recognizing speech and language categories, characterized in that: The identification method comprises: Provide target speech audio, and load the target speech audio into a pre-built language recognition model, so as to perform language recognition processing on the target speech audio using the language recognition model, wherein: When the language recognition model performs language recognition processing on the target speech audio, it at least includes speech feature extraction processing, language embedding extraction processing and recognition classification processing performed in sequence, wherein: After performing speech feature extraction processing, corresponding speech feature information is generated based on the target speech audio; After executing the language embedding extraction process, generating corresponding language embedding information based on the speech feature information; After performing the recognition and classification processing, the language embedding information is mapped to language classification information representing the language corresponding to the target speech audio, wherein the language classification information is a reference language in the reference language set used when constructing the language recognition model.

2. The method for recognizing speech and language categories according to claim 1, characterized in that: The language recognition model includes a speech feature extraction module, a language embedding extraction module and a recognition classification module connected in sequence, wherein: When performing language recognition processing on the target speech audio, the speech feature extraction module is used to perform speech feature extraction processing, and the generated speech feature information is loaded into the language embedding extraction module; Using the language embedding extraction module to perform language embedding extraction processing, and loading the generated corresponding language embedding information into the recognition and classification module; The recognition and classification module is used to perform recognition and classification processing and generate language classification information.

3. The method for recognizing speech and language categories according to claim 2, characterized in that: The speech feature extraction module is generated based on the wav2vec2.0 model, and the language embedding extraction module is generated based on the x-vector model.

4. The method for recognizing speech and language categories according to claim 3, characterized in that: When building a language recognition model, the construction method includes: Build and generate speech feature extraction module based on wav2vec2.0 model; After the speech feature extraction module is generated, a language recognition basic model is constructed, wherein when the language recognition basic model is constructed, the speech feature extraction module is connected to the language embedding extraction basic module, and the language embedding extraction basic module is connected to the recognition classification basic module; A model training sample set is constructed, and the model training sample set is used to train the language recognition basic model until the language recognition basic model training reaches a target training state, and then the language recognition basic model that reaches the target training state is configured as the language recognition model, wherein: The model training sample set includes a number of model training samples. When the model training sample set is used to train the language recognition basic model, the network parameters of the speech feature extraction module are frozen, and the speech feature extraction module is used to extract the speech feature information of the model training samples, and the extracted speech feature information is loaded into the language embedding extraction basic module, wherein: The model training sample includes training speech audio and label information corresponding to the training speech audio, wherein the label information is the reference language corresponding to the training speech audio; The training speech audio and the target speech audio have the same speech feature parameters, where the speech feature parameters include speech channels, sampling rate and / or bit depth.

5. The method for recognizing speech and language categories according to claim 4, characterized in that: When building a speech feature extraction module, include: Construct a speech classification module based on the wav2vec2.0 model, and perform pre-training and fine-tuning training on the speech classification module in turn. After the fine-tuning training, generate a speech feature extraction module based on the speech classification module, wherein: When pre-training the speech classification module, a pre-training sample set is constructed, and the speech classification module is pre-trained using the pre-training sample set based on a self-supervised learning training method, and when the pre-training reaches a pre-training target state, a speech classification post-pre-training module is generated, wherein the pre-training sample set includes a plurality of pre-training samples, and the pre-training samples are unlabeled speech audio; During fine-tuning training, a fine-tuning training sample set is constructed, and the speech classification pre-trained module is trained using the fine-tuning training sample set. When the speech classification pre-trained module is fine-tuned to reach a target fine-tuning state, the speech classification pre-trained module that has reached the target fine-tuning state is configured as a speech feature extraction module, wherein: The fine-tuning training sample set includes fine-tuning training samples in multiple languages, and the fine-tuning training samples are unlabeled speech audio.

6. The method for recognizing speech and language categories according to claim 5, characterized in that: When pre-training the speech classification module, the loss function used is: Among them, L pretrain is the pre-training loss function value, z i is the potential representation feature of the i-th time step generated based on the current pre-training sample, c i is the potential representation feature z based on the i-th time step i The generated context features, c j It is the context feature generated based on the jth negative sample in the negative sample set, M is the target time step set of the current pre-training sample, N is the negative sample set of the current pre-training sample, and sim is the cosine similarity calculation.

7. The method for recognizing speech and language categories according to claim 4, characterized in that: When training the basic language recognition model, the batch loss function used is: Among them, L c is the batch loss function value during model training, B is the batch size, is the correct category y of the i-th model training sample i The predicted probability of It represents the farthest positive sample distance between the i-th model training sample and other model training samples in the same batch. It represents the nearest negative sample distance between the i-th model training sample and other model training samples in the same batch, and δ is the distance boundary threshold.

8. The method for recognizing speech and language categories according to any one of claims 3 to 7, characterized in that: The language embedding extraction module includes a language local feature extraction layer, a statistical pooling layer and a language level feature extraction layer connected in sequence, wherein: Connecting with the speech feature extraction module through the language local feature extraction layer; It is connected to the recognition and classification module through the language-level feature extraction layer.

9. The method for recognizing speech and language categories according to any one of claims 2 to 8, characterized in that: The recognition and classification module includes a fully connected layer and a softmax layer, wherein: When mapping language embedding information to language classification information, the language embedding information is processed in sequence using the fully connected layer and the softmax layer, and the language classification information is output through the softmax layer.

10. A speech and language recognition system, characterized in that: The invention comprises a language recognition device, and a language recognition model as described in any one of claims 1 to 9 is deployed in the language recognition device, wherein: The language recognition model is used to perform language recognition processing on the target speech audio, and after the language recognition processing, language classification information corresponding to the target speech audio is generated.

Images