A method for recognizing a speech language category

Abstract

The application relates to a speech language category recognition method, which comprises the following steps: providing target speech audio and loading the target speech audio into a pre-constructed language recognition model to perform language recognition processing on the target speech audio by using the language recognition model; when the language recognition model performs language recognition processing on the target speech audio, at least the following processes are sequentially performed: speech feature extraction processing, language embedding extraction processing and recognition classification processing; after the speech feature extraction processing is performed, corresponding speech feature information is generated based on the target speech audio; after the language embedding extraction processing is performed, corresponding language embedding information is generated based on the speech feature information; after the recognition classification processing is performed, the language embedding information is mapped into language classification information representing the language of the target speech audio. The application can efficiently recognize multiple languages, and improves the accuracy and recognition efficiency of language recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to a recognition method, in particular a speech language class recognition method. BACKGROUND

[0002] Language recognition refers to the process of automatically recognizing a given speech signal to determine the language class of the given speech signal. Language recognition plays a crucial role in various applications such as multilingual speech recognition, machine translation, and call routing in customer service centers. In these scenarios, machine learning techniques are often used to train on large datasets to accurately distinguish between different languages. With the increasing trend of internationalization, language exchange between countries is gradually increasing, and language recognition technology has become an indispensable part of modern speech processing systems.

[0003] Similar to other speech signal processing classification tasks, early language recognition systems rely on traditional hand-crafted features such as MFCC and FBANK. These features mainly describe the acoustic characteristics of speech and cannot effectively capture linguistic features in sound. Moreover, they lack flexibility and cannot adapt to complex speech data. Subsequently, machine learning algorithms such as GMM, SVM, and decision trees can be used for speech classification based on traditional acoustic features. However, this classification method is susceptible to environmental interference and has poor generalization ability.

[0004] With the development of deep learning, deep neural networks have shown excellent non-linear feature extraction capabilities in various speech signal processing tasks. Although deep neural networks have achieved good performance in language recognition, they still rely on traditional hand-crafted features. Moreover, most acoustic feature-based language recognition models do not support end-to-end training, which limits the optimization of language recognition systems to some extent.

[0005] In recent years, self-supervised learning has emerged as a promising alternative in the field of language recognition. Unlike traditional methods, self-supervised learning uses a large amount of unlabeled data for pre-training and fine-tunes on labeled task-specific data. This approach enables the learning of speech representations without the need for extensive human labeling. Such models can capture rich and complex features, surpassing the limitations of hand-crafted features and providing a more robust and generalizable understanding of data.

[0006] It should be noted that self-supervised learning involves a large number of parameters, which increases the complexity of training. Moreover, relatively simple classification strategies cannot effectively capture the speech features extracted by self-supervised learning. In addition, there may be domain shift issues between pre-training data and downstream task (i.e., language recognition task) data. For example, pre-training data may not include the languages required for recognition in downstream tasks, which can affect the performance of language recognition.

[0007] In addition, using traditional manual features and conventional classification algorithms, when the training data is balanced, the performance can meet the needs of the industry, but when dealing with unbalanced training data, the accuracy and generalization ability of the model decrease sharply, especially for low-resource language recognition, which is a serious challenge.

[0008] The application file with publication number CN113889090A discloses a multi-task learning multi-language recognition model, which uses unsupervised pre-training and adaptive pre-training to train the multi-language model. However, the wav2vec model used in the application has a large number of parameters, which increases the complexity of training; and when the language samples in the application are unbalanced, the classification effect will be affected.

[0009] The application file with publication number CN112133291A discloses a language recognition model obtained by a discriminant probability and a prediction language label training generation network, a neural network and a discriminant network, but the application needs an additional data preprocessing step, and when the model faces a different actual scene from the training data distribution, the generalization ability may be challenged.

[0010] In summary, there are still deficiencies in the recognition of speech language categories, which cannot meet the current application requirements. SUMMARY

[0011] The purpose of the present application is to overcome the deficiencies in the prior art, and to provide a speech language category recognition method which can efficiently recognize multiple languages and improve the accuracy and efficiency of language recognition.

[0012] According to the technical scheme provided by the present application, a speech language category recognition method comprises:

[0013] A target speech audio is provided and loaded into a pre-constructed language recognition model to perform language recognition processing on the target speech audio using the language recognition model, wherein

[0014] When the language recognition model performs language recognition processing on the target speech audio, it at least includes sequentially executed speech feature extraction processing, language embedding extraction processing and recognition classification processing, wherein

[0015] After performing the speech feature extraction processing, the corresponding speech feature information is generated based on the target speech audio;

[0016] After performing the language embedding extraction processing, the corresponding language embedding information is generated based on the speech feature information;

[0017] After performing the recognition classification processing, the language embedding information is mapped to language classification information representing a language of the target voice audio, wherein the language classification information is one of the reference languages in the reference language set used when constructing the language recognition model.

[0018] The language recognition model comprises a speech feature extraction module, a language embedding extraction module, and a recognition classification module connected in sequence, wherein,

[0019] When performing language recognition processing on the target voice audio, the speech feature extraction module is used to perform speech feature extraction processing, and the generated speech feature information is loaded into the language embedding extraction module;

[0020] The language embedding extraction module is used to perform language embedding extraction processing, and the generated corresponding language embedding information is loaded into the recognition classification module;

[0021] The recognition classification module is used to perform recognition classification processing and generate language classification information.

[0022] The speech feature extraction module is constructed based on a wav2vec2.0 model, and the language embedding extraction module is constructed based on an x-vector model.

[0023] When constructing the language recognition model, the construction method comprises:

[0024] The speech feature extraction module is constructed based on a wav2vec2.0 model;

[0025] After generating the speech feature extraction module, a language recognition basic model is constructed, wherein when constructing the language recognition basic model, the speech feature extraction module is connected with the language embedding extraction basic module, and the language embedding extraction basic module is connected with the recognition classification basic module;

[0026] A model training sample set is constructed, and the language recognition basic model is trained using the model training sample set until the language recognition basic model is trained to a target training state, after which the language recognition basic model that has reached the target training state is configured as the language recognition model, wherein,

[0027] The model training sample set comprises a plurality of model training samples, and when training the language recognition basic model using the model training sample set, the network parameters of the speech feature extraction module are frozen, and the speech feature information of the model training sample is extracted using the speech feature extraction module, and the extracted speech feature information is loaded into the language embedding extraction basic module, wherein,

[0028] The model training sample comprises a training voice audio and label information corresponding to the training voice audio, and the label information is a reference language corresponding to the training voice audio;

[0029] The training voice audio and the target voice audio have the same voice feature parameters, and the voice feature parameters include voice channels, sampling rates and / or bit depths.

[0030] When constructing the voice feature extraction module, the following steps are included:

[0031] A voice classification module based on a wav2vec2.0 model is constructed, and the voice classification module is sequentially pre-trained and fine-tuned, and after fine-tuning, a voice feature extraction module is generated based on the voice classification module, wherein

[0032] When pre-training the voice classification module, a pre-training sample set is constructed, and the voice classification module is pre-trained based on a self-supervised learning training method using the pre-training sample set, and when the pre-training reaches a pre-training target state, a voice classification pre-training module is generated, wherein the pre-training sample set includes a plurality of pre-training samples, and the pre-training samples are unlabeled voice audio;

[0033] During fine-tuning, a fine-tuning sample set is constructed, and the voice classification pre-training module is trained using the fine-tuning sample set, and when the voice classification pre-training module is fine-tuned to reach a target fine-tuning state, the voice classification pre-training module that reaches the target fine-tuning state is configured as the voice feature extraction module, wherein

[0034] The fine-tuning sample set includes fine-tuning samples of multiple languages, and the fine-tuning samples are unlabeled voice audio.

[0035] When pre-training the voice classification module, the loss function used is:

[0036]

[0037] wherein, L pretrain is the pre-training loss function value, z i is the latent representation feature of the i-th time step based on the current pre-training sample, c i is the context feature generated based on the latent representation feature z i of the i-th time step, c j is the context feature generated based on the j-th negative sample in the negative sample set, M is the target time step set of the current pre-training sample, N is the negative sample set of the current pre-training sample, and sim is the cosine similarity calculation.

[0038] When training the language recognition basic model, the batch loss function used is:

[0039]

[0040] wherein, Lc is the batch loss function value during model training, B is the batch size, is the correct class y of the i-th model training sample i is the prediction probability of the i-th model training sample, represents the farthest positive sample distance of the i-th model training sample from other model training samples in the same batch, represents the nearest negative sample distance of the i-th model training sample from other model training samples in the same batch, and δ is the distance boundary threshold.

[0041] The language embedding extraction module comprises a language local feature extraction layer, a statistical pooling layer and a language level feature extraction layer connected in sequence, wherein,

[0042] The language local feature extraction layer is connected with the speech feature extraction module;

[0043] The language level feature extraction layer is connected with the recognition classification module.

[0044] The recognition classification module comprises a full connection layer and a softmax layer, wherein,

[0045] When mapping the language embedding information into language classification information, the full connection layer and the softmax layer are used to process the language embedding information in sequence, and the language classification information is output through the softmax layer.

[0046] A speech language category recognition system comprises a language recognition device, and the language recognition model described above is deployed in the language recognition device, wherein,

[0047] The language recognition model is used to perform language recognition processing on the target speech audio, and the language classification information corresponding to the target speech audio is generated after the language recognition processing.

[0048] The advantages of the present application are: when constructing the language recognition model, the speech feature extraction module after pre-training and self-adaptive fine-tuning is used as the feature extractor in the language recognition model, so that the language recognition model can extract features closer to the downstream task; at the same time, the speech feature extraction module is connected with the language embedding extraction module based on the x-vector model, which can fully capture the complex relationship between features, so that the recognition effect of language recognition is better.

[0049] During the construction of the language recognition model, the parameter update of the speech feature extraction module based on the wav2vec2.0 model is frozen, which reduces the trainable parameters and reduces the time cost of training, i.e. the training time of the language recognition model is shorter

[0050] The language recognition model does not depend on a large amount of language label data, reduces the human and time cost of obtaining label data, and the self-supervised model learns general features in the voice, improves the generalization ability of the model for language recognition, and the data enhancement processing method is used to disturb the training data to improve the precision, recall rate, F1 value and other training indicators of the model. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 An embodiment flow diagram for the speech language category recognition of the present application.

[0052] Figure 2 An embodiment diagram of the language recognition model constructed by the present application.

[0053] Figure 3 An embodiment diagram of the language recognition model of the present application.

[0054] Figure 4 An embodiment diagram of the reference language set of the present application.

[0055] BRIEF DESCRIPTION OF DRAWINGS: 1-voice feature extraction module, 2-language embedding extraction module, 3-recognition classification module. DETAILED DESCRIPTION

[0056] The present application will be further described below in conjunction with specific drawings and embodiments.

[0057] In order to efficiently recognize multiple languages, improve the accuracy and efficiency of language recognition, the present application provides a language recognition method, specifically, the recognition method comprises:

[0058] A target voice audio is provided and loaded into a pre-constructed language recognition model to perform language recognition processing on the target voice audio using the language recognition model, wherein

[0059] The language recognition model performs language recognition processing on the target voice audio, at least including sequentially executed voice feature extraction processing, language embedding extraction processing and recognition classification processing, wherein

[0060] After performing the voice feature extraction processing, the corresponding voice feature information is generated based on the target voice audio;

[0061] After performing the language embedding extraction processing, the corresponding language embedding information is generated based on the voice feature information;

[0062] After performing the recognition classification processing, the language embedding information is mapped to language classification information representing the language corresponding to the target voice audio, wherein the language classification information is one of the reference languages in the reference language set used when constructing the language recognition model.

[0063] It should be noted that the speech language category recognition is to recognize the language category of the speech, such as recognizing the speech category of the speech as Chinese, English, French, etc. The meaning of the speech language category is consistent with the prior art. Figure 1 An embodiment of the speech language category recognition of the present application is shown in FIG. 1. As shown in the figure, when the speech language category recognition is performed, a language recognition model needs to be constructed. Thereafter, the constructed language recognition model can be used to perform the language category recognition on the target speech audio, wherein the target speech audio is the speech to be recognized in the language category.

[0064] Figure 1 An embodiment of the speech recognition on the target speech audio using the language recognition model is shown in FIG. 2. As shown in the figure, when the language category recognition is performed, the language recognition model is used to perform the language recognition processing on the target speech audio. Specifically, the language recognition processing should at least include the speech feature extraction processing, the language embedding extraction processing, and the recognition classification processing. After the speech feature extraction processing is performed on the target speech audio, the speech feature information corresponding to the target speech audio can be generated. Thereafter, the language embedding extraction processing is performed on the speech feature information, the corresponding language embedding information can be generated. After the recognition classification processing is performed on the language embedding information, the language classification information can be generated.

[0065] It should be noted that based on the generated language classification information, the language category of the target speech audio can be determined, that is, the language category recognition of the target speech audio is realized. When the language recognition model is constructed, a set of reference languages should be configured, such as using the speech audio of each reference language in the configured set of reference languages for training. In an embodiment of the present application, when the language category recognition of the target speech audio is performed, the language category of the target speech audio is recognized as one of the reference languages in the set of reference languages. Specifically, the reference languages in the set of reference languages can include Chinese, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Fur language, and / or Oromo language. At this time, after the language category recognition of the target speech audio is performed, the language category of the target speech audio can be recognized as one of Chinese, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Fur language, and / or Oromo language.

[0066] In a specific implementation, for a target voice audio loaded into the language recognition model, a voice feature parameter of the target voice audio is configured, wherein the voice feature parameter includes a voice channel, a sampling rate and / or a bit depth, for example, the voice channel of the target voice audio should be a single channel, the sampling rate can be 16K, and the bit depth can be 16 bits; the voice feature parameter of the target voice audio can be set according to actual language recognition parameters, which will not be illustrated one by one here.

[0067] In an embodiment of the present application, the language recognition model includes a voice feature extraction module 1, a language embedding extraction module 2 and a recognition classification module 3 connected in sequence, wherein,

[0068] When the target voice audio is subjected to language recognition processing, the voice feature extraction module 1 is used to perform voice feature extraction processing, and the generated voice feature information is loaded into the language embedding extraction module 2.

[0069] The language embedding extraction module 2 is used to perform language embedding extraction processing, and the generated corresponding language embedding information is loaded into the recognition classification module 3.

[0070] The recognition classification module 3 is used to perform recognition classification processing, and language classification information is generated.

[0071] Figure 2 And Figure 3 An embodiment of the language recognition model is shown in FIG. 1, as shown in the figure, the language recognition model can include a voice feature extraction module 1, a language embedding extraction module 2 and a recognition classification module 3, wherein the voice feature extraction module 1 can perform the voice feature extraction processing in the language recognition processing as described above, the language embedding extraction module 2 can perform the language embedding extraction processing in the language recognition processing as described above, and the recognition classification module 3 can perform the recognition classification processing in the language recognition processing as described above.

[0072] In an embodiment of the present application, the voice feature extraction module 1 is generated based on a wav2vec2.0 model, and the language embedding extraction module 2 is generated based on an x-vector model. Of course, the voice feature extraction module 1 and the language embedding extraction module 2 can also adopt other forms. Hereinafter, the method for constructing the language recognition model of the present application will be specifically explained and described by taking the voice feature extraction module 1 generated based on the wav2vec2.0 model and the language embedding extraction module 2 generated based on the x-vector model as an example.

[0073] In an embodiment of the present application, when the language recognition model is constructed, the construction method includes:

[0074] The voice feature extraction module 1 is constructed based on the wav2vec2.0 model;

[0075] After the speech feature extraction module 1 is generated, a language recognition basic model is constructed, wherein when the language recognition basic model is constructed, the speech feature extraction module is connected with a language embedding extraction basic module, and the language embedding extraction basic module is connected with a recognition and classification basic module;

[0076] A model training sample set is constructed, and the language recognition basic model is trained using the model training sample set until the language recognition basic model is trained to a target training state, after which the language recognition basic model that reaches the target training state is configured as a language recognition model, wherein

[0077] The model training sample set includes a plurality of model training samples, and when the language recognition basic model is trained using the model training sample set, the network parameters of the speech feature extraction module are frozen, and the speech feature information of the model training sample is extracted using the speech feature extraction module, and the extracted speech feature information is loaded into the language embedding extraction basic module, wherein

[0078] The model training sample includes a training speech audio and label information corresponding to the training speech audio, and the label information is a reference language corresponding to the training speech audio;

[0079] The training speech audio and the target speech audio have the same speech feature parameters, and the speech feature parameters include speech channels, sampling rates, and / or bit depths.

[0080] In specific implementation, when constructing the language recognition model, the speech feature extraction module 1 should be constructed first, and then the language recognition basic model is constructed, and the language recognition basic model can be used to construct the required language recognition model. The process of constructing the speech feature extraction module 1 is described below.

[0081] In an embodiment of the present application, when constructing the speech feature extraction module 1, the following steps are included:

[0082] A speech classification module based on the wav2vec2.0 model is constructed, and the speech classification module is sequentially pre-trained and fine-tuned, and after fine-tuning, the speech feature extraction module 1 is generated based on the speech classification module, wherein

[0083] When the speech classification module is pre-trained, a pre-training sample set is constructed, the speech classification module is pre-trained using the pre-training sample set based on a self-supervised learning training method, and when the pre-training reaches a pre-training target state, a speech classification pre-training module is generated, wherein the pre-training sample set includes a plurality of pre-training samples, and the pre-training samples are unlabeled speech audios;

[0084] During fine-tuning training, a fine-tuning training sample set is constructed, and the speech classification pre-trained module is trained by using the fine-tuning training sample set. When the speech classification pre-trained module is fine-tuned to the target fine-tuned state, the speech classification pre-trained module in the target fine-tuned state is configured as the speech feature extraction module, wherein,

[0085] The fine-tuning training sample set includes fine-tuning training samples of multiple languages, and the fine-tuning training samples are unlabeled speech audios.

[0086] As can be seen from the above description, the wav2vec2.0 model can be a commonly used recognition model, and the wav2vec2.0 model can be consistent with the existing model. For example, the existing wav2vec2.0 model can be used to directly construct a speech classification module, that is, the network parameters of the speech classification module can be the default network parameters of the wav2vec2.0 model. The constructed speech classification module should be pre-trained and fine-tuned in sequence, as shown in Figure 2 After fine-tuning training, a speech feature extraction module 1 can be generated. The specific methods and processes of pre-training and fine-tuning training are described below.

[0087] When the speech classification module is pre-trained, a pre-training sample set needs to be constructed, wherein the pre-training sample set includes pre-training samples, and each pre-training sample is an unlabeled speech audio. In specific implementation, the speech audio can be obtained from the network, a public speech data set, etc. by using the existing technology, and used as a pre-training sample. As can be seen, since no labeled data needs to be made, the construction of the pre-training sample set will not increase the workload. After the pre-training sample set is constructed, the speech classification module can be trained by using the pre-training sample set. The specific training method can be consistent with the existing technology.

[0088] Since the speech classification module is constructed and generated based on the wav2vec2.0 model, the speech classification module has the characteristics of the wav2vec2.0 model. For any pre-training sample, the speech classification module first extracts features of the pre-training sample by using a 7-layer convolutional feature extractor, and outputs latent representation information of the speech, wherein the output of each layer of convolution in the convolutional feature extractor is used as the input of the next layer, the step length of each layer of convolution is (5, 2, 2, 2, 2, 2, 2) respectively, the convolution kernel size is (10, 3, 3, 3, 3, 2, 2) respectively, and finally a three-dimensional vector with a channel number of 512 is output, that is, the latent representation information is a three-dimensional vector with a channel number of 512.

[0089] The latent representation information output by the convolution feature extractor is mapped to a channel number of 1024 dimensions by a full connection layer; thereafter, after relative position coding, it is input into a context feature encoder composed of 24 layers of transformers, wherein each layer of the transformer in the context feature encoder comprises a multi-head attention mechanism and a feedforward neural network to capture global context information in the speech signal; after encoding by the context feature encoder, a three-dimensional vector with a channel number of 1024 can be obtained, and the three-dimensional vector with a channel number of 1024 is the context information.

[0090] In order to predict the target, the wav2vec2.0 model randomly masks a preset number of time steps of features output by the convolution feature extractor by a masking strategy, and reconstructs the features masked from the remaining context information, forcing the wav2vec2.0 model to pay more attention to global and local context information, so as to learn rich feature representations without display labels, wherein the preset number of time step features can be 10; a time step refers to that the pre-training sample is divided into small time windows by the convolution feature extractor, each time window corresponds to a time step, and feature extraction is performed on each time step to generate a group of feature vectors, thereby generating corresponding latent representation information.

[0091] In an embodiment of the present application, when the speech classification module is pre-trained, the loss function used is:

[0092]

[0093] Wherein, L pretrain is the pre-training loss function value, z i is the latent representation feature of the i-th time step based on the current pre-training sample, c i is the context feature generated based on the latent representation feature z i of the i-th time step, c j is the context feature generated based on the j-th negative sample in the negative sample set, M is the target time step set of the current pre-training sample, N is the negative sample set of the current pre-training sample, and sim is the cosine similarity calculation.

[0094] As can be known from the above wav2vec2.0 model, the latent representation feature z i of the i-th time step is a feature vector of the i-th time step, which is mapped to a channel number of 1024 dimensions by a full connection layer and is a feature before relative position coding. The context feature c i is a feature generated by the latent representation feature z i of the i-th time step after relative position coding, context feature encoder feature capturing, and masking strategy (mask).i It is a three-dimensional vector with 1024 channels.

[0095] The target time step set M consists of time steps selected from all pre-trained samples in a batch, corresponding to the positive sample features c that the wav2vec2.0 model is expected to learn. i For the latent representation feature z at the i-th time step i Construct a negative sample set N, which contains the latent representation features z at the i-th time step. i The corresponding negative sample features are typically derived from the latent representation features z at the i-th time step in other pre-trained sample data from the same batch. i The latent representation features corresponding to different time steps allow for the determination of corresponding positive and negative samples at each time step.

[0096] To effectively train the speech classification module based on the wav2vec2.0 model, the aforementioned pre-training loss can be sampled. After training the speech classification module once using the pre-training sample set, the corresponding pre-training loss function value L is calculated based on the aforementioned loss function. pretrain Of course, when pre-training the speech classification module, some pre-training configuration parameters should also be set to update the network parameters of the speech classification module. Specifically, during the iterative pre-training process, when the pre-training loss function value L... pretrain Once the process stabilizes, the pre-training of the speech classification module can be terminated, and the pre-trained speech classification module can then be generated.

[0097] As can be seen from the above explanation, pre-training is... Figure 2 Self-supervised pre-training is performed using a large amount of unlabeled data, which refers to the unlabeled speech and audio samples mentioned above as the pre-training samples. Pre-training reaches the pre-training target state, specifically referring to the training loss function value L. pretrain The training state is approaching stability. Of course, other loss functions and corresponding pre-training target states can be used when pre-training the speech classification module. The specific options can be selected as needed, and will not be listed here.

[0098] After obtaining the pre-trained speech classification module through the above pre-training, the speech classification pre-trained module should be fine-tuned. During fine-tuning, a fine-tuning training sample set needs to be constructed. The fine-tuning training sample set includes fine-tuning training samples in multiple languages. The languages ​​corresponding to the fine-tuning training samples in the fine-tuning training sample set should be consistent with the benchmark languages ​​in the above-mentioned benchmark language set. In addition, the fine-tuning training samples are unlabeled speech audio.

[0099] When constructing the required fine-tuning training sample set, a feasible method is to determine the open source language recognition data set Fleurs and the open source language recognition data set Slingua as the fine-tuning training sample source data set. Thereafter, one piece of training data in the open source language recognition data set Fleurs and one piece of training data in the open source language recognition data set Slingua can be determined as a corresponding fine-tuning training source sample. For example, one piece of voice data in the training data part of the open source language recognition data set Fleurs can be determined as a fine-tuning training source sample. Similarly, one piece of voice data in the training data part of the open source language recognition data set Slingua can be determined as another fine-tuning training source sample. In specific implementation, all data in the training data part of the open source language recognition data set Fleurs and the open source language recognition data set Slingua can be selected as corresponding fine-tuning training source samples, or part of the data can be selected as fine-tuning training source samples. It should be noted that when selecting the fine-tuning training source samples, all reference languages in the reference language set should be included.

[0100] The fine-tuning training source samples extracted and generated above should be processed to have the same speech feature parameters as the target voice audio, for example, to be processed into single-channel, 16k sampling rate, and 16-bit depth voice audio. At this time, each fine-tuning training source sample can generate a corresponding fine-tuning training source pre-processing sample after processing.

[0101] In order to generate corresponding fine-tuning training samples from fine-tuning training source pre-processing samples, the fine-tuning training source pre-processing samples should also be subjected to data enhancement processing. In specific implementation, the data enhancement processing mode of the fine-tuning training source pre-processing samples derived from the open source language recognition data set Fleurs is different from that of the fine-tuning training source pre-processing samples derived from the open source language recognition data set Slingua. The following will be described in detail.

[0102] When the fine-tuning training source pre-processing sample is derived from the open source language recognition data set Fleurs, the data enhancement processing of each open source language recognition data set Fleurs can be specifically as follows: The fine-tuning training source pre-processing sample is cut into 4-second segments, then the cut voice segments are randomly spliced at one end or two ends, and then speed variation, noise, music, reverberation, and other data enhancement are performed, so as to generate a corresponding fine-tuning training sample after data enhancement.

[0103] When the fine-tuning training source pre-processing sample is derived from the open source language recognition data set Slingua, data enhancement processing is performed on each open source language recognition data set Slingua. Specifically, the fine-tuning training source pre-processing sample can be segmented into 15 seconds, and then 1.2 times and 0.8 times speed perturbation is performed to generate a corresponding fine-tuning training sample after speed perturbation.

[0104] In specific implementation, the above-mentioned data enhancement processing can be performed by using the kaldi tool, wherein the parameter setting is to start enhancement from the beginning of the fine-tuning training source pre-processing sample, and the signal-to-noise ratio is randomly selected in the range of 0-10. In addition, the offline source for noise, music, and reverberation noise can use the offline source of the open source musan or the offline source of rirs. Of course, other ways of data enhancement processing can also be used, which will not be illustrated one by one here.

[0105] As can be seen from the above description, when the fine-tuning training sample is made in the above-mentioned manner, the label of each fine-tuning training sample does not need to be made, so it can be known that the fine-tuning training is to Figure 2 adaptively fine-tune the constructed language recognition unlabeled data in the middle.

[0106] Based on the voice classification pre-training module, the corresponding weight can be obtained. During fine-tuning training, the weight finetune is initialized, and self-supervised learning is continued, wherein the goal of the fine-tuning training stage is to make the voice classification pre-training module better adapt to the data distribution of the language recognition task. Specifically, in the fine-tuning stage, the initial parameter of the voice classification pre-training module is θ finetune (0), and the loss function to be optimized can be consistent with the loss function used in the pre-training stage. The loss function used can refer to the above description. As can be known, during fine-tuning training, the weight θ finetune of the voice classification pre-training module can be further fine-tuned so as to perform better on the multi-language recognition task. Specifically, the goal of fine-tuning training can be represented as:

[0107]

[0108] wherein, is the weight of the voice classification pre-training module when the fine-tuning training reaches the target fine-tuning state, and L finetune is the loss function used in the fine-tuning training stage, and θ is the weight of the voice classification pre-training module.

[0109] As can be known from the above description, the loss function L finetune can refer to the above-mentioned pre-training loss function description. In specific implementation, during fine-tuning training, the loss function L finetuneThe stable training state tends to be a fine-tuning training target state, at which the speech classification post-training module reaching the target fine-tuning state is configured as the speech feature extraction module 1 by closing the mask and quantization modules in the speech classification post-training module, so that the speech feature extraction module 1 is only used for feature extraction. In specific implementation, the parameters of the speech classification post-training module can be frozen and the mask and quantization modules can be closed by using existing common technical means, which can meet the configuration of the speech feature extraction module 1.

[0110] After the fine-tuning training and obtaining the speech feature extraction module 1, a language recognition basic model can be constructed, wherein the architecture of the language recognition basic model should be consistent with the above-mentioned language recognition model, and therefore, as known from the above description, the language recognition basic model should also include the speech feature extraction module 1 and the corresponding language embedding extraction basic module and recognition classification basic module. When the language recognition basic model is trained to obtain a language recognition model, the corresponding language embedding extraction module 2 can be generated by the language embedding extraction basic module, and the corresponding recognition classification module 3 can be generated by the recognition classification basic module, Figure 2 and Figure 3 An embodiment of the speech feature extraction module 1 corresponding to the language recognition basic model and the language recognition model is shown in FIG.

[0111] In an embodiment of the present application, the language embedding extraction module 2 includes a language local feature extraction layer, a statistical pooling layer and a language level feature extraction layer connected in sequence, wherein,

[0112] The language local feature extraction layer is connected with the speech feature extraction module;

[0113] The language level feature extraction layer is connected with the recognition classification module.

[0114] Figure 2 and Figure 3 An embodiment of the language embedding extraction module 2 is also shown in FIG. As shown in the figure, the language embedding extraction module 2 based on the x-vector model can include a language local feature extraction layer, a statistical pooling layer and a language level feature extraction layer. It can be understood that when the language embedding extraction module 2 based on the x-vector model, the language local feature extraction layer, the statistical pooling layer and the language level feature extraction layer can adopt the form commonly used in the existing x-vector model, such as Figure 3In the specific implementation, the language partial feature extraction layer can be formed by five TDNN layers connected in sequence, the local features are extracted from the frame level of the speech signal by the five-layer TDNN, and the short-time information of the speech is captured; the statistical pooling layer can adopt the existing common form, the local features of the entire speech segment are globally aggregated, the mean and standard deviation of the frame-level features are calculated, and thus a global speech embedding is generated. The language-level feature extraction layer is composed of a fully connected layer, a ReLU activation function layer, and a batch normalization layer, the global speech embedding generated by the statistical pooling layer can be extracted by the language-level feature extraction layer to generate language embedding information, and the language embedding information is loaded into the recognition classification module 3.

[0115] In an embodiment of the present application, the recognition classification module 3 includes a fully connected layer and a softmax layer, wherein,

[0116] When the language embedding information is mapped to the language classification information, the language embedding information is processed by the fully connected layer and the softmax layer in sequence, and the language classification information is output through the softmax layer.

[0117] Figure 3 An embodiment of the recognition classification module 3 is shown in the specific implementation, Figure 3 In the specific implementation, Linear is a fully connected layer, of course, the recognition classification module 3 can also adopt other forms, which can be selected as needed. In the specific implementation, the language embedding extraction module 2 can fully aggregate the speech feature information extracted by the speech feature extraction module 1, and then the language embedding information of a fixed length can be extracted, and the language embedding information can be mapped to the language classification result through the recognition classification module 3.

[0118] The above gives an embodiment of constructing a language recognition basic model, thereafter, a model training sample set should be constructed, specifically, when constructing the model training sample set, the fine-tuning training sample set described above can be used as the basis. As can be known from the above description, there is no label information in the fine-tuning training sample in the fine-tuning training sample set, in order to meet the needs of model training, the label information of each fine-tuning training sample should be determined by using the technical means commonly used in the technical field, the label information is the reference language corresponding to the fine-tuning training sample, thereafter, a model training sample can be formed according to the fine-tuning training sample and the corresponding language label, and a model training sample set can be constructed.

[0119] As can be known from the above description, when the model training sample set is used to train the language recognition basic model, the network parameters of the speech feature extraction module 1 are frozen, that is, the network parameters of the speech feature extraction module 1 are stopped from being updated, and the model training is Figure 2The model training manner of the language recognition basic model by using the model training sample set can be consistent with the prior art, that is, the language recognition basic model is iteratively trained by using the model training sample set, and after each epoch training, the parameters of the model are updated according to the loss function value, that is, the corresponding parameters of the language embedding extraction basic module and the recognition classification basic module are updated, until the language recognition basic model is trained to reach a target training state, and thereafter, the language recognition basic model reaching the target training state is configured as a language recognition model.

[0120] In an embodiment of the present application, when the language recognition basic model is model trained, the batch loss function used is:

[0121]

[0122] Wherein, L c is the batch loss function value during model training, B is the batch size, is the predicted probability of the correct class y i of the i-th model training sample, represents the farthest positive sample distance between the i-th model training sample and other model training samples in the same batch, represents the nearest negative sample distance between the i-th model training sample and other model training samples in the same batch, and δ is the distance boundary threshold.

[0123] When model training is performed, the batch size is B, that is, B model training samples are loaded into the language recognition basic model at a time. For the B model training samples in the same batch, the cosine distance between each two model training samples is calculated to evaluate the similarity between each pair of model training samples. When calculating the cosine distance between two model training samples, the language embedding information corresponding to each model training sample is determined, and the cosine distance is calculated based on the language embedding information. The manner and process of calculating the cosine distance can be consistent with the prior art, which will not be described here.

[0124] The process of calculating the above loss function value is described below. Specifically:

[0125] Suppose there is a batch of model training samples with an output dimension of (32, 512) obtained by the language recognition basic model, where 32 is the batch size and 512 is the feature dimension, and at this time, the batch size B is 32. The steps of calculating the batch loss function value L c are as follows:

[0126] 1) Calculate the distance matrix: calculate the cosine distance between each two model training samples in the same batch to obtain a distance matrix cdist of size (32, 32), where cdist[i][j] represents the cosine distance between the ith model training sample and the jth model training sample, that is, for any model training sample, the corresponding distance matrix cdist between the current model training sample and other model training samples can be calculated.

[0127] It should be noted that the diagonal elements in the distance matrix cdist are the cosine distance between the ith model training sample and the ith model training sample, at this time, the cosine distance should be 0, and the elements of the diagonal line in the distance matrix cdist should be 0.

[0128] 2) Create a mask matrix: create a mask matrix mask_pos according to the language label of each model training sample, which is a matrix of size (32, 32), used to indicate which samples are same class samples (positive samples, at this time, the language labels are the same), where the value of the mask is 1 indicates that it is a same class sample, and the value of the mask is 0 indicates that it is a different class sample (the language labels are different), for example, assuming that the 0th model training sample and the 1st model training sample are the same class sample, then the values of mask_pos[0][1] and mask_pos[1][0] are both 1, mask_pos[0][1] and mask_pos[1][0] are the values of the corresponding positions in the mask matrix, that is, for any model training sample, the corresponding mask matrix mask_pos between the current model training sample and other model training samples can be calculated.

[0129] It should be noted that the diagonal elements in the mask matrix mask_pos are the mask value between the ith model training sample and the ith model training sample, at this time, since the same class language label, the diagonal elements in the mask matrix mask_pos should be 1.

[0130] 3) Calculate the farthest positive sample distance: use the mask matrix mask_pos to calculate the farthest positive sample distance of each model training sample, that is, the sample distance farthest from the current model training sample among all same class model training samples, where the calculated sample distance result is a vector of size 32.

[0131] It should be noted that when calculating the distance of the farthest positive sample, the distance matrix cdist can be multiplied by the mask matrix mask_pos, wherein the multiplication elements refer to the multiplication of each corresponding matrix element. Thereafter, in the result matrix obtained by multiplication, the maximum value of each row is determined. After determining the maximum value of each row, the maximum value of all row maximum values is determined, which is the farthest positive sample distance between the i-th model training sample and other model training samples in the same batch.

[0132] 4) Calculate the nearest negative sample distance: use the mask matrix mask_pos to calculate the nearest negative sample distance of each model training sample, that is, the sample distance closest to the current model training sample among all model training samples of different classes; wherein the sample distance result obtained by calculation is also a vector with a size of 32.

[0133] It should be noted that when calculating the distance of the nearest negative sample, each element in the mask matrix mask_pos is multiplied by an infinite value to obtain a mask calculation matrix. Thereafter, the distance matrix cdist can be multiplied by the mask calculation matrix, wherein the meaning of multiplication operation here can be referred to the above description, which will not be described here. Thereafter, in the result matrix obtained by multiplication, the minimum value of each row is determined. After determining the minimum value of each row, the minimum value of all row minimum values is determined, which is the nearest negative sample distance between the i-th model training sample and other model training samples in the same batch.

[0134] In specific implementation, the farthest positive sample distance is used to evaluate the similarity between samples of the same class, and the nearest negative sample distance is used to evaluate the similarity between samples of different classes. For the distance boundary threshold δ, it can be specifically set to 0.2, that is, to ensure that the distance between positive samples and negative samples is at least 0.2, so as to encourage a significant distance difference between samples of the same class and samples of different classes, and to enhance the discrimination ability of features of different categories.

[0135] As can be seen from the above batch loss function, the language classification loss and the vector loss can be optimized at the same time, so as to improve the accuracy of language classification and the discriminability of embedding. It should be noted that when the model training is performed, the network parameters of the language recognition basic model can be updated after the language recognition basic model is trained by using the model training samples of each batch. When the batch loss function value tends to be stable, it is considered that the language recognition basic model is trained to reach the target training state. Of course, the target training state can also be other conditions, which can be selected as needed.

[0136] Specifically, after obtaining the language recognition model, the target voice audio can be subjected to language category recognition. In order to prove the effectiveness of the language recognition model of the present application in performing language category recognition on the target voice audio, the following can be compared with the existing language recognition model, specifically:

[0137] Table 1 Multilingual recognition indicators under different methods

[0138]

[0139] Specifically, the above embodiment shows that the reference language set includes 15 reference languages. In specific implementation, Fula and Oromo can be used for language recognition in the case of unbalanced training data, that is, the other 13 languages can be used as the mandatory reference language. In the 13 mandatory reference languages, the number distribution in the model training sample set can be as shown in the histogram. Figure 4 In addition, Figure 4 The number distribution in other cases is also shown in Table 1.

[0140] For the above Fleurs validation set, it can be generally constructed based on the validation data part in the open source language recognition data set Fleurs. When the validation data set is generated based on the open source language recognition data set Fleurs, the same data processing as the above-mentioned training data set in the open source language recognition data set Fleurs needs to be performed, such as processing into a single channel, 16K sampling rate, 16 bit depth, and then performing data enhancement processing, which can be referred to the above description. The specific construction method of the Fleurs test set and the Slingua validation set can refer to the construction method of the Fleurs validation set described herein, which will not be described here.

[0141] The above Table 1 shows an embodiment of using 13 reference languages to compare different language recognition methods. “Our method” in Table 1 is the language recognition model of the present application. As can be seen from Table 1, compared with other mainstream language recognition methods, the language recognition model of the present application achieves the best effect on three open source test sets.

[0142] In Table 1, XLS-R+Linear represents the weights of the open-source pre-trained model XLS-R based on the wav2vec2.0 structure, and then a full connection layer and a Softmax activation function are connected for classification. FBANK+x-vector represents extracting the FBANK features of the speech and then connecting the x-vector structure based on the Time Delay Neural Network (TDNN) for classification. FBANK+F-TDNN represents extracting the FBANK features of the speech and then connecting the x-vector structure based on the Factorized Time-Delay Neural Network (F-TDNN) for classification. FBANK+ECAPA represents extracting the FBANK features of the speech and then connecting the x-vector structure based on the Emphasized Channel Attention-based Time-Delay Neural Network (ECAPA-TDNN) for classification. XLS-R+x-vector represents freezing the weights of the open-source XLS-R part, only updating the weight parameters of the x-vector part, and then classifying the language.

[0143] It can be understood that, in Table 1, the models corresponding to the other methods except the language identification model of the application are all language identification model methods existing in the prior art, and the language identification conditions of the models are well known to those skilled in the art, and will not be described here.

[0144] In specific implementation, the balance of the training data can also be further explored. At this time, 20,000 pieces of data are selected from each of the above 13 languages, and then two languages, Fur and Oromo, with less data are added to construct an experiment of identifying 15 languages. The training set, the validation set and the test set of the Fur language have 3236, 273 and 660 pieces of data respectively, and the training set, the validation set and the test set of the Oromo language have 1701, 19 and 41 pieces of data respectively. The experimental results are shown in Table 2.

[0145] Table 2 Comparison of various mainstream methods under the condition of unbalanced training data.

[0146]

[0147] As can be seen from Table 2, compared with other mainstream methods, the language identification model of the application can also achieve the best effect on the validation set and the test set of Fleurs under the condition of unbalanced training data, proving the effectiveness of the proposed model.

[0148] According to the above description, the speech language category recognition system can be obtained, and in an embodiment of the present application, the speech language recognition device is included, and the speech language recognition model is arranged in the speech language recognition device.

[0149] The speech language recognition model is used to perform speech language recognition processing on the target speech audio, and the speech language classification information corresponding to the target speech audio is generated after the speech language recognition processing.

[0150] Specifically, the speech language recognition device can be a commonly used computer device, and the speech language recognition model can be arranged in the speech language recognition device by using the existing technical means. Thereafter, the speech language category recognition and classification of the target speech audio can be realized. The specific speech language category recognition and classification can be referred to the above description, and will not be described here.

[0151] The above has described the present application and its embodiments in a schematic manner, and the description is not restrictive, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application. The embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto, and any reference signs in the claims should not limit the claims. Therefore, if a person skilled in the art is inspired by the present application, and without departing from the spirit of the present application, the similar structure and embodiments can be designed without creative design, and all should belong to the protection scope of the present application. In addition, the word "comprising" does not exclude other elements or steps, and the word "one" before the element does not exclude "multiple" elements. The multiple elements stated in the product claim can also be realized by one element through software or hardware. The words "first", "second" and the like are used to represent the names, and do not represent any specific order.

Claims

1. A method for recognizing speech language categories, characterized in that, The identification method includes: A target speech audio is provided, and the target speech audio is loaded into a pre-built language recognition model to perform language recognition processing on the target speech audio using the language recognition model. When a language identification model performs language identification processing on target speech audio, it includes at least the following steps performed sequentially: speech feature extraction, language embedding extraction, and identification and classification. After performing speech feature extraction processing, corresponding speech feature information is generated based on the target speech audio. After performing language embedding extraction processing, corresponding language embedding information is generated based on speech feature information; After performing the recognition and classification process, the language embedding information is mapped to language classification information that represents the language corresponding to the target speech audio, wherein the language classification information is a benchmark language in the benchmark language set used when constructing the language recognition model; The language recognition model includes a speech feature extraction module, a language embedding extraction module, and a recognition and classification module connected in sequence. When performing language recognition processing on the target speech audio, the speech feature extraction module is used to perform speech feature extraction processing, and the generated speech feature information is loaded into the language embedding extraction module. The language embedding extraction module performs language embedding extraction processing, and the generated corresponding language embedding information is loaded into the recognition and classification module. The identification and classification module is used to perform identification and classification processing and generate language classification information; The speech feature extraction module is built and generated based on the wav2vec2.0 model, and the language embedding extraction module is built and generated based on the x-vector model; When constructing a language recognition model, the construction methods include: A speech feature extraction module was built based on the wav2vec2.0 model; After generating the speech feature extraction module, a basic language recognition model is constructed. When constructing the basic language recognition model, the speech feature extraction module is connected to the basic language embedding extraction module, and the basic language embedding extraction module is connected to the basic recognition and classification module. A model training sample set is constructed, and the basic language recognition model is trained using this sample set until it reaches the target training state. Afterward, the language recognition model that has reached the target training state is configured as the language recognition model. The model training sample set includes several model training samples. When training the basic language recognition model using the model training sample set, the network parameters of the speech feature extraction module are frozen, and the speech feature extraction module is used to extract the speech feature information of the model training samples. The extracted speech feature information is then loaded into the basic language embedding extraction module. The model training samples include training speech audio and label information corresponding to the training speech audio, wherein the label information is the base language corresponding to the training speech audio; The training audio and the target audio have the same speech feature parameters, including the speech channel, sampling rate and / or bit depth; When constructing the speech feature extraction module, the following are included: A speech classification module based on the wav2vec2.0 model is constructed, and the speech classification module is pre-trained and fine-tuned sequentially. After fine-tuning, a speech feature extraction module is generated based on the speech classification module. When pre-training the speech classification module, a pre-training sample set is constructed. The speech classification module is pre-trained using a self-supervised learning training method based on the pre-training sample set. When the pre-training reaches the pre-training target state, a pre-trained speech classification module is generated. The pre-training sample set includes several pre-training samples, and the pre-training samples are unlabeled speech audio. The pre-trained module based on speech classification can obtain corresponding weights. During fine-tuning training, the weights are adjusted. Initialization is performed, and self-supervised learning continues. The goal of the fine-tuning training phase is to make the pre-trained speech classification module better adapt to the data distribution of the language recognition task. During fine-tuning training, a fine-tuning training sample set is constructed, and the pre-trained speech classification module is trained using this sample set. Once the pre-trained speech classification module reaches the target fine-tuning state, it is configured as the speech feature extraction module. The fine-tuning training sample set includes fine-tuning training samples in multiple languages, and the fine-tuning training samples are unlabeled speech and audio. For the speech classification pre-trained module that has reached the target fine-tuning state, the mask and quantization modules in the speech classification pre-trained module are turned off, and this configuration is used as the speech feature extraction module, so that the speech feature extraction module is only used for feature extraction. The language embedding extraction module includes a language local feature extraction layer, a statistical pooling layer, and a language-level feature extraction layer connected in sequence. The language local feature extraction layer is connected to the speech feature extraction module; The language-level feature extraction layer is connected to the recognition and classification module; The language-level local feature extraction layer extracts local features from the frame-level speech feature information to capture short-term speech information; the statistical pooling layer performs global aggregation of the local features of the entire speech segment, calculates the mean and standard deviation of the frame-level features, and thus generates a global speech embedding; the language-level feature extraction layer extracts the global speech embedding generated by the statistical pooling layer to generate language embedding information, and loads it into the recognition and classification module. The loss function used during pre-training of the speech classification module is: in, For the pre-training loss function value, The first generation is generated based on the current pre-trained samples. The latent representation features of each time step Based on the Latent representation features at each time step The generated context features It is a context feature generated based on the j-th negative sample in the negative sample set. It is the target time step set of the current pre-trained samples. It is the set of negative samples of the current pre-trained samples. Calculate the cosine similarity; When training the basic language identification model, the batch loss function used is: in, This refers to the batch loss function value during model training. For batch size, It is the first The correct category of each training sample for the model The predicted probability, This represents the distance between the i-th model training sample and the farthest positive sample in the same batch of model training samples. This represents the distance between the i-th model training sample and the nearest negative sample of other model training samples in the same batch. It is the distance to the boundary threshold.

2. The method for recognizing speech categories according to claim 1, characterized in that: The identification and classification module includes a fully connected layer and a softmax layer, wherein... When mapping language embedding information to language classification information, the language embedding information is processed sequentially using a fully connected layer and a softmax layer, and the language classification information is output through the softmax layer.

3. A speech category recognition system, characterized in that, The system includes a language recognition device, and deploys the language recognition model according to any one of claims 1 to 2 within the language recognition device, wherein... The target speech audio is processed by a language recognition model to identify the language, and language classification information corresponding to the target speech audio is generated after the language recognition process.

Images