Dialect language recognition method and system based on self-supervised speaker representation decoupling

Abstract

The present application relates to a dialect language recognition method and system based on self-supervised speaker representation decoupling, and belongs to the technical field of natural language processing.The present application comprises: inputting original speech into a context encoder of a self-supervised model to extract an output context representation; performing weighted summation on the context representation, and then converting the result into a self-supervised speech representation using an average pooling layer; inputting the self-supervised speech representation into a speaker gender classifier and a dialect classifier to obtain decoupled speaker representation and dialect representation, respectively; training the self-supervised model based on a gradient descent algorithm using the speaker representation and the dialect representation, adjusting the weight parameters of the network by back-propagating the gradient of the loss function with respect to each weight in the network, and using the trained self-supervised model to recognize dialects.The model proposed in the present application has shown excellent performance in dialect language recognition under low resource conditions.

Description

Technical Field

[0001] The present invention relates to a dialect recognition method and system based on self-supervised speaker representation decoupling, and to the technical field of natural language processing. Background Art

[0002] Speech recognition technology has a wide range of applications, including voice assistants, smart homes, and car voice interaction. Dialects are a popular form of daily communication among Chinese people. Dialect identification (DID) involves analyzing and processing input speech sequences to determine the dialect to which they belong. The core challenge of DID lies in extracting features that can effectively distinguish different dialects.

[0003] Traditional dialect recognition methods use a supervised learning approach, using large amounts of labeled data for training. They first extract underlying acoustic features from the speech signal, such as Mel-frequency cepstral coefficients, filter banks, i-vectors, and x-vectors. A front-end encoder is then used to extract the corresponding dialect representation. A independently trained back-end classifier is then applied to identify the dialect category. Thanks to the rapid development of deep neural network technology, these front-end and back-end methods have been integrated into a single network through an end-to-end approach. While supervised learning has become a core approach for speech processing, it requires a large amount of labeled data for each task and scenario.

[0004] In recent years, self-supervised learning (SSL) has made significant progress in speech processing. Similar to traditional acoustic features, self-supervised speech representations contain a wealth of information that is highly beneficial for downstream tasks such as speech emotion recognition, language identification, and speaker verification. Although self-supervised representations have made progress in dialect recognition tasks, it is unclear whether all the information contained in these representations is beneficial for dialect recognition tasks. Therefore, it is crucial to analyze the information in self-supervised representations and decouple irrelevant or unnecessary information. Summary of the Invention

[0005] The technical problem solved by the present invention is: the present invention provides a dialect variety recognition method and system based on self-supervised speaker representation decoupling, which is used to decouple unnecessary speaker information in the self-supervised model to extract the difference representations between dialects. The present invention improves the effect of dialect variety tasks.

[0006] The technical solution of the present invention is: a dialect recognition method based on self-supervised speaker representation decoupling, the method comprising:

[0007] Step 1: Input the original speech into the context encoder of the self-supervised model to extract the output context representation, where the self-supervised model consists of a convolutional neural network feature extractor and a Transformer-based context encoder;

[0008] Step 2: Perform a weighted summation of the context representations and then use an average pooling layer to convert the result into a self-supervised speech representation.

[0009] Step 3: The self-supervised speech representation is input into the speaker gender classifier and dialect classifier to obtain the decoupled speaker representation S and dialect representation D respectively;

[0010] Step 4: Use speaker representation and dialect representation to train the self-supervised model based on the gradient descent algorithm. Adjust the network weight parameters by backpropagating the gradient of the loss function relative to each weight in the network, and use the trained self-supervised model to identify the dialect.

[0011] Furthermore, in Step 2, the generation process of the self-supervised speech representation U is expressed as follows:

[0012]

[0013] Among them, w i is the weight of layer i, C i represents the output of the i-th layer of the context encoder, Mean represents the mean pooling operation, and N represents the number of encoder layers.

[0014] Furthermore, in Step 3, the speaker gender classifier and dialect classifier both use a cross-entropy loss function to calculate their respective losses; the loss functions for dialect and gender classification are defined as follows:

[0015]

[0016] Among them, d i and s i Represents the true dialect and speaker gender label respectively; and Represents the predicted probability score; N d and N s is the number of all dialect labels and gender labels; L d and L s are the loss functions for dialect and gender classification respectively.

[0017] Furthermore, the Step 3 also includes adding a gradient reversal layer before the speaker gender classifier to decouple speaker information.

[0018] Furthermore, in Step 4, during the training of the self-supervised model, the gradient reversal layer does not substantially participate in the forward propagation of the self-supervised model. During the reverse propagation of the self-supervised model training, unlike the general gradient descent algorithm, the gradient reversal layer multiplies the gradient of the subsequent layer by -λ and passes it to the previous layer, so that the weight parameters of the model will be trained to be far away from the target distribution when updated. The weight parameters θ of the model m The update formula is as follows:

[0019]

[0020] Where μ is the learning rate, λ is a manually adjustable hyperparameter, and θ s represents the weight parameter of the speaker classifier, μ represents the learning rate, θ d Represents the weight parameter of the dialect classifier.

[0021] The present invention also provides a dialect recognition system based on self-supervised speaker representation decoupling, the system comprising: a module for executing the above-mentioned dialect recognition method based on self-supervised speaker representation decoupling.

[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned dialect recognition method based on self-supervised speaker representation decoupling when executing the program.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned dialect language recognition method based on self-supervised speaker representation decoupling.

[0024] The present invention also provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the method for dialect recognition based on self-supervised speaker representation decoupling is implemented.

[0025] This paper explores the application of the self-supervised model (SSM) in dialect recognition tasks; analyzes the impact of speaker information on dialect recognition and visualizes speaker information between different layers of the self-supervised model; and proposes a method to decouple speaker-related information from the self-supervised representation using a layered gradient reversal layer (GRL).

[0026] The beneficial effects of the present invention are:

[0027] 1. The feature decoupling method proposed in this invention successfully decouples speaker information from the self-supervised model, improving the effect of dialect tasks.

[0028] 2. Experimental results on the KeSpeech dialect public dataset show that the model proposed in this paper demonstrates excellent performance in dialect language recognition tasks.

[0029] 3. The model proposed in this invention shows excellent performance in low-resource language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the architecture diagram of the dialect recognition model based on self-supervised speaker representation decoupling proposed by the present invention; DETAILED DESCRIPTION

[0031] Example 1: Figure 1 As shown, a dialect recognition method based on self-supervised speaker representation decoupling includes:

[0032] Step 1: Input the original speech into the context encoder of the self-supervised model to extract the output context representation, where the self-supervised model consists of a convolutional neural network feature extractor and a Transformer-based context encoder;

[0033] Step 2: Perform a weighted summation of the context representations and then use an average pooling layer to convert the result into a self-supervised speech representation.

[0034] Furthermore, in Step 2, the generation process of the self-supervised speech representation U is expressed as follows:

[0035]

[0036] Among them, w i is the weight of layer i, C i represents the output of the i-th layer of the context encoder, Mean represents the mean pooling operation, and N represents the number of encoder layers.

[0037] Step 3: The self-supervised speech representation is input into the speaker gender classifier and dialect classifier to obtain the decoupled speaker representation S and dialect representation D respectively;

[0038] Furthermore, in Step 3, the speaker gender classifier and dialect classifier both use a cross-entropy loss function to calculate their respective losses; the loss functions for dialect and gender classification are defined as follows:

[0039]

[0040] Among them, d i and s i Represents the true dialect and speaker gender label respectively; and Represents the predicted probability score; N d and N s is the number of all dialect labels and gender labels; L d and L s are the loss functions for dialect and gender classification respectively.

[0041] Furthermore, the Step 3 also includes adding a gradient reversal layer before the speaker gender classifier to effectively decouple speaker information.

[0042] Step 4: Use speaker representation and dialect representation to train the self-supervised model based on the gradient descent algorithm. Adjust the network weight parameters by backpropagating the gradient of the loss function relative to each weight in the network, and use the trained self-supervised model to identify the dialect.

[0043] Furthermore, in Step 4, during the training of the self-supervised model, the gradient reversal layer does not substantially participate in the forward propagation of the self-supervised model. During the reverse propagation of the self-supervised model training, unlike the general gradient descent algorithm, the gradient reversal layer multiplies the gradient of the subsequent layer by -λ and passes it to the previous layer, so that the weight parameters of the model will be trained to be far away from the target distribution when updated. The weight parameters θ of the model m The update formula is as follows:

[0044]

[0045] Among them, μ is the learning rate, λ is a manually adjustable hyperparameter, which is set to 0.15 in this paper, and θ s represents the weight parameter of the speaker classifier, μ represents the learning rate, θ d Represents the weight parameter of the dialect classifier.

[0046] The present invention also provides a dialect recognition system based on self-supervised speaker representation decoupling, the system comprising:

[0047] A context representation extraction module is used to extract the output context representation by inputting the original speech into the context encoder of the self-supervised model, where the self-supervised model consists of a convolutional neural network feature extractor and a Transformer-based context encoder;

[0048] A self-supervised speech representation generation module, which performs a weighted summation of the context representations and then uses an average pooling layer to convert the result into a self-supervised speech representation;

[0049] The speaker representation and dialect representation generation module is used to input the self-supervised speech representation into the speaker gender classifier and dialect classifier to obtain the decoupled speaker representation and dialect representation respectively;

[0050] The dialect recognition module is used to train the self-supervised model based on the gradient descent algorithm using speaker and dialect representations. The weight parameters of the network are adjusted by backpropagating the gradient of the loss function relative to each weight in the network, and the trained self-supervised model is used for dialect recognition.

[0051] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned dialect recognition method based on self-supervised speaker representation decoupling when executing the program.

[0052] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned dialect language recognition method based on self-supervised speaker representation decoupling.

[0053] The present invention also provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the method for dialect recognition based on self-supervised speaker representation decoupling is implemented.

[0054] To decouple unnecessary speaker information from the self-supervised model, this paper explores the application of the self-supervised model (SSM) in dialect recognition tasks; analyzes the impact of speaker information on dialect recognition and visualizes speaker information between different layers of the self-supervised model; and proposes a method to decouple speaker-related information from the self-supervised representation by using a hierarchical gradient reversal layer (GRL).

[0055] For dialect recognition tasks, current research usually adds an average pooling layer and a dialect classifier to the self-supervised model; the input of the self-supervised model is the original speech sequence, and then the feature extractor based on the convolutional neural network extracts the features and the context encoder based on the Transformer extracts the context representation; the dialect classifier usually contains complex structures, such as convolutional neural networks or self-attention layers. In order to fully demonstrate the superiority of the proposed representation decoupling method, the classifiers used in the present invention are all simple linear projection layers. Figure 1 As shown in Figure 1, an average pooling layer, a gradient reversal layer, a speaker gender classifier, and a dialect classifier following the self-supervised model are integrated on top of the self-supervised model.

[0056] In order to illustrate the effect of the present invention, the present invention has done the following experiment:

[0057] (a) Experimental setup and evaluation metrics

[0058] The main experiments and analyses were conducted on Kespeech, a publicly available dataset of Chinese dialects. The Kespeech dataset consists of 1,542 hours of annotated speech data from 27,237 speakers across 34 cities in China. The dataset includes data from eight Mandarin dialects and Mandarin: Northeastern, Jilin-Lu, Central Plains, Jiaoliao, Jianghuai, Southwest, Lanyin, and Beijing.

[0059] The evaluation index in the experiment mainly uses the multi-classification accuracy (Acc), which is the number of correctly classified samples L c The total number of samples L a Ratio:

[0060]

[0061] All audio samples in the experiment were sampled at 16 kHz, with a single channel and a 16-bit bit depth. Three pre-trained self-supervised models (Wav2vec2.0, WavLM, and HuBERT) were open-sourced from the Huggingface community, using the base model. Fine-tuning was performed for 10 epochs, during which the convolutional feature extractor was frozen and parameters were not updated. The learning rate was set to 0.003, and the lambda value of the gradient reversal layer was set to 0.15.

[0062] (b) Comparative experiment

[0063] The purpose of this experiment is to compare the performance of different models for dialect recognition. The experimental results are shown in Table 1. The table shows the baseline method and the existing state-of-the-art method on the KeSpeech multi-dialect dataset. Each column corresponds to the dialect classification accuracy of a specific dialect, and the last column calculates the overall classification accuracy.

[0064] Table 1 shows the comparative experiments of different models on the test set.

[0065] Model northeast Giroud Central Plains Jiaoliao Jianghuai southwest Blue Silver Beijing overall M1 - 43.22 52.63 58.56 53.19 82.47 43.76 - 56.34 M2 - 39.37 76.65 37.84 64.44 82.32 42.67 - 61.13 M3 - 48.45 68.54 36.52 68.56 80.91 44.18 - 60.77 M4 - - - - - - - - 78.57 M5 - - - - - - - - 79.10 M6 76.53 62.41 67.91 76.45 72.33 89.34 69.51 64.13 75.86 M7 78.26 68.32 73.61 76.94 72.75 89.79 70.13 64.91 75.50 M8 80.47 70.56 71.25 77.45 75.31 90.34 71.16 66.35 76.45 M9 83.76 72.25 63.69 81.57 77.92 91.64 76.41 69.21 78.12 M10 85.18 71.72 73.81 82.43 78.65 92.03 76.13 70.09 78.62 M11 87.72 72.45 77.19 82.06 82.34 93.73 77.68 72.33 79.93

[0066] M1-M3 correspond to traditional methods and network architectures, which are not suitable for low-resource conditions, such as the Northeastern Mandarin and Beijing Mandarin in the table. M1 is the ResNet-34 model, which uses a stacked residual convolutional network structure. M2 uses the Kaldi open source toolkit to extract x-vector features from audio for dialect classification. M3 is ECAPA-TDNN, a speaker model based on a time-delay neural network. Due to its superior performance, it has been widely used in the field of speaker recognition. ECAPA-TDNN has also been applied to tasks such as speaker classification and language identification. Because the network architecture is not suitable for low-resource tasks, no evaluation metrics are provided for low-resource dialects such as the Northeastern and Beijing dialects.

[0067] M4 and M5 correspond to DIMNet and Qifusion-Net, respectively, and are multi-task models trained simultaneously on dialect recognition and speech recognition tasks. Due to the multi-task environment, the accuracy of each dialect classification is not provided separately.

[0068] M6-M8 are the experimental results of Wav2vec2.0, HuBERT and WavLM after fine-tuning using multi-dialect data, while M9-M11 are the experimental results of the three self-supervised models after using the feature decoupling method proposed in this paper.

[0069] Analysis of Table 1 clearly shows that even without the use of feature decoupling, the experimental results of the self-supervised models (M6-M8) after fine-tuning are superior to those of traditional methods and model architectures, reaching between 75.96% and 76.45%, respectively. Because the multi-task architecture simultaneously learns phonetic and semantic representations, corresponding to dialect recognition tasks and speech recognition tasks, it may be richer in representational information than self-supervised representations. The performance of the self-supervised models (M9-M11) fine-tuned using the feature decoupling method was further improved, with accuracy rates reaching 78.12%, 78.62%, and 79.93%, respectively, surpassing all current baseline models, thus demonstrating the superiority of the method.

[0070] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A dialect recognition method based on self-supervised speaker representation decoupling, characterized by: The method comprises: Step 1: Input the original speech into the context encoder of the self-supervised model to extract the output context representation; Step 2: Perform a weighted summation of the context representations and then use an average pooling layer to convert the result into a self-supervised speech representation. Step 3: The self-supervised speech representation is input into the speaker gender classifier and dialect classifier to obtain the decoupled speaker representation and dialect representation respectively; Step 4: Use speaker and dialect representations to train the self-supervised model based on the gradient descent algorithm. Adjust the network weight parameters by backpropagating the gradient of the loss function relative to each weight in the network. Use the trained self-supervised model to identify the dialect. In Step 3, the speaker gender classifier and dialect classifier both use the cross entropy loss function to calculate their respective losses; the loss functions for dialect and gender classification are defined as follows: ； ； in, and Represents the true dialect and speaker gender label respectively; and represents the predicted probability score; and is the number of all dialect labels and the number of gender labels; and These are the loss functions for dialect and gender classification respectively; In the step 4, during the training of the self-supervised model, the gradient reversal layer does not participate substantially in the forward propagation of the self-supervised model; during the backward propagation of the self-supervised model, the gradient reversal layer multiplies the gradient of the subsequent layer by And pass it to the previous layer, so that the weight parameters of the model will be trained to be away from the target distribution when updated; the weight parameters of the model The update formula is as follows: ； ； ； in, is the learning rate, is a manually adjustable hyperparameter, represents the weight parameter of the speaker classifier, Represents the weight parameter of the dialect classifier.

2. The dialect identification method based on self-supervised speaker representation decoupling according to claim 1, characterized in that: In Step 2, the self-supervised speech representation The generation process is as follows: ； in, yes The weights of the layers, Represents the context encoder The output of the layer, represents the mean pooling operation, and N represents the number of encoder layers.

3. The dialect identification method based on self-supervised speaker representation decoupling according to claim 1, characterized in that: The Step 3 also includes adding a gradient reversal layer before the speaker gender classifier to decouple speaker information.

4. A dialect recognition system based on self-supervised speaker representation decoupling, characterized by: The system comprises: a module for executing the dialect language recognition method based on self-supervised speaker representation decoupling according to any one of claims 1 to 3.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the dialect language recognition method based on self-supervised speaker representation decoupling as described in any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for dialect recognition based on self-supervised speaker representation decoupling according to any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for dialect recognition based on self-supervised speaker representation decoupling according to any one of claims 1 to 3 is implemented.

Images