A dialect classification method and system based on self-supervised speech representation

Abstract

The application discloses a dialect classification method based on self-supervised speech representation and belongs to the field of speech recognition. Audio samples of different dialects are acquired, and original waveform sequences are extracted; a pre-training model is used to further extract speech representation sequences and down-sample, so that frame-level input sequences are obtained; a frame-level input sequence is encoded by using a Transform encoder; an attention aggregation method is used to pool and count the encoding results, so that sentence-level encoding sequences are obtained; the sentence-level encoding sequences are linearly projected, so that sample prediction probabilities are obtained; trainable parameters in the Transform encoder and the attention aggregation method and linear projection are trained, and the dialect types in the audio are predicted based on the trained parameters. The application uses self-supervised speech representation as input features and introduces a content-based self-attention mechanism in the Transform to extract global and local information of the audio, so that the classification accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech recognition, in particular to a dialect classification method based on self-supervised speech representation. BACKGROUND

[0002] Self-supervised speech representation has been successfully used in language or dialect classification tasks. The task usually uses a self-supervised pre-training model as a feature extractor and adds a classifier after the feature extractor to fine-tune the target data set. However, fine-tuning the self-supervised pre-training model requires a lot of time and storage costs. Secondly, the Transformer has a big defect in local information modeling, which leads to its inability to be directly applied to the field of speech processing tasks. SUMMARY

[0003] To solve the above problems, the present application provides a dialect classification method and system based on self-supervised speech representation, which uses pre-trained self-supervised speech representation instead of fine-tuned input features for dialect classification, and improves the modeling ability of the Transformer for local information through an improved self-attention mechanism, thereby improving the accuracy of dialect classification.

[0004] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0005] In a first aspect, the present application provides a dialect classification method based on self-supervised speech representation, comprising:

[0006] Obtaining audio samples of different dialects and extracting the original waveform sequence of each audio sample;

[0007] Using a pre-training model to extract the speech representation sequence of the original waveform sequence and downsample to obtain a frame-level input sequence;

[0008] Using a Transformer encoder to encode the frame-level input sequence to obtain a frame-level encoding sequence;

[0009] Using an attention aggregation method to pool and statistically process the frame-level encoding sequence to obtain a sentence-level encoding sequence;

[0010] Linearly projecting the sentence-level encoding sequence to obtain a sample prediction probability;

[0011] Training the Transformer encoder, and the trainable parameters in the attention aggregation method and the linear projection using audio samples of different dialects, and predicting the dialect type in the audio based on the trained parameters.

[0012] Further, the pre-training model is a HuBert-Large model.

[0013] Further, the downsampling process comprises:

[0014] The sequence R is doubled downsampled using a convolutional neural network and activated using a LeakyReLU activation function to obtain a sequence R';

[0015] The sequence R' is doubled downsampled using a convolutional neural network and activated using a LeakyReLU activation function to obtain a sequence R";

[0016] The sequence R" is dimensionally reduced to obtain a frame-level input sequence, wherein each frame of the frame-level input sequence has a dimension as an input dimension of the Transformer encoder.

[0017] Further, when encoding the frame-level input sequence using the Transformer encoder, the attention mechanism in the Transformer encoder is replaced by a content-based attention mechanism, specifically:

[0018] A similarity matrix E is obtained from the frame-level input sequence.

[0019] The first k largest elements in each row of the similarity matrix E and the position information of the first k largest elements in the matrix E are extracted to obtain a maximum value matrix L and a position information matrix P.

[0020] All elements in the similarity matrix E are set to zero.

[0021] The maximum value matrix L is normalized row by row to obtain a normalized maximum value matrix L'.

[0022] According to the position information matrix P, the elements at the same positions in the normalized maximum value matrix L' are filled into the zeroed similarity matrix E according to the stored position information.

[0023] The frame-level encoding sequence F is obtained based on the updated similarity matrix E.

[0024] Further, the similarity matrix E is obtained from the frame-level input sequence, comprising:

[0025] The frame-level input sequence is projected to obtain a query set Q, a key value set K, and a value set V, respectively.

[0026] The similarity matrix E is calculated based on the query set Q and the key value set K.

[0027] Further, the frame-level encoding sequence F is obtained based on the updated similarity matrix E, comprising:

[0028] The updated similarity matrix E is multiplied by the value set V to obtain the output of the attention mechanism.

[0029] After adding the attention mechanism output and the frame-level input sequence, and after layer normalization and a feedforward neural network, a frame-level encoding sequence F is obtained.

[0030] Further, the attention aggregation method specifically comprises:

[0031] The scalar score of each frame encoding sequence in the frame-level encoding sequence and the weight of each frame scalar score are calculated through the attention mechanism, and an average frame encoding sequence is calculated.

[0032] According to each frame encoding sequence in the frame-level encoding sequence, the weight and the average frame encoding sequence, a standard deviation encoding is calculated.

[0033] The average frame encoding sequence and the standard deviation encoding are spliced, and after regularization processing, a sentence-level encoding sequence is obtained.

[0034] In a second aspect, the present application provides a dialect classification system based on self-supervised speech representation, which is used to realize the above method.

[0035] The present application has the beneficial effects that: the present application uses self-supervised speech representation as input features and introduces content-based self-attention mechanism in the Transformer encoder to extract global and local information of the audio, and then obtains the prediction result through attention aggregation and linear projection, and the test results prove that the method proposed in the present application has better performance than the traditional dialect recognition method and has higher classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a schematic diagram of the overall architecture of a dialect classification method based on self-supervised speech representation shown in an embodiment of the present application;

[0037] Figure 2 is a schematic diagram of the content-based self-attention mechanism shown in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0039] As shown in Figure 1 A dialect classification method based on self-supervised speech representation, comprising:

[0040] S1: Obtain audio samples of different dialects, extract each audio original waveform sequence, and obtain a sequence A;

[0041] S2: extracting a speech representation sequence from the sequence A obtained in step S1 using a self-supervised pre-training model to obtain a self-supervised speech representation sequence R;

[0042] In this embodiment, the self-supervised pre-training model used in step S2 is a HuBert-Large model, and the model output dimension is 1024.

[0043] S3: performing down-sampling processing on the speech representation sequence R obtained in step S2 to obtain a down-sampled sequence S;

[0044] In this embodiment, the step S3 down-sampling processing is as follows:

[0045] S3-1: using a convolutional neural network (CNN) to perform two times down-sampling on the sequence R obtained in step S2, where f is the frame number, d is the dimension of each frame, the CNN convolution kernel size is 3x3, the step length is 2, and the LeakyReLU activation function is used for activation to obtain a sequence R' where c is the channel number, f' is the down-sampled frame number, d' is the down-sampled dimension of each frame, and the process is as follows:

[0046] R' = LeakyReLU(Con(R))

[0047] S3-2: for the sequence R' obtained in step S3-1, again using a CNN with a size of 3x3 and a step length of 2 to perform two times down-sampling, and again using the LeakyReLU activation function for activation to obtain a sequence R" where f" is the down-sampled frame number, d" is the down-sampled dimension of each frame, and the process is as follows:

[0048] R" = LeakyReLU(Con(R'))

[0049] S3-3: performing dimension reduction operation on the sequence R" obtained in step S3-2 to finally obtain a frame-level input sequence S where dt is the input dimension of the Transformer.

[0050] S4: using a Transformer encoder to encode the frame-level input sequence S obtained in step S3 to obtain a frame-level encoding sequence F;

[0051] In this embodiment, the Transformer encoder in step S4 adopts a content-based self-attention mechanism when encoding, and the process is as follows:

[0052] S4-1: projecting the frame-level input sequence S obtained in step S3 to obtain a query set Q, a key-value set K, and a value set V, respectively, and the process is as follows: ​

[0053] Q = W Q S

[0054] K = W K S

[0055] V = W V S

[0056] wherein W Q , W K , W V are learnable parameters.

[0057] S4-2: Calculate the similarity matrix by querying the set Q and the key value set K The calculation process is as follows:

[0058]

[0059] wherein E is a matrix of f" rows and f" columns, and the element e i，j in the matrix represents the dot product of the i-th speech representation frame and the j-th speech representation frame.

[0060] S4-3: According to the similarity matrix E obtained in step S4-2, extract the top k largest elements in each row of the matrix E and the position information of the top k largest elements in the matrix E, to obtain the maximum value matrix and the position information matrix This process can be summarized as follows:

[0061] L, P = top_k(M, k)

[0062] wherein the top_k(.) function returns the top k largest elements in each row of the input matrix and the position information of the top k largest elements in the input matrix.

[0063] S4-4: Set all elements in the similarity matrix E obtained in step S4-2 to zero;

[0064] S4-5: Normalize each row of the maximum value matrix L obtained in step S4-3, the process is as follows:

[0065] L' = softmax(L)

[0066] wherein softmax(.) is an exponential normalization function.

[0067] S4-6: Traverse the position information matrix P obtained in step S4-3, and fill the elements at the same position in the normalized maximum value matrix L' obtained in step S4-5 into the similarity matrix E obtained in step S4-4 according to the stored position information.

[0068] As Figure 2The leftmost matrix represents the similarity matrix E. The maximum elements of the first two values in each row of the matrix E and the position information of the maximum elements in the matrix E are extracted to obtain the maximum value matrix L and the position information matrix P (not shown in the figure). The maximum value matrix L is normalized by row to obtain the normalized maximum value matrix L'. The elements in the maximum value matrix L' are filled into the zeroed similarity matrix E according to the position relationship stored in the position information matrix P.

[0069] S4-7: According to the similarity matrix E obtained in step S4-6, multiply E with the value set V obtained in step S4-1 to finally obtain the output Attention (Q, K, V) of the attention mechanism, and the calculation process is as follows:

[0070] Attention (Q, K, V) = V r ·E

[0071] Where T is the matrix transposition operation.

[0072] S4-8: Add the attention mechanism output Attention (Q, K, V) obtained in S4-7 to the frame-level input sequence S, and pass through layer normalization and a feedforward neural network to obtain the frame-level encoding sequence F = {fr1, fr2,..., fr f″} after layer normalization and a feedforward neural network.

[0073] S5: Use the attention aggregation method to pool and statistically process the frame-level encoding sequence F obtained in step S4 to obtain the sentence-level encoding sequence U';

[0074] In this embodiment, the specific process of step S5 is as follows:

[0075] S5-1: According to the frame-level encoding sequence F = {fr1, fr2,..., fr f″} obtained in step S4, the scalar score of each frame encoding sequence is calculated by the attention mechanism, and the calculation process is as follows:

[0076] sco t = ReLU (W1·fr t )·W2

[0077] Where fr t represents the frame-level encoding sequence of the t-th frame, sco t represents the scalar score of the t-th frame, ReLU(.) is an activation function, and W1 and W2 are learnable parameters.

[0078] S5-2: According to the scalar score of each frame encoding sequence obtained in step S5-1, the weight of each scalar score is calculated, and the calculation process is as follows:

[0079]

[0080] where exp(.) denotes the exponential function, w t denotes the weight of the t-th frame.

[0081] S5-3: Weighted sum the frame-level encoding sequence F obtained in step S4 according to the weight obtained in step S5-2, to obtain the average frame encoding sequence μ, and the calculation process is as follows:

[0082]

[0083] S5-4: Calculate the standard deviation σ of all frames of the frame-level encoding sequence F obtained in step S4 and the average frame encoding sequence μ obtained in step S5-3, and the calculation process is as follows:

[0084]

[0085] where ⊙ denotes Hadamard product.

[0086] S5-5: Concatenate the average frame encoding sequence μ obtained in step S5-3 and the standard deviation encoding σ obtained in step S5-4 to obtain the concatenated sequence U.

[0087] S5-6: Regularize the concatenated sequence U obtained in step S5-5, and the processing process is as follows:

[0088]

[0089] where ||.||2 denotes the 2-norm, and U' denotes the sentence-level encoding sequence.

[0090] S6: Project the sequence U obtained in step S5 to obtain the prediction probability sequence P of each sample for each class, and the transformation process is as follows:

[0091] P = W p ·U + bias

[0092] where W p , bias are learnable parameters.

[0093] S7: Calculate the training loss based on the cross-entropy loss function, and minimize the training loss.

[0094] In this embodiment, the training loss based on the cross-entropy loss function in step S7 is The calculation process is as follows:

[0095]

[0096] where B represents batch size, N represents the number of dialect categories, y i，j represents the true label of the ith sample for the jth category, p i，j represents the prediction probability of the ith sample for the jth category.

[0097] The present application uses the KeSpeech dataset to test the effect of the present application. The KeSpeech dataset involves a total of 1542 hours of speech data from 27237 speakers, including Mandarin and its 8 sub-dialects with multiple labels, which can be applied to various tasks, and the present application mainly focuses on the dialect recognition task.

[0098] China has a vast geographical area, so there are various Mandarin dialects. Generally speaking, the official language is divided into 8 categories, including Northeastern Mandarin (NE), Ji-Lu Mandarin (Ji-Lu), Zhongyuan Mandarin (ZY), Jiao-Liao Mandarin (Jiao-Liao), Jianghuai Mandarin (JH), Southwestern Mandarin (SW), Lanyin Mandarin (LY), and Beijing Mandarin (BJ). Audio data is collected through the shell platform, with single-channel 16kHZ sampling. Since the number of Beijing Mandarin speech data is too small to be recognized, the present application deletes Beijing Mandarin from the KeSpeech data for subsequent testing of a total of 7 dialects.

[0099] The present application first simply fine-tunes the pre-trained self-supervised model and compares it to select the optimal framework as the speech feature extractor. In the fine-tuning stage, the present application selects wav2vec2.0-large, hubert-large and XLSR-53-large models to train for 10 epochs, with a batch size of 64. An average pooling layer and a randomly initialized linear layer are added after the pre-trained model to predict the dialect category. The output dimension of the pre-trained model is 1024. The present application uses the Adam optimizer for optimization, with a learning rate of 1×10 -4 for the first 2 epochs, and the learning rate linearly decays for the remaining epochs.

[0100] After the fine-tuning pre-test, the present application selects the hubert-large framework as the best feature extractor. The method proposed by the present application trains 7 dialects of KeSpeech, with a batch size of 64 and 10 iterations. The hubert-large model is frozen during the training process. In the first 3 epochs, the learning rate is raised to 1×10 -3 , and then it is lowered according to the exponential decay method with gamma = 0.5. The number of transformer layers is 6, the head of each transformer layer is 8, the model dimension is 512, and the hidden layer dimension is 2048. The present application uses the cost average performance (Cavg ), equal error rate (EER) and accuracy (Acc). The results are shown in Table 1. MFCCs are mel-frequency cepstral coefficients, Fbnaks are filter bank features, and the present application compares the current mainstream methods using MFCCs and Fbnaks features, including x-vector, ResNet-34 and ECAPA-TDNN. As can be seen from the table, the method proposed in the present application has better performance and higher classification accuracy than the traditional dialect recognition method.

[0101] Table 1

[0102]

[0103] In the present embodiment, a dialect classification system based on self-supervised speech representation is also provided, which is used to implement the above-mentioned embodiments. The terms "module", "unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible.

[0104] The dialect classification system based on self-supervised speech representation provided in the present embodiment comprises:

[0105] A data acquisition module is configured to acquire audio samples of different dialects and extract original waveform sequences of each audio sample.

[0106] A preprocessing module is configured to extract a speech representation sequence of the original waveform sequence using a pre-trained model and downsample to obtain a frame-level input sequence.

[0107] A Transformer encoder module is configured to encode the frame-level input sequence using a Transformer encoder to obtain a frame-level encoding sequence.

[0108] An attention aggregation module is configured to pool and statistically process the frame-level encoding sequence using an attention aggregation method to obtain a sentence-level encoding sequence.

[0109] A linear projection module is configured to linearly project the sentence-level encoding sequence to obtain a sample prediction probability.

[0110] A training module is configured to train the Transformer encoder and the trainable parameters in the attention aggregation method and the linear projection using audio samples of different dialects.

[0111] A classification module is configured to predict the dialect type in the audio based on the trained parameters.

[0112] The implementation process of the functions and roles of each module in the system is specifically described in the implementation process of the corresponding steps in the above method. For example, in the Transformer encoder module, the attention mechanism in the Transformer encoder is replaced by a content-based attention mechanism, specifically:

[0113] Obtaining a similarity matrix E according to the frame-level input sequence;

[0114] Extracting the first k maximum elements in each row of the similarity matrix E and the position information of the first k maximum elements in the matrix E to obtain a maximum value matrix L and a position information matrix P;

[0115] Zeroing all elements in the similarity matrix E;

[0116] Normalizing the maximum value matrix L by row to obtain a normalized maximum value matrix L';

[0117] According to the position information matrix P, the elements at the same position in the normalized maximum value matrix L' are filled into the zeroed similarity matrix E according to the stored position information;

[0118] Obtaining a frame-level encoding sequence F based on the updated similarity matrix E.

[0119] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts are described in the method embodiment. The implementation method of the remaining modules is not described here. The system embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Some or all modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0120] The system embodiment of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiment can be realized by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running through the processor of the device with data processing capability.

[0121] Obviously, the above-mentioned embodiments and drawings are only some examples of the present application, and for those skilled in the art, the present application can also be applied to other similar situations according to the drawings without the need for creative labor. In addition, it can be understood that although the work done in the development process here may be complex and long, but for those skilled in the art, some design, manufacture or production changes according to the technical content disclosed in the present application are only routine technical means and should not be regarded as insufficient disclosure of the present application. Without departing from the concept of the present application, a number of modifications and improvements can also be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A dialect classification method based on self-supervised speech representation, characterized in that, The method comprises the following steps: obtaining audio samples of different dialects, and extracting original waveform sequences of the audio samples; extracting speech representation sequences of the original waveform sequences using a pre-trained model and down-sampling to obtain frame-level input sequences; encoding the frame-level input sequences using a Transformer encoder to obtain frame-level encoding sequences; pooled statistical processing of the frame-level encoding sequences using an attention aggregation method to obtain sentence-level encoding sequences; linear projection of the sentence-level encoding sequences to obtain sample prediction probabilities; training the Transformer encoder and the trainable parameters in the attention aggregation method and linear projection using audio samples of different dialects, and predicting the dialect type in the audio based on the trained parameters.

2. The dialect classification method based on self-supervised speech representation according to claim 1, characterized in that, The pre-trained model is a HuBert-Large model.

3. The dialect classification method based on self-supervised speech representation according to claim 1, characterized in that, The down-sampling process comprises: using a convolutional neural network to perform two times down-sampling on the speech representation sequence R and using a LeakyReLU activation function to activate to obtain a sequence R'; using a convolutional neural network to perform two times down-sampling on the sequence R' and using a LeakyReLU activation function to activate to obtain a sequence R"; dimension reduction operation is performed on the sequence R" to finally obtain the frame-level input sequence, wherein the dimension of each frame in the frame-level input sequence is the input dimension of the Transformer encoder.

4. The dialect classification method based on self-supervised speech representation according to claim 1, characterized in that, When encoding the frame-level input sequence using the Transformer encoder, the attention mechanism in the Transformer encoder is replaced by a content-based attention mechanism, specifically: obtaining a similarity matrix E from the frame-level input sequence; extracting the maximum elements of the first k values in each row of the similarity matrix E and the position information of the maximum elements in the matrix E to obtain a maximum value matrix L and a position information matrix P; setting all elements in the similarity matrix E to zero; normalizing the maximum value matrix L by row to obtain a normalized maximum value matrix L'; filling the elements at the same position in the normalized maximum value matrix L' into the zeroed similarity matrix E according to the position information in the position information matrix P; obtaining the frame-level encoding sequence F based on the updated similarity matrix E.

5. The dialect classification method based on self-supervised speech representation according to claim 4, characterized in that, The similarity matrix E is obtained from the frame-level input sequence, comprising: projecting the frame-level input sequence to obtain a query set Q, a key value set K and a value set V; calculating the similarity matrix E from the query set Q and the key value set K; where d t denotes the input dimension of the Transformer encoder, and the superscript T denotes the transpose.

6. The dialect classification method based on self-supervised speech representation according to claim 5, characterized in that, The frame-level encoding sequence F is obtained based on the updated similarity matrix E, comprising: multiplying the updated similarity matrix E and the value set V to obtain the output of the attention mechanism; adding the attention mechanism output and the frame-level input sequence, and then performing layer normalization and feedforward neural network to obtain the frame-level encoding sequence F.

7. The dialect classification method based on self-supervised speech representation according to claim 4, characterized in that, The attention aggregation method specifically comprises: calculating the scalar score of each frame encoding sequence in the frame-level encoding sequence and the weight of each frame scalar score through the attention mechanism to calculate the average frame encoding sequence; calculating the standard deviation encoding according to each frame encoding sequence in the frame-level encoding sequence, the weight and the average frame encoding sequence; The frame-level encoding sequence is concatenated with the standard deviation encoding, and a sentence-level encoding sequence is obtained after regularization processing.

8. The dialect classification method based on self-supervised speech representation according to claim 7, characterized in that, The scalar score of each frame-level encoding sequence in the frame-level encoding sequence is calculated according to the following formula: sco t = ReLU(W1 · fr t ) · W2 where fr t sco t sco ReLU(.) is an activation function, and W1, W2 are learnable parameters.

9. The dialect classification method based on self-supervised speech representation according to claim 7, characterized in that, The calculation formula of the standard deviation encoding is: where σ denotes standard deviation coding, denotes Hadamard product, w t denotes frame-level coded sequence weight of the t-th frame, fr t denotes frame-level coded sequence of the t-th frame, f" denotes the number of frames in the frame-level coded sequence, and μ denotes average frame coded sequence.

10. A dialect classification system based on self-supervised speech representations, characterized in that, The method comprises the following steps: a data acquisition module configured to acquire audio samples of different dialects and extract original waveform sequences of the audio samples; a preprocessing module configured to extract speech representation sequences of the original waveform sequences by using a pre-trained model and to obtain frame-level input sequences by down-sampling; a Transformer encoder module configured to encode the frame-level input sequences by using a Transformer encoder to obtain frame-level encoding sequences; an attention aggregation module configured to pool and statistically process the frame-level encoding sequences by using an attention aggregation method to obtain a sentence-level encoding sequence; a linear projection module configured to linearly project the sentence-level encoding sequence to obtain a sample prediction probability; a training module configured to train the Transformer encoder and trainable parameters in the attention aggregation method and the linear projection by using audio samples of different dialects; a classification module configured to predict a dialect type in audio based on the trained parameters.

Images