An accent recognition method based on persistent accent memory
By constructing an encoder-decoder structure of the self-supervised pre-training model WavLM and the persistent accent memory PAM, the problem of strong data dependence of the end-to-end accent recognition system is solved, and efficient accent recognition is achieved under low-resource conditions.
Patent Information
- Application Number
- CN202310326132.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-30
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2043-03-30
AI Technical Summary
Existing end-to-end accent recognition systems are highly dependent on data, resulting in poor model robustness and high computational cost. In addition, due to the scarcity of accent data, recognition performance is poor in low-resource scenarios.
An encoder-decoder infrastructure based on the self-supervised pre-training model WavLM is constructed. The representation features of the English spoken training set are extracted through the self-supervised pre-training model. Combined with the persistent accent memory PAM, the model is optimized to improve performance.
It significantly improves the accuracy of accent recognition under low-resource conditions, reduces computational costs, and improves the robustness of the model.
Smart Images
Figure CN116486790B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech recognition, and in particular to an accent recognition method based on persistent accent memory. Background Art
[0002] Currently, the mainstream framework for speech recognition is the end-to-end (E2E) accent recognition system. However, training a mature end-to-end system requires a large amount of data. Due to the scarcity of accent data, most current methods focus on using various data augmentation methods to expand the data and thus improve the performance of accent recognition.
[0003] Since existing technologies are dependent on data, the quality of the data will affect the performance of the model, resulting in poor robustness of the model. In addition, a large amount of training data will also lead to an increase in computing costs, which is not conducive to model training. Summary of the Invention
[0004] The present application provides an accent recognition method based on persistent accent memory, which can solve the problem of existing accent recognition methods that rely on data quality and quantity.
[0005] The technical solution of this application is an accent recognition method based on persistent accent memory, comprising:
[0006] S1: Build the WavLM-encoder-decoder infrastructure including the self-supervised pre-trained model WavLM, where the decoder includes an AR module for accent recognition.
[0007] S2: obtaining a spoken English training set including several audio segments covering different accents, inputting the spoken English training set into a WavLM-encoder-decoder infrastructure, obtaining encoder outputs accordingly, and sequentially performing adjustment processing and clustering processing on the encoder outputs to obtain a persistent accent memory (PAM);
[0008] S3: Optimizing the WavLM-encoder-decoder infrastructure. The optimization includes adjusting the AR module to receive PAM and fusing the encoder output with PAM, thereby obtaining an optimized WavLM-encoder-decoder architecture.
[0009] S4: The optimized WavLM encoder-decoder structure is trained using the spoken English training set, and an accent recognition model is obtained that takes the audio to be recognized as input and outputs the accent recognition result;
[0010] S5: Acquire the audio to be recognized and input the audio to be recognized into the accent recognition model to obtain an accent recognition result corresponding to the audio to be recognized.
[0011] Optionally, step S1 includes:
[0012] S11: Build encoder-decoder infrastructure;
[0013] S12: The self-supervised pre-training model WavLM is pre-placed to construct an encoder-decoder infrastructure, and a WavLM-encoder-decoder infrastructure is correspondingly obtained, wherein the decoder includes an ASR module for speech recognition and an AR module for accent recognition.
[0014] Optionally, step S2 includes:
[0015] S21: Obtain an English spoken training set and an English spoken test set, each of which includes a plurality of audio segments covering different accents, and input the English spoken training set into a WavLM-encoder-decoder infrastructure;
[0016] S22: The self-supervised pre-training model WavLM performs feature extraction on the spoken English training set to obtain SSLRs corresponding to the spoken English training set, and obtains a plurality of encoder-decoder optional structures through the SSLRs to the encoder-decoder infrastructure.
[0017] S23: inputting the spoken English test set into several encoder-decoder optional structures respectively, and obtaining encoder outputs corresponding to different encoder-decoder optional structures and test results including average recognition accuracy;
[0018] S24: Selecting, in accordance with the test results, the encoder output corresponding to the accent output by the encoder-decoder optional structure having the highest average recognition accuracy corresponding to each accent, thereby obtaining an encoder output set including a plurality of vectors corresponding to different accents;
[0019] S25: concatenate the mean and standard deviation of the vectors in the encoder output set to obtain a processed data set;
[0020] The splicing formula is as follows:
[0021]
[0022] Where, represents the encoder output of the i-th audio corresponding to the j-th accent;
[0023] represents the data item in the processed dataset corresponding to the encoder output;
[0024] S23: Perform K-means clustering on the processed data set to obtain a persistent accent memory (PAM) comprising several embedding groups corresponding to different accents, each embedding group comprising several embeddings;
[0025] The clustering processing formula is as follows:
[0026]
[0027] E PAM ={E 1 ,...,E J};
[0028] Where C represents the number of embeddings in the embedding group.
[0029] Optionally, each embedding group includes 32 embeddings.
[0030] Optionally, step S3 includes:
[0031] S31: Optimizing the WavLM-encoder-decoder infrastructure, the optimization including: setting a filtering module capable of filtering the PAM based on the cosine distance and obtaining a plurality of filtered embeddings corresponding to the encoder output, and adjusting the AR module to receive the filtered embeddings and further perform fusion processing on the encoder output and the filtered embeddings, thereby obtaining an optimized WavLM-encoder-decoder architecture.
[0032] The screening formula of the screening module is as follows:
[0033]
[0034]
[0035] Where, Consine-dist represents the cosine distance;
[0036] g i represents Gaussian noise that follows a Gaussian distribution;
[0037] T represents the temperature control parameter;
[0038] And, the formula for the fusion process is as follows:
[0039]
[0040] Where MHA represents the multi-head attention in the AR module.
[0041] Beneficial effects:
[0042] This application constructs a WavLM-encoder-decoder infrastructure including a self-supervised pre-training model WavLM. This model can be used to extract representations (SSLRs) of the spoken English training set and replace traditional acoustic features (Fbank) with SSLRs, significantly improving system performance. This can address the problem of scarce accent data and poor accent recognition in low-resource scenarios.
[0043] In addition, this application further improves the performance of the system by integrating PAM into the WavLM-encoder-decoder infrastructure, guiding the model to fully utilize the information carried by SSLRs during training. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0045] Figure 1 Schematic diagram of the flow of an accent recognition method based on persistent accent memory in an embodiment of the present application;
[0046] Figure 2 This is a schematic diagram of the basic structure of the encoder-decoder in the embodiment of the present application.
[0047] Figure 3 This is the Transformer model architecture in the embodiment of this application;
[0048] Figure 4 This is a schematic diagram of the structure of the AR module in the decoder in an embodiment of the present application;
[0049] Figure 5 This is a schematic diagram of the encoder-decoder adjustment structure in an embodiment of the present application;
[0050] Figure 6 Schematic diagram of the process of cross-attention fusion in the decoder in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.
[0052] This application provides an accent recognition method based on persistent accent memory, such as Figure 1 As shown, Figure 1 The following is a flow chart of an accent recognition method based on persistent accent memory according to an embodiment of the present application. The method includes:
[0053] S1: Build the WavLM-encoder-decoder infrastructure including the self-supervised pre-trained model WavLM, where the decoder includes an AR module for accent recognition.
[0054] Wherein, step S1 includes:
[0055] S11: Build encoder-decoder infrastructure;
[0056] S12: The self-supervised pre-training model WavLM is pre-placed to construct an encoder-decoder infrastructure, and a WavLM-encoder-decoder infrastructure is correspondingly obtained, wherein the decoder includes an ASR module for speech recognition and an AR module for accent recognition.
[0057] Specifically, if Figure 2 As shown, Figure 2 This is a schematic diagram of the encoder-decoder infrastructure in an embodiment of the present application.
[0058] The backbone network of the accent recognition model in the embodiment of the present application adopts an encoder-decoder architecture and uses automatic speech recognition (ASR) task-assisted training.
[0059] The encoder parameters are shared and consist of a 12-layer Transformer encoder, such as Figure 3 As shown on the left side, Figure 3 This is the Transformer model architecture in the embodiment of this application.
[0060] The decoder includes an ASR module for speech recognition and an AR module for accent recognition. The ASR module consists of a connectionist temporal classification (CTC) and a 6-layer ASR Attention module. Figure 3 Shown on the right.
[0061] like Figure 4 As shown, Figure 4 This is a structural diagram of the AR module in the decoder in an embodiment of the present application. The accent recognition (AR) part is composed of the Accent Attention.
[0062] S2: Obtain an English spoken training set including several audio segments covering different accents, and input the English spoken training set into the WavLM-encoder-decoder infrastructure to obtain the encoder output accordingly, and perform adjustment processing and clustering processing on the encoder output in sequence to obtain a persistent accent memory PAM.
[0063] S21: obtaining an English spoken training set and an English spoken test set, both of which include several audio segments covering different accents, and inputting the English spoken training set into the WavLM-encoder-decoder infrastructure.
[0064] Specifically, the embodiment of the present application uses an English accent dataset, which contains a total of 160 hours of training data and includes 8 accents.
[0065] S22: The self-supervised pre-training model WavLM performs feature extraction on the spoken English training set to obtain SSLRs corresponding to the spoken English training set, and obtains several encoder-decoder optional structures through the SSLRs to the encoder-decoder infrastructure.
[0066] Specifically, the self-supervised pre-trained model WavLM is used to extract SSLRs of the English spoken training set at different encoding layers.
[0067] The self-supervised pre-training model WavLM includes a 24-layer encoder. The outputs of the 8th, 12th, 16th, 20th, and 24th layers of WavLM, namely SSLRs, are used as the input of the encoder-decoder infrastructure for training, resulting in systems 1 to 5 as shown in Table 1.
[0068] Table 1 Average accuracy of accent recognition using encoder-decoder alternative structures trained with SSLRs
[0069]
[0070] S23: Inputting the spoken English test set into several encoder-decoder optional structures respectively, and accordingly obtaining encoder outputs corresponding to different encoder-decoder optional structures and test results including average recognition accuracy.
[0071] Specifically, five encoder-decoder optional structures, namely systems 1-5 shown in Table 1, were tested using the spoken English test set, and data such as the average accuracy shown in Table 1 were obtained.
[0072] S24: Based on the test results, select in turn the encoder output corresponding to the accent output by the encoder-decoder optional structure with the highest average recognition accuracy corresponding to each accent, and obtain an encoder output set including several vectors corresponding to different accents.
[0073] Specifically, based on the test set results, the model with the highest accuracy for the current accent is selected to extract the encoder outputs for all audio in the training set labeled with that accent. For example, for an American accent, System 2 is selected, and for a Chinese accent, System 5 is used. This means that all audio from the eight accents in the spoken English test set is extracted using different systems to generate 8 x 10,000 vectors.
[0074] S25: Concatenate the mean and standard deviation of the vectors in the encoder output set to obtain a processed data set.
[0075] The splicing formula is as follows:
[0076]
[0077] Where, represents the encoder output of the i-th audio corresponding to the j-th accent.
[0078] Represents the data item in the processed dataset that corresponds to the encoder output.
[0079] S26: Perform K-means clustering on the processed data set to obtain a persistent accent memory PAM including several embedding groups corresponding to different accents, each embedding group including several embeddings.
[0080] The clustering processing formula is as follows:
[0081]
[0082] E PAM ={E 1 ,...,E J}.
[0083] Where C represents the number of embeddings in the embedding group.
[0084] Specifically, the encoder output is concatenated with the mean and standard deviation, and then K-means is used to cluster each accent into 32 embeddings. Each embedding has 256 dimensions, and there are 8 accents, for a total of 256 embeddings, forming the PAM.
[0085] The splicing formula and clustering processing formula are as follows:
[0086]
[0087] E PAM ={E 1 ,...,E J}.
[0088] Where, represents the encoder output of the i-th audio corresponding to the j-th accent.
[0089] Represents the data items in the processed data set corresponding to the encoder output, C=32, J=8.
[0090] PAM can be understood as a codebook, a matrix obtained by clustering the encoder output. In this embodiment of the present application, the spoken English training set has 8 accents, each with 10,000 audio clips. The encoder output consists of 8 * 10,000 embeddings, each with a dimension of 256. Embeddings are vectors, meaning each accent is clustered into 32 vectors, ultimately resulting in a three-dimensional matrix of (8, 32, 256), which is called a PAM.
[0091] S3: Optimize the WavLM-encoder-decoder infrastructure. The optimization includes adjusting the AR module to receive PAM and fusing the encoder output with PAM, thereby obtaining an optimized WavLM-encoder-decoder structure.
[0092] Wherein, step S3 includes:
[0093] S31: Optimize the WavLM-encoder-decoder infrastructure, which includes setting a filtering module that can filter PAM based on cosine distance and obtain several filtered Embeddings corresponding to the encoder output, and adjusting the AR module to receive the filtered Embeddings and also perform fusion processing on the encoder output and the filtered Embeddings, thereby obtaining the WavLM-encoder-decoder optimized structure accordingly.
[0094] The filtering formula of the filtering module is as follows:
[0095]
[0096] Where Consine-dist represents the cosine distance.
[0097] g i represents Gaussian noise that follows a Gaussian distribution.
[0098] T represents the temperature control parameter;
[0099] And, the formula for the fusion process is as follows:
[0100]
[0101] Where MHA represents the multi-head attention in the AR module.
[0102] Specifically, (1) PAM screening part:
[0103] PAM can provide effective additional information during model training, but PAM includes 8 accents. When the model is trained, the information of 8 accents must be considered each time, which will lead to program redundancy. Therefore, an optimal PAM selection method is proposed.
[0104] like Figure 4 As shown in the figure, the similarity between the encoder output and the 256 embeddings in PAM is first calculated based on the cosine distance. The N most similar embeddings are selected based on the similarity score, concatenated with the encoder output in time sequence, and then input into the multi-head attention module for accent classification.
[0105] In some embodiments, the common dot product may also be used to calculate the similarity.
[0106] The filtering formula of the filtering module is as follows:
[0107]
[0108] Where Consine-dist represents the cosine distance.
[0109] g i represents Gaussian noise that follows a Gaussian distribution.
[0110] T represents the temperature control parameter;
[0111] (2) Fusion processing part: The formula of the fusion processing is as follows:
[0112]
[0113] Where MHA represents the multi-head attention in the AR module.
[0114] As well as Figure 5 and Figure 6 As shown, Figure 5 This is a schematic diagram of the encoder-decoder adjustment structure in an embodiment of the present application. Figure 6 This is a flow chart of the cross-attention fusion in the decoder in the embodiment of the present application. Based on the cross-attention method, the PAM is simply fused with the output of the encoder in the Accent Attention module, and the encoder-decoder adjustment structure is obtained accordingly.
[0115] The formula for the fusion process is as follows:
[0116]
[0117] Where MHA represents the multi-head attention in the AR module.
[0118] S4: The WavLM-encoder-decoder optimization structure is trained using the spoken English training set, and accordingly an accent recognition model is obtained that takes the audio to be recognized as input and the accent recognition result as output.
[0119] Specifically, after obtaining the WavLM-encoder-decoder optimized structure, it can be trained using a data set, and after training, an accent recognition model can be obtained.
[0120] S5: Acquire the audio to be recognized and input the audio to be recognized into the accent recognition model to obtain the accent recognition result corresponding to the audio to be recognized.
[0121] Specifically, as shown in Table 1, Systems 1 to 7 are the results of training models using SSLRs extracted from different coding layers of WavLM. Compared with System 0 trained with traditional acoustic features (Fbank), the average accuracy is absolutely improved by 6.1%, proving that the use of pre-trained models can significantly improve the performance of accent recognition.
[0122] As shown in Table 2, System 8 uses a cross-attention fusion system, achieving an absolute average accuracy improvement of 0.1% compared to the common weighted sum method (System 6). The optimal PAM selection method proposed in this embodiment, as shown in System 11, achieved an average accuracy of 81.4%, further improving the average accuracy of accent recognition. Finally, for Systems 9 and 10, we constructed PAMs using Systems 5 and 6, respectively, achieving an average accuracy of 80.9%. This demonstrates the versatility of our proposed method. In other words, even without prior knowledge of which system performs best for a particular accent, our proposed method can still improve accent recognition performance.
[0123] Table 2 Average accuracy of accent recognition models based on the optimal PAM selection method
[0124]
[0125] The above detailed description of the embodiments of the present application is only for the preferred embodiments of the present application and should not be considered as limiting the scope of implementation of the present application. All equivalent changes and improvements made within the scope of the present application should still fall within the scope of the patent application.
Claims
1. An accent recognition method based on persistent accent memory, characterized in that: include: S1: Build the WavLM-encoder-decoder infrastructure including the self-supervised pre-trained model WavLM, where the decoder includes an AR module for accent recognition. S2: obtaining a spoken English training set including several audio segments covering different accents, inputting the spoken English training set into a WavLM-encoder-decoder infrastructure, obtaining encoder outputs accordingly, and sequentially performing adjustment processing and clustering processing on the encoder outputs to obtain a persistent accent memory (PAM); S3: Optimizing the WavLM-encoder-decoder infrastructure. The optimization includes adjusting the AR module to receive PAM and fusing the encoder output with PAM, thereby obtaining an optimized WavLM-encoder-decoder architecture. S4: The optimized WavLM encoder-decoder structure is trained using the spoken English training set, and an accent recognition model is obtained that takes the audio to be recognized as input and outputs the accent recognition result; S5: Acquire the audio to be recognized and input the audio to be recognized into the accent recognition model to obtain an accent recognition result corresponding to the audio to be recognized.
2. The accent recognition method based on persistent accent memory according to claim 1, characterized in that: The step S1 comprises: S11: Build encoder-decoder infrastructure; S12: The self-supervised pre-training model WavLM is pre-placed to construct an encoder-decoder infrastructure, and a WavLM-encoder-decoder infrastructure is correspondingly obtained, wherein the decoder includes an ASR module for speech recognition and an AR module for accent recognition.
3. The accent recognition method based on persistent accent memory according to claim 2, characterized in that: The step S2 comprises: S21: Obtain an English spoken training set and an English spoken test set, each of which includes a plurality of audio segments covering different accents, and input the English spoken training set into a WavLM-encoder-decoder infrastructure; S22: The self-supervised pre-training model WavLM performs feature extraction on the spoken English training set to obtain SSLRs corresponding to the spoken English training set, and obtains a plurality of encoder-decoder optional structures through the SSLRs to the encoder-decoder infrastructure. S23: inputting the spoken English test set into several encoder-decoder optional structures respectively, and obtaining encoder outputs corresponding to different encoder-decoder optional structures and test results including average recognition accuracy; S24: Selecting, in accordance with the test results, the encoder output corresponding to the accent output by the encoder-decoder optional structure having the highest average recognition accuracy corresponding to each accent, thereby obtaining an encoder output set including a plurality of vectors corresponding to different accents; S25: concatenate the mean and standard deviation of the vectors in the encoder output set to obtain a processed data set; The splicing formula is as follows: Where, represents the encoder output of the i-th audio corresponding to the j-th accent; represents the data item in the processed dataset corresponding to the encoder output; S23: Perform K-means clustering on the processed data set to obtain a persistent accent memory (PAM) comprising several embedding groups corresponding to different accents, each embedding group comprising several embeddings; The clustering processing formula is as follows: AND PAM ={And 1 ,...,AND J }; Where C represents the number of embeddings in the embedding group.
4. The accent recognition method based on persistent accent memory according to claim 3, characterized in that: Each embedding group includes 32 embeddings.
5. The accent recognition method based on persistent accent memory according to claim 3, characterized in that: The step S3 comprises: S31: Optimizing the WavLM-encoder-decoder infrastructure, the optimization including: setting a filtering module capable of filtering the PAM based on the cosine distance and obtaining a plurality of filtered embeddings corresponding to the encoder output, and adjusting the AR module to receive the filtered embeddings and further perform fusion processing on the encoder output and the filtered embeddings, thereby obtaining an optimized WavLM-encoder-decoder architecture. The screening formula of the screening module is as follows: Where, Consine-dist represents the cosine distance; g i represents Gaussian noise that follows a Gaussian distribution; T represents the temperature control parameter; And, the formula for the fusion process is as follows: Where MHA represents the multi-head attention in the AR module.