Non-native Chinese speech recognition method and system based on offset and convolution adaptation
By introducing Token Bias Adapter and convolutional adaptive module into the pre-trained model, selective training is implemented, which solves the problem of insufficient adaptability of existing models to non-native accent data, improves recognition accuracy and adaptability, and maintains the Lightweight properties of the model.
Patent Information
- Application Number
- CN202411783635.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-02-25
- Estimated Expiration
- 2044-12-06
AI Technical Summary
The existing non-native Chinese speech recognition model lacks adaptability when processing non-native accent data, and the traditional fine-tuning method consumes a lot of computing resources, making it difficult to adapt to multi-task learning scenarios.
Using non-native Chinese speech recognition method based on offset and convolution adaptation, selective training is achieved by inserting Token Bias Adapter (TBA) module and convolution adaptation module into the pre-trained model. Improve the adaptability of the model.
It significantly improves the model's adaptability to specific downstream tasks, enhances recognition accuracy, and performs well in multilingual environments, and maintains the lightweight nature of the model.
Smart Images
Figure CN119252235B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a non-native Chinese speech recognition method and system based on offset and convolution adaptation. Background Art
[0002] The existing data on non-native accents is scarce, and the training data for most models is based on the standard pronunciation of the native speaker, which results in the lack of adaptability of the models to the different accents of non-native learners. Low-resource speech recognition refers to speech recognition research conducted with limited data resources. It usually involves non-mainstream languages or dialects, whose annotated data is scarce and difficult to use to train powerful speech recognition systems. In this context, researchers have tried to use various techniques such as transfer learning, multi-task learning, self-supervised learning, etc. to improve the performance of the system. For example, pre-trained models such as wav2vec 2.0 and HuBERT have shown significant improvements in low-resource speech recognition in multilingual environments.
[0003] In low-resource accented speech recognition tasks, fine-tuning of self-supervised pre-trained models can lead to performance improvements. However, traditional fine-tuning methods are computationally expensive and difficult to adapt to multi-task learning scenarios, such as personalized ASR. To address these issues, adaptive fine-tuning methods have emerged. Adaptive fine-tuning avoids large-scale modifications to the encoder parameters by inserting lightweight adaptive modules between the layers of the pre-trained encoder and only updating the parameters of these modules. Summary of the invention
[0004] The purpose of the present invention is to solve the problems in the prior art.
[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a non-native Chinese speech recognition method based on offset and convolution adaptation, comprising the following steps:
[0006] Select a pre-trained model to build a non-native Chinese speech recognition model;
[0007] Use selective training to train non-native Chinese speech recognition models;
[0008] Use the trained non-native Chinese speech recognition model to realize non-native Chinese speech recognition;
[0009] The non-native Chinese speech recognition model includes a preprocessing layer, a transformer encoder and a fully connected layer connected in sequence;
[0010] The preprocessing layer converts the input raw speech signal into a feature vector;
[0011] The transformer encoder includes multiple encoder layers connected in sequence, each encoder layer includes a self-attention sublayer, a TBA module, a feedforward layer and a convolutional adaptive module connected in sequence; the self-attention sublayer converts the input signal into a speech frame, the input signal of the self-attention sublayer of the first encoder layer is the feature vector output by the preprocessing layer, and the input signal of the self-attention sublayer of other encoder layers is the feature weighted vector output by the previous encoder layer; the TBA module generates importance weights and bias items according to the speech frame, and outputs the bias vector to the feedforward layer; the feedforward layer converts the bias vector into a representation vector; the convolutional adaptive module converts the representation vector into a feature weighted vector;
[0012] The fully connected layer outputs a transformation vector based on the feature weighted vector output by the last encoder layer, and the predicted text is obtained through the transformation vector.
[0013] Preferably, the selecting a pre-trained model to construct a non-native Chinese speech recognition model comprises the following steps:
[0014] Select the pre-trained speech model wav2vec2.0 as the pre-trained model;
[0015] Insert the TBA module between the self-attention sublayer and the feedforward layer of the pre-trained speech model wav2vec2.0, and add a convolutional adaptation module and a fully connected layer after the feedforward layer;
[0016] The TBA module includes a trainable bias vector and a linear layer ; The convolution adaptive module includes layer normalization, three one-dimensional convolutions and a compression excitation module.
[0017] Preferably, the self-attention sublayer converts the input signal into a speech frame, specifically: the output of the input signal after passing through the self-attention sublayer is added to the input signal through a residual connection, and then layer normalization is performed to obtain a speech frame , represents the i-th speech frame segment, Indicates the number of speech frame segments.
[0018] Preferably, the TBA module generates importance weights and bias items according to the speech frames and outputs the bias vector to the feedforward layer, specifically:
[0019] Speech Frame Input to TBA module, linear layer Generate corresponding speech frames Importance weight , expressed as:
[0020] ;
[0021] in, For the The weight of a speech frame, The value is 1, 2, ... ;
[0022] By speech frame Importance weight With the bias vector Weighted, bias term generated , expressed as:
[0023] ;
[0024] in, represents the Hadamard product;
[0025] The bias term With speech frame Perform residual connection to obtain context-dependent bias vector , expressed as:
[0026] .
[0027] Preferably, the convolution adaptive module converts the representation vector into a feature weighted vector, comprising the following steps:
[0028] After the representation vector is aligned, it enters the convolutional adaptive module and is normalized to obtain ;
[0029] Will Input into three one-dimensional convolutional layers in sequence to obtain the convolution representation vector , the calculation formula is as follows:
[0030] ;
[0031] in, , , Represent the operations of the first to third one-dimensional convolutional layers respectively;
[0032] Convolution represents vector Input to the compression excitation module to obtain the feature weighted vector .
[0033] Preferably, the compression excitation module obtains a feature weighted vector The process includes the following steps:
[0034] The convolution of the input vector Perform global average pooling to obtain a global pooling vector containing all speech frames and all channels , expressed as:
[0035] ;
[0036] in, is the number of speech frames, is the channel dimension of the speech frame, for In the The eigenvalues on the channels;
[0037] For the global pooling vector Perform the transformation of two fully connected layers to obtain a global transformation vector , expressed as:
[0038] ;
[0039] in, and Respectively represent the Sigmoid activation function and the ReLU activation function, and Represent the weight matrices of the two fully connected layers respectively;
[0040] Representing vectors for convolution and the global transformation vector Multiply to get the feature weight vector .
[0041] Preferably, the non-native Chinese speech recognition model is trained in a selective training manner, specifically: only the parameters of the TBA module and the convolutional adaptive layer are trained, and the parameters of the pre-trained model are frozen;
[0042] The optimization function used in training is expressed as:
[0043] ;
[0044] ;
[0045] ;
[0046] in, are the parameters of the TBA module and the convolutional adaptation layer, is the optimal parameter after training, yes Loss function, is the size of the training dataset, It is The conversion vector output by the model after the speech sample passes through, It is The label sequence of speech samples, Is The derived conditional probability, is the learning rate.
[0047] Preferably, the method of realizing non-native Chinese speech recognition using a trained non-native Chinese speech recognition model comprises the following steps:
[0048] Input the original speech signal to be recognized into the trained non-native Chinese speech recognition model to obtain the conversion vector ;
[0049] According to the pre-set word and vector conversion table, the conversion vector The probability of the candidate character sequence is derived, and the character sequence with the highest probability is selected as the predicted text, which is expressed as:
[0050] ;
[0051] in, Indicates predictive text, According to the transformation vector The derived conditional probability.
[0052] The present invention also provides a non-native Chinese speech recognition system based on offset and convolution adaptation, comprising:
[0053] Model building module, select the pre-trained model to build a non-native Chinese speech recognition model;
[0054] Model training module, which uses selective training to train non-native Chinese speech recognition models;
[0055] Speech recognition module, which uses the trained non-native Chinese speech recognition model to realize non-native Chinese speech recognition;
[0056] The non-native Chinese speech recognition model includes a preprocessing layer, a transformer encoder and a fully connected layer connected in sequence;
[0057] The preprocessing layer converts the input raw speech signal into a feature vector;
[0058] The transformer encoder includes multiple encoder layers connected in sequence, each encoder layer includes a self-attention sublayer, a TBA module, a feedforward layer and a convolutional adaptive module connected in sequence; the self-attention sublayer converts the input signal into a speech frame, the input signal of the self-attention sublayer of the first encoder layer is the feature vector output by the preprocessing layer, and the input signal of the self-attention sublayer of other encoder layers is the feature weighted vector output by the previous encoder layer; the TBA module generates importance weights and bias items according to the speech frame, and outputs the bias vector to the feedforward layer; the feedforward layer converts the bias vector into a representation vector; the convolutional adaptive module converts the representation vector into a feature weighted vector;
[0059] The fully connected layer outputs a transformation vector based on the feature weighted vector output by the last encoder layer, and the predicted text is obtained through the transformation vector.
[0060] The present invention has the following beneficial effects:
[0061] (1) In order to improve the transferability of task domains, the present invention significantly improves the adaptability of the model to specific downstream tasks by introducing a task representation offset (Token Bias Adapter, TBA) based on speech tokens in the intermediate representation of the pre-trained speech model. The TBA module dynamically adjusts the representation offset according to the importance of each speech token, which not only retains the general language knowledge obtained in the pre-training stage, but also enables the model to more accurately map the representation vector to the relevant task domain, especially in scenarios such as non-native Chinese speech recognition;
[0062] (2) This paper introduces a lightweight adaptive adapter ConvAdapter based on convolutional neural networks (CNNs), i.e., a convolution adaptive module, which effectively extracts and utilizes the encoding knowledge in the pre-trained model by fusing the channel information within the local receptive field while maintaining the lightweight characteristics of the model. This design enables ConvAdapter to perform well in learning specific task information without significantly increasing the computational burden of the model;
[0063] (3) Combining the technical features of TBA and convolutional adaptive modules, the present invention demonstrates significant beneficial effects in non-native Chinese speech recognition tasks. By improving the model's understanding of the importance of speech tokens and effectively extracting encoded knowledge, this technical combination not only enhances the model's recognition accuracy, but also improves its adaptability in multilingual environments or global applications.
[0064] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1A method step diagram of an embodiment of the present invention;
[0066] Figure 2 4 is a system architecture diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0067] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:
[0068] S101, selecting a pre-trained model to build a non-native Chinese speech recognition model;
[0069] S102, training a non-native Chinese speech recognition model by selective training;
[0070] S103, realizing non-native Chinese speech recognition using the trained non-native Chinese speech recognition model;
[0071] The non-native Chinese speech recognition model includes a preprocessing layer, a transformer encoder and a fully connected layer connected in sequence;
[0072] The preprocessing layer converts the input raw speech signal into a feature vector;
[0073] The transformer encoder includes multiple encoder layers connected in sequence, each encoder layer includes a self-attention sublayer, a TBA module, a feedforward layer and a convolutional adaptive module connected in sequence; the self-attention sublayer converts the input signal into a speech frame, the input signal of the self-attention sublayer of the first encoder layer is the feature vector output by the preprocessing layer, and the input signal of the self-attention sublayer of other encoder layers is the feature weighted vector output by the previous encoder layer; the TBA module generates importance weights and bias items according to the speech frame, and outputs the bias vector to the feedforward layer; the feedforward layer converts the bias vector into a representation vector; the convolutional adaptive module converts the representation vector into a feature weighted vector;
[0074] The fully connected layer outputs a transformation vector based on the feature weighted vector output by the last encoder layer, and the predicted text is obtained through the transformation vector.
[0075] Specifically, the step of selecting a pre-trained model to construct a non-native Chinese speech recognition model in S101 includes the following steps:
[0076] Select the pre-trained speech model wav2vec2.0 as the pre-trained model;
[0077] Insert the TBA module between the self-attention sublayer and the feedforward layer of the pre-trained speech model wav2vec2.0, and add a convolutional adaptation module and a fully connected layer after the feedforward layer;
[0078] The TBA module includes a trainable bias vector and a linear layer The convolutional adaptive module includes layer normalization, three one-dimensional convolutions and a compression excitation module. The convolutional adaptive module extracts the encoding knowledge in the pre-trained model parameters by fusing the channel information in the local receptive field.
[0079] Specifically, S102 includes constructing and obtaining a Chinese accent dataset from multiple countries as a speech recognition model training dataset. The dataset is , any of the voice , N=22753 is the size of the data set. See Table 1 for data description:
[0080] Table 1 Description of the Chinese accent dataset of non-native Chinese learners from multiple countries
[0081]
[0082] Specifically, the non-native Chinese speech recognition model processes the input audio signal The process includes the following steps:
[0083] S201, given audio signal , when the input signal After passing through the self-attention layer and feed-forward layer of each block of the transformer encoder, the corresponding speech frame is output =token[ , , These speech frame tokens are then fed into the TBA module, which generates their corresponding weights based on the importance of the speech frame tokens. .
[0084] S202, through the bias vector Speech Frame Importance weight Weighted Generation Bias .
[0085] S203, speech frame Importance weight With the bias term Weighted generation bias vector .
[0086] S204, will After the output of the self-attention layer and the feed-forward layer of the transformer encoder module, the output representation vector is obtained , after alignment and layer normalization, we get Then, Input into three one-dimensional convolutional layers in sequence to obtain the convolution representation vector .
[0087] S205, Input to the compression excitation module to obtain the feature weighted vector .
[0088] Specifically, the compression excitation module in S205 includes three operations: global pooling Squeeze, two fully connected layers Excitation and a Sigmoid activation function Scale; S205 includes the following operations:
[0089] Squeeze operation: Perform global average pooling on the input to obtain a global pooling vector containing all speech frames and all channels ;
[0090] Excitation operation: global pooling vector Perform the transformation of two fully connected layers to obtain a global transformation vector ;
[0091] Scale operation: Yes and the global transformation vector Multiply to get the feature weight vector .
[0092] Specifically, the S102 is specifically: setting the optimization function by adopting a selective training method, that is, only training the parameters of the TBA module and the convolutional adaptive layer, and freezing the parameters of the pre-trained model; Loss Function It is expressed as:
[0093] ;
[0094] Next, The loss function is iteratively optimized, and beam search is used to select the most likely character sequence from the model's output as the algorithm for text output.
[0095] Specifically, the S103 includes the following steps:
[0096] Input the original speech signal to be recognized into the trained non-native Chinese speech recognition model to obtain the conversion vector ;
[0097] According to the pre-set word and vector conversion table, the conversion vector The probability of the candidate character sequence is derived, and the character sequence with the highest probability is selected as the predicted text using beam search, which is expressed as:
[0098] ;
[0099] in, Indicates predictive text, According to the transformation vector The derived conditional probability.
[0100] See also Figure 2 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:
[0101] Model building module 201, selecting a pre-trained model to build a non-native Chinese speech recognition model;
[0102] A model training module 202, which trains a non-native Chinese speech recognition model by selective training;
[0103] The speech recognition module 203 implements non-native Chinese speech recognition using the trained non-native Chinese speech recognition model.
[0104] It can be seen that this patent significantly improves the adaptability of the model to specific downstream tasks by introducing a task representation offset (Token Bias Adapter, TBA) based on speech tokens in the intermediate representation of the pre-trained speech model: the TBA module dynamically adjusts the representation offset according to the importance of each speech token, which not only retains the general language knowledge obtained in the pre-training stage, but also enables the model to more accurately map the representation vector to the relevant task domain, especially in scenarios such as non-native Chinese speech recognition. In addition, the patent also introduces a lightweight adaptive adapter ConvAdapter based on convolutional neural networks (CNNs), that is, a convolutional adaptive module, which effectively extracts and utilizes the encoded knowledge in the pre-trained model by fusing the channel information within the local receptive field, while maintaining the lightweight characteristics of the model. This design enables ConvAdapter to perform well in learning specific task information without significantly increasing the computational burden of the model. Combining the technical features of TBA and ConvAdapter, this patent shows significant beneficial effects in non-native Chinese speech recognition tasks. By improving the model's understanding of the importance of speech tokens and effectively extracting encoded knowledge, this combination of technologies not only enhances the model's recognition accuracy, but also improves its adaptability in multilingual environments or global applications.
[0105] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A non-native Chinese speech recognition method based on offset and convolution adaptation, characterized in that: The following steps are involved: Select a pre-trained model to build a non-native Chinese speech recognition model; Use selective training to train non-native Chinese speech recognition models; Use the trained non-native Chinese speech recognition model to realize non-native Chinese speech recognition; The non-native Chinese speech recognition model includes a preprocessing layer, a transformer encoder and a fully connected layer connected in sequence; The preprocessing layer converts the input raw speech signal into a feature vector; The transformer encoder includes multiple encoder layers connected in sequence, each encoder layer includes a self-attention sublayer, a TBA module, a feedforward layer and a convolutional adaptive module connected in sequence; the self-attention sublayer converts the input signal into a speech frame, the input signal of the self-attention sublayer of the first encoder layer is the feature vector output by the preprocessing layer, and the input signal of the self-attention sublayer of other encoder layers is the feature weighted vector output by the previous encoder layer; the TBA module generates importance weights and bias items according to the speech frame, and outputs the bias vector to the feedforward layer; the feedforward layer converts the bias vector into a representation vector; the convolutional adaptive module converts the representation vector into a feature weighted vector; The fully connected layer outputs a transformation vector based on the feature weighted vector output by the last encoder layer, and the predicted text is obtained through the transformation vector; The TBA module generates importance weights and bias items according to the speech frames and outputs the bias vector to the feedforward layer, specifically: The speech frame x is input to the TBA module, and the linear layer f α Generate the importance weight f of the corresponding speech frame x a (x), expressed as: f a (x)=[a1,a2,…,a n ]; Among them, α i is the weight of the i-th speech frame, i takes values of 1, 2, …n; The importance weight f of speech frame x α (x) is weighted with the bias vector b to generate the bias term bias, which is expressed as: in, represents the Hadamard product; The bias term bias is residually connected to the speech frame x to obtain the context-dependent bias vector x b , expressed as: x b =x+bias。 2. The non-native Chinese speech recognition method based on offset and convolution adaptation according to claim 1 is characterized in that: The method of selecting a pre-trained model to construct a non-native Chinese speech recognition model comprises the following steps: Select the pre-trained speech model wav2vec2.0 as the pre-trained model; Insert the TBA module between the self-attention sublayer and the feedforward layer of the pre-trained speech model wav2vec2.0, and add a convolutional adaptation module and a fully connected layer after the feedforward layer; The TBA module consists of a trainable bias vector b and a linear layer f α ; The convolution adaptive module includes layer normalization, three one-dimensional convolutions and a compression excitation module.
3. The non-native Chinese speech recognition method based on offset and convolution adaptation according to claim 2 is characterized in that: The self-attention sublayer converts the input signal into a speech frame. Specifically, the output of the input signal after passing through the self-attention sublayer is added to the input signal through a residual connection, and then layer normalization is performed to obtain a speech frame x=token[t1,t2,…,t i ,…,t n ], t i represents the i-th speech frame segment, and n represents the number of speech frame segments.
4. The non-native Chinese speech recognition method based on offset and convolution adaptation according to claim 2 is characterized in that: The convolutional adaptive module converts the representation vector into a feature weighted vector, comprising the following steps: After the representation vector is aligned, it enters the convolutional adaptive module and is normalized to obtain Will Input into three one-dimensional convolutional layers in sequence to obtain the convolution representation vector c i , the calculation formula is as follows: Among them, conv1, Conv2, and Conv3 represent the operations of the first to third one-dimensional convolutional layers respectively; Convolution represents the vector c i Input to the compression excitation module to obtain the feature weighted vector u c .
5. The non-native Chinese speech recognition method based on offset and convolution adaptation according to claim 4 is characterized in that: The compression excitation module obtains the feature weighted vector u c The process includes the following steps: The convolution of the input represents the vector c i Perform global average pooling to obtain a global pooling vector z containing all speech frames and all channels c , expressed as: Where n is the number of speech frames, J is the channel dimension of the speech frame, and c ij c i The eigenvalue on the jth channel; For the global pooling vector z c Perform the transformation of two fully connected layers to obtain a global transformation vector s c , expressed as: s c =σ(W2δ(W1z c )); Among them, σ and δ represent the Sigmoid activation function and the ReLU activation function respectively, and W1 and W2 represent the weight matrices of the two fully connected layers respectively; For the convolution representation vector c i and the global transformation vector s c Multiply to get the feature weight vector u c .
6. The non-native Chinese speech recognition method based on offset and convolution adaptation according to claim 4 is characterized in that: The selective training method is used to train the non-native Chinese speech recognition model, specifically: only the parameters of the TBA module and the convolutional adaptive layer are trained, and the parameters of the pre-trained model are frozen; The optimization function used in training is expressed as: <h2 style=";text-align:left;direction:ltr">θ<h2 style=";text-align:left;direction:ltr"> * <h2 style=";text-align:left;direction:ltr"> =argmin<h2 style=";text-align:left;direction:ltr"> θ <h2 style=";text-align:left;direction:ltr"> L<h2 style=";text-align:left;direction:ltr"> CTC <h2 style=";text-align:left;direction:ltr"> ; Among them, θ is the parameter of TBA module and convolutional adaptation layer, θ * is the optimal parameter after training, L CTC is the CTC loss function, N is the size of the training dataset, g i is the conversion vector output by the i-th speech sample after passing through the model, y i is the label sequence of the i-th speech sample, P(y i |g i ) is g i The derived conditional probability, α is the learning rate.
7. The non-native Chinese speech recognition method based on offset and convolution adaptation according to claim 1 is characterized in that: The method of realizing non-native Chinese speech recognition by using the trained non-native Chinese speech recognition model comprises the following steps: Input the original speech signal to be recognized into the trained non-native Chinese speech recognition model to obtain the conversion vector g; According to the pre-set word and vector conversion table, the probability of the candidate character sequence is derived from the conversion vector g, and the character sequence with the largest probability is selected as the predicted text, which is expressed as: Among them, y represents the predicted text, and P(y|g) is the conditional probability derived from the transformation vector g.
8. A non-native Chinese speech recognition system based on offset and convolution adaptation, characterized in that: include: Model building module, select the pre-trained model to build a non-native Chinese speech recognition model; Model training module, which uses selective training to train non-native Chinese speech recognition models; Speech recognition module, which uses the trained non-native Chinese speech recognition model to realize non-native Chinese speech recognition; The non-native Chinese speech recognition model includes a preprocessing layer, a transformer encoder and a fully connected layer connected in sequence; The preprocessing layer converts the input raw speech signal into a feature vector; The transformer encoder includes multiple encoder layers connected in sequence, each encoder layer includes a self-attention sublayer, a TBA module, a feedforward layer and a convolutional adaptive module connected in sequence; the self-attention sublayer converts the input signal into a speech frame, the input signal of the self-attention sublayer of the first encoder layer is the feature vector output by the preprocessing layer, and the input signal of the self-attention sublayer of other encoder layers is the feature weighted vector output by the previous encoder layer; the TBA module generates importance weights and bias items according to the speech frame, and outputs the bias vector to the feedforward layer; the feedforward layer converts the bias vector into a representation vector; the convolutional adaptive module converts the representation vector into a feature weighted vector; The fully connected layer outputs a transformation vector based on the feature weighted vector output by the last encoder layer, and the predicted text is obtained through the transformation vector; The TBA module generates importance weights and bias items according to the speech frames and outputs the bias vector to the feedforward layer, specifically: The speech frame x is input to the TBA module, and the linear layer f α Generate the importance weight f of the corresponding speech frame x a (x), expressed as: f a (x)=[a1,a2,…,a n ]; Among them, α i is the weight of the i-th speech frame, i takes values of 1, 2, …n; The importance weight f of speech frame x α (x) is weighted with the bias vector b to generate the bias term bias, which is expressed as: in, represents the Hadamard product; The bias term bias is residually connected to the speech frame x to obtain the context-dependent bias vector x b , expressed as: x b =x+bias。