Improved WeNet-based dysarthria speech recognition method

Abstract

The invention discloses an improved WeNet-based dysarthria speech recognition method, which is characterized by comprising the following steps of: 1, performing speech recognition on dysarthria; the method comprises the following steps: step 1, constructing an end-to-end WeNet framework, marking and segmenting dysarthria voice, extracting features through two-dimensional convolution and down-sampling, standardizing the features, adding position codes and time sequence information by an encoder, and introducing a local attention enhancement mechanism; 2, dynamically adjusting the weight of a loss function and predicting phonemes by adopting a dynamic loss function optimization and phoneme level supervision strategy; step 3, pre-training the optimized model by adopting a mandarin data set, then performing fine tuning on dysarthria voice, and performing field adaptation in combination with KL divergence loss; and 4, inputting corresponding voice data into the model, performing voice recognition, transcribing the voice data into a text, and outputting the text. The problem that in the prior art, in the aspect of dysarthria speech processing, the accuracy is still lowered is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech recognition, and in particular relates to a method for recognizing dysarthric speech based on an improved WeNet. Background Art

[0002] Dysarthria is a motor speech disorder caused by damage to the nervous system. It is a dysfunction of the neuromuscular system that controls the vocal muscles, resulting in changes in muscle strength, speed, tension, stability, or coordination during speech production. Cortical lesions can lead to a series of neuropathological characteristics in dysarthria, the severity of which depends on the location and severity of the neuropathy. Patients often exhibit symptoms such as slurred pronunciation, slow speech rate, abnormal intonation, and unstable volume. These abnormal speech characteristics make it difficult for patients to speak clearly in daily communication, making it extremely difficult to communicate with others. This not only causes tremendous physical and mental stress, but also hinders their ability to better integrate into society. Therefore, it is of great practical significance to study effective strategies for the rehabilitation of speech disorders, improve their communication skills, and help them re-participate effectively in society.

[0003] Because dysarthric patients easily tire when speaking, their facial expressions and intonation are limited, and the recording process is difficult to control, collecting speech data is extremely difficult. Most domestic research uses small, self-built datasets that fail to cover the full range of dysarthric speech sounds. This makes deep models prone to overfitting and difficult to adapt to unseen speakers.

[0004] In China, the Chumenwenwen voice team, in collaboration with the Speech Laboratory of Northwestern Polytechnical University, has developed WeNet, an open-source speech recognition toolkit for industrial applications. Drawing on other outstanding open-source speech projects, WeNet's core goal is to provide a high-performance, easy-to-deploy, industrial-grade solution for speech recognition. Focusing on speech recognition tasks, WeNet offers an exceptional end-to-end solution for common speech recognition scenarios. This clear design goal ensures that WeNet maintains simplicity and ease of use while achieving excellent speech recognition accuracy, enabling direct implementation in industrial scenarios. However, WeNet still faces a decline in accuracy in processing dysarthric speech, making it a key area for optimization. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for recognizing dysarthric speech based on an improved WeNet, which solves the problem of decreased accuracy in the prior art in processing dysarthric speech.

[0006] The technical solution adopted by the present invention is a method for recognizing dysarthric speech based on an improved WeNet, comprising the following steps: Step 1: Build an end-to-end WeNet framework to annotate and segment the dysarthric speech. Extract and normalize features through two-dimensional convolution and downsampling. The encoder adds positional encoding and temporal information, and introduces a local enhanced attention mechanism to fuse local windows with global self-attention. Step 2: Dynamic loss function optimization and phoneme-level supervision are used to dynamically adjust the loss function weights and additionally predict phonemes. Step 3: Pre-train the optimized model using the Mandarin Chinese dataset, then fine-tune it with dysarthric speech and perform domain adaptation in combination with the KL divergence loss. Step 4: Input the corresponding voice data into the model, and output it after voice recognition and transcription into text.

[0007] The present invention is also characterized in that The local enhanced attention mechanism fusion includes the following steps: After layer normalization, the input features are sent to the local window attention module and the global attention enhancement module respectively. The local window attention module calculates the attention distribution in the adjacent window of the current frame. The calculation expression is shown in the following formula: ; Where K w Is the key of the local window, V w is the value of the local window, d k is the feature vector dimension; The global attention enhancement module calculates the global representation of the entire sequence through global pooling. The calculation expression is shown as follows: ; Where K G , V G Represents the global pooled keys and values: , ; The features extracted by the two modules are subjected to Dropout processing respectively; Dynamically adjust the proportion of local and global attention through the learnable parameter λ, and perform weighted fusion of the processed features; Finally, the fusion result is residually connected with the original input, and the output features are passed to the convolution module and the feedforward module.

[0008] Step 1 also includes the decoder inputting the embedded representation of the target text, aligning the speech features output by the encoder through a multi-layer attention module, and generating character probability distribution through a fully connected layer and a softmax activation function.

[0009] In step 2, the weight of the loss function is dynamically adjusted to calculate the adaptive weight α based on the input speech features. The calculation expression of the adaptive weight is shown in the following formula: ; Where X represents the characteristics of the input speech; W and b are learnable parameters; It is the Sigmoid activation function, which makes α take the value between (0,1); During the training process, the model dynamically adjusts the balance between CTC and Attention based on the input speech features. The training adopts a staged optimization method. In the first stage, the CTC loss function is trained separately, and in the second stage, the CTC-related parameters are fixed to train the Attention branch.

[0010] In step 2, the additional predicted phonemes use CTC to predict the phoneme sequence. The loss function expression of CTC supervision is shown as follows: ; Where, L phoneme is the phoneme-level CTC loss, To control the influence of phoneme prediction.

[0011] In step 3, the Mandarin Chinese dataset is AISHELL-1. The optimized WeNet model is pre-trained based on the dataset AISHELL-1. The loss function in pre-training is shown as follows: ; Where, α is 0.3~0.5; X p is ordinary voice data; Y p The target text for normal speech.

[0012] In step 3, fine-tuning the dysarthric speech includes: Load the pre-trained WeNet model parameters and freeze some feature layers of the encoder; Fine-tuning training is performed based on the dysarthric speech dataset. The loss function of fine-tuning training is shown as follows: ; Where, X d is the dysarthric speech data, Y d is the target text.

[0013] In order to alleviate the distribution difference between the target speech features and the pre-trained speech features, KL divergence loss is used for domain adaptation. The expression of KL divergence loss is shown as follows: ; ; Where, P(X P ) is the speech feature distribution of pre-training data; P(X d ) is the characteristic distribution of dysarthric speech; λ is the weight parameter of KL loss.

[0014] The beneficial effects of the present invention are: The present invention designs a local enhanced attention mechanism, which combines local window attention with global attention enhancement to adaptively fuse local and global features to improve the model's ability to model local abnormal pronunciation patterns; secondly, to address the problems of fixed weights and blurred phoneme boundaries in the joint training of unsupervised sequence learning algorithms and attention mechanisms, a dynamic loss function weight adjustment and phoneme-level supervision fusion strategy is proposed, which effectively optimizes the frame-level alignment and phoneme modeling process; finally, to alleviate the scarcity of dysarthric speech data, a model adaptation method based on transfer learning is designed. Through three-stage training of pre-training, fine-tuning and domain adaptation, the generalization ability of the model in dysarthric speech recognition tasks is improved, thereby improving the accuracy and stability of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 1 is a schematic diagram of an algorithm for the dysarthric speech recognition method based on the improved WeNet of the present invention; Figure 2 This is a training flow chart for the fusion of the balanced loss function and phoneme supervision in the present invention; Figure 3 This is a diagram of the fine-tuning architecture of transfer learning for dysarthric speech recognition in the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] Example 1 The present invention is based on an improved WeNet method for recognizing dysarthric speech. Figure 1 As shown, the following steps are included.

[0018] Step 1: Build an end-to-end WeNet framework to perform text annotation and speech segmentation on dysarthric speech. Speech signals undergo two-dimensional convolution and downsampling to extract local features. Global mean-variance normalization is used to standardize the data to improve training stability. Positional encoding in the encoder adds temporal information to the speech sequence. Given that dysarthric speech is often unclear, a local enhanced attention mechanism is introduced to fuse local windows with global self-attention. Step 2: Dynamic loss function optimization and phoneme-level supervision are used to dynamically adjust the loss function weights to enable the model to dynamically adapt to different speech characteristics. Phoneme-level supervision is used to additionally predict phonemes, thereby improving the recognition ability of dysarthric speech. Step 3: A speech adaptation strategy based on transfer learning was used to pre-train the optimized model using a Mandarin Chinese dataset, then fine-tune it with dysarthric speech and perform domain adaptation in combination with KL divergence loss. Through this three-stage strategy of pre-training, fine-tuning, and domain adaptation, the WeNet model not only inherits the general speech modeling capabilities learned from normal speech data, but also effectively adapts to the pronunciation characteristics of dysarthric speech through the fine-tuning phase. This transfer learning strategy not only alleviates the engineering challenges of small sample training but also enhances the ability to transfer target speech features from a modeling perspective.

[0019] Step 4: Input the corresponding voice data into the model, and output it after voice recognition and transcription into text.

[0020] Furthermore, step 1 also includes the decoder inputting the embedded representation of the target text, aligning the speech features output by the encoder through a multi-layer attention module, and generating character probability distribution through a fully connected layer and a softmax activation function.

[0021] Example 2 Based on the above-mentioned embodiment 1, step 1 of the present invention is specifically designed to enhance the model's ability to model local abnormal pronunciations and improve the robustness to phenomena such as phoneme substitution and abnormal speaking speed. It consists of three parts: local window attention (LWA), global refinement attention (GRA) and dynamic fusion strategy.

[0022] Specifically, the local enhanced attention mechanism fusion includes the following steps: After layer normalization, the input features are fed into the local window attention module and the global attention enhancement module, respectively. In the standard self-attention calculation, each frame can focus on the entire input sequence, but this approach cannot model local abnormal phonemes. LWA only calculates the attention distribution within the neighboring window W of the current frame t to avoid attention diffusion. The mathematical definition of the local window size is W (W=5, indicating that the attention calculation range is limited to the current frame and the 5 frames before and after it). The calculation expression is shown in the following formula (1): (1); Where K w Is the key of the local window, V w is the value of the local window, d k is the feature vector dimension; K W ,V W Only the Key and Value in the local window are included. The advantage of this is that the computational complexity can be reduced from O(N 2 ) is reduced to O(NW), which speeds up the inference process. This allows the model to focus more on local speech features, thereby improving the recognition accuracy of dysarthric speech.

[0023] Using only local attention may cause the model to be unable to learn global context information, affecting the ability to recognize long sentences. Therefore, the global attention enhancement module calculates the global representation of the entire sequence through global pooling. The calculation expression is shown in the following formula (2): (2); Where K G , V G Represents the global pooled keys and values: , ; The features extracted from the two modules were subjected to Dropout processing respectively. Dropout refers to a regularization operation performed on the extracted features during model training: by randomly inactivating some feature neurons (i.e., randomly setting some feature values ​​to 0 temporarily), the model's excessive dependence on local features is reduced, overfitting is avoided, and the model's generalization ability for dysarthric speech features is enhanced.

[0024] The ratio of local and global attention is dynamically adjusted through the learnable parameter λ, and the processed features are weighted and fused. The expression is shown in the following formula (3): (3); λ is a learnable parameter that dynamically controls the balance between local and global attention. This ensures that λ favors local attention in dysarthric speech and global attention in normal speech. Finally, the fused result is residually connected to the original input, and the output features are passed to the convolutional module and the feedforward module.

[0025] Example 3 Based on the above embodiment 2, step 2 of the present invention is mainly to improve the loss function structure and introduce phoneme-level supervision to improve the alignment accuracy of phonemes. Specifically, in order to improve the adaptability of the model, the loss function weight is dynamically adjusted in step 2 to calculate the adaptive weight α based on the input speech features. The calculation expression of the adaptive weight α is shown in the following formula (4): (4); Where X represents the characteristics of the input speech; W and b are learnable parameters; It is the Sigmoid activation function, which makes α take the value between (0,1); During training, the model dynamically adjusts the balance between CTC and Attention based on the input speech features. Typically, the CTC ratio is higher for dysarthric speech. The CTC branch (Connectionist Temporal Classification) is primarily responsible for frame-level alignment of speech and text sequences. By ignoring the length difference between the input and output sequences, it solves the "many-to-many" temporal alignment problem in speech recognition. It is the foundational module for mapping frame-level features to text in WeNet. The Attention branch, based on the attention mechanism, calculates the association weights between speech features and target text to model the semantics of long sequences, improving the recognition of complete sentences. It is a key module for handling sequence dependencies in WeNet.

[0026] Furthermore, in order to prevent CTC from affecting Attention training, the present invention adopts phased optimization. In the first phase, the CTC loss function is trained separately, and the CTC is first trained to train the frame-level alignment information. During the training process, the Attention decoder does not participate in the optimization to avoid interference with the CTC. In the second stage, the CTC-related parameters are fixed to train the Attention branch. On the basis of CTC training, the Attention branch is added.

[0027] Example 4 Based on the above-mentioned embodiment 3, in step 2 of the present invention, in order to improve the interpretability of dysarthric speech, a phoneme-level supervision task is added, CTC is used to predict the phoneme sequence, and the phoneme boundary detection capability is improved. The loss function expression of CTC supervision is shown in the following formula (5): (5); Where, L phoneme It is the phoneme-level CTC loss, which is used to assist in learning phoneme boundaries; To control the influence of phoneme prediction.

[0028] like Figure 2The figure shows the introduction of dynamic alpha calculation and a two-stage training strategy. First, dynamic alpha calculation calculates the optimal alpha value based on input speech features, ensuring a balance between CTC and Attention, enabling the model to adapt to different speech inputs. Second, two-stage training: In the first stage, CTC+LEA is trained to enable the model to learn frame-level temporal alignment information, improving pronunciation recognition. In the second stage, CTC is fixed and Attention+LEA is trained for text decoding, improving semantic understanding. In phoneme-level supervision, CTC additionally predicts phonemes, improving recognition of dysarthric speech and reducing errors caused by ambiguous pronunciation. After training, the final output is generated by a beam search decoder, combining the CTC and Attention predictions.

[0029] Example 5 Based on the above-mentioned Example 4, this embodiment improves the model's adaptability to articulation-delayed speech data and enhances its generalization ability to articulation-delayed speech through a transfer learning strategy. In order to learn common speech features, the present invention pre-trains the optimized WeNet model on a large-scale Mandarin dataset.

[0030] The Mandarin dataset selected is AISHELL-1, which contains rich speech samples and annotation information. The optimized WeNet model is pre-trained based on the AISHELL-1 dataset to learn common speech features. During the pre-training process, the CTC+Attention training strategy is used to enable the model to grasp the frame-level alignment information of the speech. The pre-trained model parameters are saved for subsequent migration to the dysarthric speech dataset. The loss function in the pre-training is shown in the following formula (6): (6); Where, α is 0.3~0.5; X p is ordinary voice data; Y p The target text for normal speech.

[0031] After training, the model has a good foundation for speech recognition, but it has never seen dysarthric speech. Therefore, to adapt to the particularity of dysarthric speech, the present invention migrates the pre-trained model to the dysarthric speech dataset and performs fine-tuning. The specific steps are as follows: First, load the pre-trained WeNet model parameters. To preserve the common features learned by the pre-trained model, freeze some feature layers of the encoder; only train the decoder and some unfrozen encoder layers. During the fine-tuning process, the model improves its adaptability and recognition performance for dysarthric speech by learning the complex features of dysarthric speech. The loss function of fine-tuning training is shown in the following formula (7): (7); Where, X d is the dysarthric speech data, Y d is the target text.

[0032] Example 6 This embodiment is based on the above embodiment 5. Since the phoneme distribution of normal speech and dysarthric speech is different, direct fine-tuning may affect the stability of the model. In order to alleviate the distribution difference between the target speech features and the pre-trained speech features, the present invention adopts KL divergence loss for domain adaptation. The expression of KL divergence loss is shown in the following equations (8) and (9): (8); (9); Where, P(X P ) is the speech feature distribution of pre-training data; P(X d ) is the characteristic distribution of dysarthric speech; λ is the weight parameter of KL loss.

[0033] like Figure 3 As shown in the figure, the WeNet speech recognition model optimization scheme based on transfer learning adopted by the present invention is demonstrated; WeNet is trained on the Mandarin dataset Aishell-1 to acquire the ability to model ordinary speech; then it is fine-tuned on the dysarthria dataset CDSD to adapt the model to the characteristics of dysarthric speech; the feature distributions of Aishell-1 and CDSD are different, and the KL loss is calculated for adjustment; after the final training is completed, the model is suitable for dysarthric speech recognition, improving the accuracy and stability of speech recognition.

[0034] In order to verify the effect of the method of the present invention, experiments were conducted to test the three-point design proposed in the present invention. The test results are shown in the following table: Table 1. Character error rate test results of model improvement

[0035] As can be seen in Table 1, the introduction of each improvement method has brought about performance improvements. The local enhanced attention mechanism reduced the CER by 4.73 percentage points, the loss function optimization reduced the CER by 2.99 percentage points, and transfer learning had the most significant effect, reducing the CER by 7.32 percentage points. This shows that each improvement method has made a significant contribution to the improvement of model performance.

[0036] In addition, the recognition model of the present invention was tested and compared with the existing model. The comparison results are shown in Table 1 below: Table 1 Model comparison character error rate test results

[0037] The comparative experimental results in Table 1 demonstrate that the optimized model proposed in this paper achieves significantly better performance than mainstream speech recognition frameworks in the dysarthric speech recognition task. The proposed model, combined with structural optimization and transfer learning, achieves a 10.56% reduction in CER compared to the WeNet baseline, approximately 11.35% reduction compared to DeepSpeech2, 11% reduction compared to ESPnet, 11.98% reduction compared to Whisper-small, 10.28% reduction compared to Whisper-medium, approximately 9.53% reduction compared to Whisper-Perceiver-Prompt, and approximately 11.14% reduction compared to WeNet (CDSD), demonstrating its effectiveness and advantages in speech recognition for dysarthric individuals.

[0038] Experimental results show that the proposed improved model significantly improves all metrics compared to the baseline model when handling dysarthric speech recognition, demonstrating greater robustness and adaptability. The introduction of a local enhanced attention mechanism, loss function optimization, and transfer learning strategies significantly enhances the model's recognition performance. The improved WDSR model achieves an overall decrease of approximately 10.56 percentage points in character error rate (CER) compared to the baseline model, demonstrating that the three designed optimization strategies effectively enhance the model's modeling and generalization capabilities for dysarthric speech.

[0039] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0040] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. The method for recognizing dysarthric speech based on the improved WeNet is characterized by: The following steps are involved: Step 1: Build an end-to-end WeNet framework to annotate and segment the dysarthric speech. Extract and normalize features through two-dimensional convolution and downsampling. The encoder adds positional encoding and temporal information, and introduces a local enhanced attention mechanism to fuse local windows with global self-attention. Step 2: Dynamic loss function optimization and phoneme-level supervision are used to dynamically adjust the loss function weights and additionally predict phonemes. Step 3: Pre-train the optimized model using the Mandarin Chinese dataset, then fine-tune it with dysarthric speech and perform domain adaptation in combination with the KL divergence loss. Step 4: Input the corresponding voice data into the model, and output it after voice recognition and transcription into text.

2. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: The local enhanced attention mechanism fusion The following steps are involved: After layer normalization, the input features are sent to the local window attention module and the global attention enhancement module respectively. The local window attention module calculates the attention distribution in the adjacent window of the current frame. The calculation expression is shown in the following formula: ； Where K w Is the key of the local window, V w is the value of the local window, d k is the feature vector dimension; The global attention enhancement module calculates the global representation of the entire sequence through global pooling. The calculation expression is shown as follows: ； Where K G , V G Represents the global pooled keys and values: , ; The features extracted by the two modules are subjected to Dropout processing respectively; The weight of local and global attention is dynamically adjusted through the learnable parameter λ to perform weighted fusion of the processed features. Finally, the fusion result is residually connected with the original input, and the output features are passed to the convolution module and the feedforward module.

3. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: The step 1 also includes the decoder inputting the embedded representation of the target text, aligning the speech features output by the encoder through a multi-layer attention module, and generating a character probability distribution through a fully connected layer and a softmax activation function.

4. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: In step 2, the weight of the loss function is dynamically adjusted to calculate the adaptive weight α based on the input speech features. The calculation expression of the adaptive weight is shown in the following formula: ； Where X represents the characteristics of the input speech; W and b are learnable parameters; It is the Sigmoid activation function, which makes α take the value between (0,1); During the training process, the model dynamically adjusts the balance between CTC and Attention based on the input speech features. The training adopts a staged optimization method. In the first stage, the CTC loss function is trained separately, and in the second stage, the CTC-related parameters are fixed to train the Attention branch.

5. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: In step 2, the additional predicted phonemes use CTC to predict the phoneme sequence. The loss function expression of CTC supervision is shown as follows: ； Where, L phoneme is the phoneme-level CTC loss, To control the influence of phoneme prediction.

6. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: The Mandarin Chinese dataset in step 3 is AISHELL-1. The optimized WeNet model is pre-trained based on the AISHELL-1 dataset. The loss function in the pre-training is shown as follows: ； Where, α is 0.3~0.5; X p is ordinary voice data; Y p The target text for normal speech.

7. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: The step 3 of fine-tuning the dysarthric speech specifically includes: Load the pre-trained WeNet model parameters and freeze some feature layers of the encoder; Fine-tuning training is performed based on the dysarthric speech dataset. The loss function of fine-tuning training is shown as follows: ； Where, X d is the dysarthric speech data, Y d is the target text.

8. The method for recognizing dysarthric speech based on improved WeNet according to claim 1, characterized in that: In order to alleviate the distribution difference between the target speech features and the pre-trained speech features, KL divergence loss is used for domain adaptation. The expression of the KL divergence loss is shown in the following formula: ； ； Where, P(X P ) is the speech feature distribution of pre-training data; P(X d ) is the characteristic distribution of dysarthric speech; λ is the weight parameter of KL loss.

Images