Protype-based dysarthria speech recognition method

By using the combination of HuBERT model and CTC and SCL loss functions in dysarthritis speech recognition, combined with the few-shot learning prototype network, it quickly adapts to the unique voice mode of dysarthritis speakers, and solves the performance degradation and data in the existing technology. The high demand problem has achieved efficient dysarthritis speech recognition.

CN118942445APending Publication Date: 2024-11-12NANKAI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410938705.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-14
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing dysarthritis speech recognition technology has deteriorated performance when adapting to unseen dysarthritis speakers, and personalized fine-tuning methods require a lot of data and high training costs, and requires continuous optimization as disease severity changes.

Method used

Using the prototype-based dysarthritis speech recognition method, fine-tuning is used to use the pre-trained HuBERT model, and combined with CTC and SCL loss functions, the prototype network can be quickly adapted to the unique speech mode of each speaker through the few-shot learning prototype network, and constructed each word The prototype is identified.

Benefits of technology

It realizes rapid adaptation and efficient recognition of dysarthria speakers without dysarthria, reduces training costs, and significantly improves the recognition performance of severe dysarthria speech.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942445A_ABST
    Figure CN118942445A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of speech recognition, and particularly relates to a dysarthria speech recognition method based on a prototype. The method comprises the following steps: S1, finely adjusting a HuBERT for feature extraction, performing fine adjustment on a pre-trained self-supervised learning model HuBERT on a dysarthria speech recognition task, using CTC loss and supervised contrast learning loss in a fine adjustment stage to effectively guide training of the model, and using the finely-adjusted HuBERT as a feature extractor; s2, constructing a prototype of each word; and S3, prototype-based classification. The invention provides a quick and effective method for improving the speech recognition of the unseen dysarthria speaker. According to the method, CTC loss and SCL loss are combined to train the DSR model, and the performance is improved by learning better feature representation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of speech recognition, and more specifically, relates to a prototype-based dysarthric speech recognition method. Background Art

[0002] Dysarthria is a speech disorder caused by a variety of factors, such as neuropathy, paralysis of muscles that affect speech, decreased muscle contraction, or uncoordinated movements, and is often associated with diseases such as cerebral palsy, Parkinson's disease, and head trauma. For people with these diseases, the ability to use keyboards or touch screens is limited, making voice the most convenient way for them to interact with devices such as smartphones and smart home devices. However, changes in breathing, resonance, pronunciation, and prosody caused by dysarthria significantly reduce the performance of speech recognition systems trained on non-dysarthric speech. Dysarthric speech recognition (DSR) technology attempts to address this issue and enable seamless interaction between dysarthric speakers and digital devices. Although some studies focus on developing speaker-independent (SI) DSR models, due to differences in the etiology, age, gender, speaking style, and severity of dysarthria, the speech patterns of each dysarthric speaker are different, and when the SIDSR model is applied to unseen dysarthric speakers, there will be a significant performance drop.

[0003] Therefore, several studies have explored using data from target dysarthric speakers to fine-tune DSR models to achieve personalized DSR. For example, Shor et al. achieved this by fine-tuning a traditional speech recognition model using data from target dysarthric speakers, selectively fine-tuning only a subset of network layers to avoid overfitting. Takashima et al. introduced a two-stage fine-tuning approach, initially using data from multiple dysarthric speakers to develop a SIDSR model that captures general dysarthric speech patterns, which is then further fine-tuned using target dysarthric speaker data. Recently, Shahamiri et al. designed a new DSR system that can learn to recognize word shapes spoken by dysarthric speakers and map them to words. The system achieved the best performance by adaptively using target speaker data. However, these personalized fine-tuning methods require a large amount of speech data from the target speakers and incur significant training costs. In addition, changes in the severity of dysarthria over time may affect the long-term effectiveness of personalized DSR models, requiring continuous data collection and model optimization. This not only increases the training cost but also poses a huge challenge for speakers with dysarthria. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a prototype-based dysarthric speech recognition (PB-DSR) method, which utilizes the unique pronunciation error characteristics of dysarthric speakers. Unlike traditional methods, which require extensive model fine-tuning for each new speaker, the method of the present invention only requires a minimal data set to effectively adapt to a single speech pattern. Pronunciation errors, such as phoneme deletion, substitution, insertion, and distortion, are consistent in the same speaker. Previous studies have attempted to solve these errors by creating an adaptive pronunciation dictionary for each speaker through analysis of the frequency of pronunciation errors. However, these solutions are limited to the most common errors and do not adequately address the problem of misidentifying severe breathing sounds and background noise as phonemes. In contrast, the method of the present invention targets word-level pronunciation errors, treats the speech of each word as a unique category, and does similar matching rather than exact matching. Inspired by the principle of few-shot learning prototype networks, the PB-DSR method uses a feature extractor to create a prototype of each word from only a small number of samples, so that it can quickly adapt to the unique speech pattern of each speaker. The present invention utilizes a pre-trained self-supervised learning model, Hidden-Unit Bidirectional Encoder Representations from Transformers (HuBERT), to extract speech features. This model is well-known for its general speech recognition capabilities.

[0005] To further improve the adaptability to dysarthric speech, HuBERT is fine-tuned using a specialized dysarthric speech dataset. In addition, the present invention integrates supervised contrastive learning (SCL) to enhance feature extraction. SCL is a strategy that uses contrastive learning to optimize feature representation by minimizing intra-class distance while maximizing inter-class separation, a technique that has been proven to be effective in various speech tasks.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The prototype-based method for dysarthric speech recognition comprises the following steps:

[0008] S1. Fine-tune HuBERT for feature extraction. Fine-tune the pre-trained self-supervised learning model HuBERT on the DSR task. In the fine-tuning stage, the Connectionist Temporal Classification (CTC) loss and SCL loss are used to effectively guide the training of the model. The fine-tuned HuBERT is used as a feature extractor.

[0009] S2, build a prototype for each word;

[0010] S3. Prototype-based classification.

[0011] The technical solution is further optimized, and the step S1 specifically includes selecting the CTC loss function as the main loss function;

[0012] In addition, the SIDSR model performs poorly in recognizing speech from speakers with more severe dysarthria or speech from unseen dysarthria speakers, which may be attributed to the challenge of the SIDSR model in distinguishing the speech of different words of these speakers. Therefore, the CTC loss is integrated with the SCL loss as the total loss function, with the aim of enhancing the model's ability to learn improved feature representations by simultaneously increasing the inter-class distance and reducing the intra-class distance;

[0013] In SCL, in a training batch, samples that share the same label are considered positive samples of each other, while samples with different labels are considered negative samples. Therefore, the SCL loss, denoted as L SCL , the formula is as follows:

[0014]

[0015] Where i∈I={1,…,N} represents the index of the speech, A(i) represents all indices except i, and x i represents the features extracted by HuBERT, P(i) represents all the indices of the positive samples of sample i, the features of the same words as sample i are positive samples, and τ is the temperature hyperparameter;

[0016] The total loss is denoted as L Total , the CTC loss L CTC With SCL loss L SCL The combination is as follows:

[0017] LTotal=LCTC+LSCL. (2)

[0018] The technical solution is further optimized, and step S2 specifically includes classifying the speech of different words spoken by an unseen dysarthric speaker into different categories, and using a limited amount of speech data provided by an unseen dysarthric speaker as a support set, and using a fine-tuned HuBERT to extract speech features from the support set; then, the embodiment averages the features of the same word to construct a prototype of each word; these prototypes serve as representations of each word.

[0019] The technical solution is further optimized, and step S3 specifically includes: using the fine-tuned HuBERT to extract the features of the test speech; it should be emphasized that the test speech and the speech in the support set are from the same unseen dysarthria speaker, and then the distance between the test speech features and each prototype is calculated, and finally, the word represented by the closest prototype is selected as the recognition result.

[0020] Different from the prior art, the above technical solution has the following beneficial effects:

[0021] The present invention proposes a prototype-based dysarthric speech recognition method, which provides a fast and effective method to improve speech recognition of unseen dysarthric speakers.

[0022] The present invention combines CTC loss with SCL loss to train the DSR model and improves the performance by learning better feature representation.

[0023] The present invention has demonstrated the effectiveness of the proposed method on the UASpeech dataset, a widely studied dysarthric speech recognition dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a prototype-based dysarthric speech recognition method;

[0025] Figure 2 A visual diagram of the distribution of speech features. DETAILED DESCRIPTION

[0026] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0027] See also Figure 1 The figure shows a schematic diagram of a prototype-based dysarthric speech recognition method. A preferred embodiment of the present invention is a prototype-based dysarthric speech recognition method (PB-DSR for short) comprising three stages: fine-tuning HuBERT for feature extraction, constructing a prototype of each word, and prototype-based classification.

[0028] Stage 1: Fine-tuning HuBERT for feature extraction

[0029] This embodiment fine-tunes the pre-trained self-supervised learning model HuBERT on the DSR task, and uses CTC loss and SCL loss to effectively guide the training of the model during the fine-tuning stage. The fine-tuned HuBERT is used as a feature extractor.

[0030] Combining CTC loss with SCL loss

[0031] CTC loss is widely used in speech recognition tasks because it can accommodate data with potential mismatches between text and speech. Considering the presence of pauses and heavy breathing sounds in dysarthric speech, this embodiment selects the CTC loss function as the main loss function.

[0032] In addition, the SIDSR model performs poorly in recognizing speech from speakers with more severe dysarthria or speech from unseen dysarthria speakers, which may be attributed to the challenge of the SIDSR model in distinguishing the speech of different words from these speakers. Therefore, this embodiment integrates the CTC loss with the SCL loss as the total loss function, with the aim of enhancing the model's ability to learn improved feature representations by simultaneously increasing the inter-class distance and reducing the intra-class distance.

[0033] In SCL, in a training batch, samples that share the same label are considered positive samples of each other, while samples with different labels are considered negative samples. Therefore, the SCL loss, denoted as L SCL , the formula is as follows:

[0034]

[0035] Where i∈I={1,…,N} represents the index of the speech, A(i) represents all indices except i, and xi represents the features extracted by HuBERT. P(i) represents all indices of positive samples of sample i. The features representing the same words as sample i are positive samples. τ is the temperature hyperparameter.

[0036] The total loss is denoted as L Total , the CTC loss L CTC With SCL loss L SCL The combination is as follows:

[0037] LTotal=LCTC+LSCL. (2)

[0038] Phase 2: Build a prototype for each word

[0039] The embodiment classifies the speech of different words spoken by unseen dysarthric speakers into different categories and uses a limited amount of speech data provided by unseen dysarthric speakers as a support set. The embodiment uses a fine-tuned HuBERT to extract speech features from the support set. Subsequently, the embodiment averages the features of the same word to construct a prototype for each word. These prototypes serve as representations of individual words.

[0040] Stage 3: Prototype-based classification

[0041] This embodiment uses a fine-tuned HuBERT to extract features of the test speech. It should be emphasized that the test speech and the speech in the support set are from the same unseen dysarthria speaker. Next, this embodiment calculates the distance between the test speech features and each prototype. Finally, this embodiment selects the word represented by the closest prototype as the recognition result.

[0042] experiment

[0043] 1. Dataset

[0044] Table 1: Intelligibility information of dysarthric speakers

[0045]

[0046] This embodiment evaluates the effectiveness of the proposed method using the UASpeech dataset, which includes speech data from 15 dysarthric speakers and 13 control speakers. Table 1 lists the intelligibility details of the 15 dysarthric speakers, where intelligibility represents the percentage of speech that the average listener understands. The dataset is divided into 3 blocks (blocks 1, 2, and 3), including a total of 455 isolated words. Each speaker reads 10 numbers, 26 radio letter words, 19 computer commands, and 100 common words three times. Each block contains a set of such speech. In addition, there are 300 uncommon words, each of which is read once by each speaker. These uncommon words are distributed in various blocks, each containing 100 uncommon words. The UASpeech data was recorded with an 8-microphone array, where microphone 1 is used for pitch synchronization and microphones 2 to 8 capture speech signals. Due to the data volume requirements for fine-tuning, the process uses data from 7 microphones. In order to speed up PB-DSR and evaluate its few-shot learning ability, the prototype of each word is constructed using only data from microphone 5.

[0047] The following is a description of the data partitioning:

[0048] • TRAIN: All speech from the control speakers and speech from Block 1 and Block 3 from the dysarthric speakers.

[0049] CTEST: Block 2 speech of all dysarthric speakers, excluding uncommon word speech.

[0050] 2. Experimental Setup

[0051] 2.1.DSR model settings

[0052] This example uses the UASpeech dataset to fine-tune HuBERT under the Fairseq open source framework1 to train the DSR model. The parameter size of the DSR model is 94.84M and is trained using a single NVIDIA GeForce RTX 3090. This example uses the pre-trained HuBERT Base2, which has been pre-trained on the Librispeech 960h dataset3. The DSR model is trained using the configuration file base_10h.yaml4, using 10- 5 The initial learning rate is 0, the warm-up steps are 32,000, and the batch size is 40. This example sets the parameter τ in Equation 1 to 0.07. The dictionary used includes 455 words from the UASpeech dataset, as well as the tokens blank, <s>、 <pad> 、< / pad> < / s> and <unk>If the loss does not decrease for 10 consecutive epochs, the training will stop. The evaluation metric is the Word Error Rate (WER), measured in percentage.

[0053]

[0054] The embodiment first uses the TRAIN partition to train a basic DSR model (denoted as V) to handle visible dysarthric speakers. In order to verify the effectiveness of the proposed PB-DSR method in enhancing the speech recognition performance of unseen dysarthric speakers, the embodiment iteratively designates each dysarthric speaker in the UASpeech dataset as "unseen". Subsequently, the data of the designated speaker is removed from the TRAIN partition, resulting in a modified partition, denoted as R-TRAIN. From this process, 15 different R-TRAINs are obtained, each corresponding to a dysarthric speaker in the dataset. The embodiment trains 15 models on R-TRAIN using only the CTC loss function; these models are collectively referred to as R. In addition, another set of 15 models are trained using the proposed combined loss function, which are denoted as R+. Subsequently, the prototype-based method is applied to the R and R+ models, resulting in the PB-DSR and PB-DSR+ configurations, respectively.

[0055] To further compare the PB-DSR method proposed by the embodiment with the traditional fine-tuning method, the embodiment fine-tunes each R model using data from the corresponding unseen speaker. These fine-tuned models are denoted as FT-R. After this, the embodiment implements the prototype-based method on FT-R, resulting in a FT-R+PB-DSR configuration. This step enables the embodiment to evaluate the efficacy of the prototype-based method in improving speech recognition performance on the fine-tuned model.

[0056] 2.2. Setup of prototype-based DSR

[0057] Due to the peak behavior in the CTC model, and the fact that the DSR model of this example is configured with words as the dictionary, it is observed that after the HuBERT features pass through the CTC layer, only the first frame is predicted as a word. Based on this observation, the performance of PB-DSR when building prototypes using the full HuBERT features and relying only on the first frame features of HuBERT is compared. It is found that using only the first frame can produce better results. Therefore, the first frame of the extracted HuBERT features is used to build the prototype of each word.

[0058] The IndexFlatL2 index from the Faiss library is used to calculate the Euclidean distance between the features of each test utterance and each word prototype.

[0059] Table 2: WER (%) under different settings. "SI" and "SD" denote speaker-independent and speaker-dependent models, respectively. "FT" denotes fine-tuning, and "+" denotes adding SCL loss.

[0060]

[0061] All experimental results are detailed in Table 2. Initially, the SIDSR model trained on the TRAIN partition was compared with other studies to verify the effectiveness of the model settings of this embodiment and the proposed loss function (Equation 2). Based on the understanding of the characteristics of dysarthric speech, Bhat et al. designed static and dynamic data enhancement techniques for dysarthric speech, which produced good speech recognition performance on the UASpeech dataset. Since the work of Bhat et al. is consistent with our data setting, that is, evaluating the model performance on the CTEST partition, it can be used as a direct comparison. Compared with the work of Bhat et al., the models of this embodiment (V and V+) have significant improvements in WER at all intelligibility levels, with p-values ​​<0.05, highlighting the enhancement effect of the method of this embodiment on dysarthric speech recognition.

[0062] Then, the DSR performance of R and PB-DSR on unseen dysarthric speakers was compared. It is clear that PB-DSR significantly enhances the DSR performance at each intelligibility level (p-value < 0.05). Specifically, it reduces the absolute value of WER by 3.98% (high), 11.17% (medium), 27.37% (low), and 19.86% (very low), respectively. Subsequently, when comparing R and R+, it can be observed that the increase in SCL loss enhances the speech recognition performance of the DSR model on unseen speakers (p-value < 0.05). The performance of PB-DSR is also further enhanced as the model learns better feature representations, as evidenced by the significant difference between PB-DSR- and PB-DSR+ (p-value < 0.05).

[0063] The comparison is extended to two recent works by Shahamiri et al. in 2023, which also emphasize adaptation of the target speaker, but adopt a fine-tuning approach. It is worth noting that even without model fine-tuning, the method of this embodiment achieves comparable performance to the recent work of Shahamiri et al. at very low intelligibility levels and outperforms them at other intelligibility levels. However, it is worth noting that Shahamiri et al. did not use HuBERT, limiting direct comparison under the same conditions. Therefore, PB-DSR was compared with FT-R, where FT-R involves fine-tuning. The performance of PB-DSR is relatively low. However, considering that PB-DSR does not require any fine-tuning, this result is considered reasonable. Then, FT-R+PB-DSR is compared with FT-R. It is observed that the performance of all intelligibility levels is improved except for the high intelligibility level (p-value < 0.05). This shows that combining the PB-DSR method with a fine-tuned model further improves the DSR performance of unseen dysarthric speakers, highlighting the adaptability and effectiveness of the PB-DSR approach.

[0064] See also Figure 2 The following is a visualization diagram of the distribution of speech features. (a) Feature distribution of visible speakers in V. (b) Feature distribution of unseen speakers in R. (c) Feature distribution of unseen speakers in R+. (d) PB-DSR+ enhances R+. The black dots in (d) represent prototypes, while the green and red dots represent samples correctly and incorrectly classified by R+, respectively, with their labels coming from R+ predictions. A red line connects the red dots to their correct prototypes through PB-DSR+. Each label is a word ID in the UASpeech dataset.

[0065] The correspondence between the DSR model performance and the distribution of speech features is illustrated by visualizing the first frame of HuBERT features in the speech data. Figure 2 (a)(b)(c) show the feature distribution of the same speaker in different settings. For the seen speakers, SIDSR model V shows good speech recognition performance, which is reflected in the clustering pattern of speech features of the same word. In contrast, for the unseen speakers, SIDSR model R shows poor performance, which is evident in the more chaotic feature distribution. However, by integrating SCL, the R+ model enhances the organization of speech feature distribution, demonstrating its potential to improve model performance in different speaker scenarios.

[0066] exist Figure 2 In (d), speech recognition results for unseen speakers are examined using R+ and PB-DSR+. It is clear that the features incorrectly recognized by the R+ model are almost identical to their correct prototypes, showing closeness, but severely lacking in precision recognition due to the model's inability to adapt to the novel speech patterns of unseen speakers. In contrast, PB-DSR+ can accurately recognize these samples by using a straightforward distance metric for classification. This improvement significantly improves speech recognition accuracy for unseen speakers, highlighting the efficacy of PB-DSR in overcoming the adaptation challenges faced by traditional SIDSR models.

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "include..." or "comprise..." do not exclude the existence of other elements in the process, method, article or terminal device including the elements. In addition, in this article, "greater than", "less than", "exceed" and the like are understood to exclude the number itself; "above", "below", "within" and the like are understood to include the number itself.

[0068] Although the above embodiments have been described, once those skilled in the art know the basic creative concepts, they can make additional changes and modifications to these embodiments. Therefore, the above description is only an embodiment of the present invention and does not limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.< / unk>

Claims

1. Prototype-based dysarthric speech recognition method, It is characterized in that The following steps are involved: S1. Fine-tune HuBERT for feature extraction. Fine-tune the pre-trained self-supervised learning model HuBERT on the dysarthric speech recognition task. In the fine-tuning stage, the connectionist temporal classification loss function and the supervised contrastive learning loss function are used to effectively guide the training of the model. The fine-tuned HuBERT is used as a feature extractor. S2, build a prototype for each word; S3. Prototype-based classification.

2. The prototype-based dysarthric speech recognition method according to claim 1, It is characterized in that The step S1 specifically includes selecting a connectionist temporal classification loss function as a main loss function; In addition, the speaker-independent model performs poorly in recognizing speech from speakers with more severe dysarthria or from unseen dysarthric speakers, which may be attributed to the challenge of the speaker-independent model in distinguishing the speech of different words from these speakers. Therefore, the connectionist temporal classification loss is integrated with the supervised contrastive learning loss as the overall loss function, with the aim of enhancing the model's ability to learn improved feature representations by simultaneously increasing the inter-class distance and reducing the intra-class distance; In supervised contrastive learning, in a training batch, samples that share the same label are considered positive samples of each other, while samples with different labels are considered negative samples. Therefore, the supervised contrastive learning loss, denoted as L SCL , the formula is as follows: Where i∈I={1,…,N} represents the index of the speech, A(i) represents all indices except i, and x i represents the features extracted by HuBERT, P(i) represents all the indices of the positive samples of sample i, the features of the same words as sample i are positive samples, and τ is the temperature hyperparameter; The total loss is denoted as L Total , the connectionist temporal classification loss L CTC Compared with supervised contrastive learning loss L SCL The combination is as follows: L Total =L CTC +L SCL . (2) 3. The prototype-based dysarthric speech recognition method according to claim 1, It is characterized in that The step S2 specifically includes classifying the speech of different words spoken by the unseen dysarthric speaker into different categories, and using a limited amount of speech data provided by the unseen dysarthric speaker as a support set, and extracting speech features from the support set using a fine-tuned HuBERT; Subsequently, the embodiment averages the features of the same word to construct a prototype for each word; These prototypes serve as representations for individual words.

4. The prototype-based dysarthric speech recognition method according to claim 1, It is characterized in that The step S3 specifically includes: using the fine-tuned HuBERT to extract features of the test speech; it should be emphasized that the test speech and the speech in the support set are from the same unseen dysarthria speaker, then calculating the distance between the test speech features and each prototype, and finally, selecting the word represented by the closest prototype as the recognition result.