Vietnamese speech recognition data filtering method based on acoustic spatial distribution constraint
Patent Information
- Application Number
- CN202310785589.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-29
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2043-06-29
AI Technical Summary
[0005]本发明提供了基于声学空间分布约束的越南语语音识别数据筛选方法，以用于解决由于低资源下越南语声调声学分布不平衡导致的偏置问题，同时过分依赖弱监督信号也会造成真实场景与训练场景下声调在声学空间的数据分布偏差导致越南语语音识别模型在部分语音空间中表征的偏置，从而影响整个越南语语音识别的性能的问题
[0030] This invention utilizes label confidence to constrain the contribution of pseudo-label weak supervision signals to the model training process; it adds a pseudo-data filtering method during self-training to effectively improve the recognition performance of Vietnamese speech recognition under low resource conditions; and it demonstrates the effectiveness of the proposed training method on the Vietnamese public dataset VIVOS through a series of ablation experiments.
Smart Images
Figure CN116863917B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for filtering Vietnamese speech recognition data based on acoustic spatial distribution constraints, and belongs to the field of natural language processing technology. Background Technology
[0002] With the development and research of deep learning in recent years, supervised speech recognition models have made significant breakthroughs in recognition performance. However, many challenges remain in speech recognition for Vietnamese, a language often characterized by low resource availability. Vietnamese is a tonal language, belonging to the melodic language category, and tones play a crucial role in distinguishing semantics. Vietnamese speech recognition still faces the challenge of insufficient language resources. This scarcity leads to an imbalance in the acoustic space distribution of data under low-resource conditions, resulting in a significant discrepancy between the input space and the real space for deep learning models. In-depth research into semi-supervised learning offers a feasible approach for exploring speech recognition in Vietnamese under low-resource conditions.
[0003] Research on semi-supervised learning methods began in the 1970s, initially encompassing only a few types, including transductive learning, self-training, and generative models. However, in the 1990s, with the rapid development of artificial intelligence and the strong demand for labeled data, semi-supervised learning entered a new stage of development. Self-training and pre-training have been applied to speech recognition. For major languages, fine-tuning through pre-training has become a paradigm. Research on Vietnamese is relatively limited; under pseudo-label conditions, the quality of training data is difficult to guarantee, and over-reliance on poor-quality pseudo-labels can negatively impact the performance of the recognition system.
[0004] This invention proposes a pseudo-data filtering method based on acoustic spatial distribution constraints, which uses label confidence to constrain the contribution of weak supervision signals from pseudo-labels to the training process. Adding a pseudo-data filtering method during self-training effectively improves the recognition performance of Vietnamese speech recognition under low-resource conditions. Summary of the Invention
[0005] This invention provides a method for filtering Vietnamese speech recognition data based on acoustic spatial distribution constraints, in order to solve the bias problem caused by the unbalanced acoustic distribution of Vietnamese tones under low resource conditions. At the same time, over-reliance on weak supervision signals can also cause the deviation of the acoustic spatial data distribution of tones in the real scene and the training scene, resulting in the bias of the Vietnamese speech recognition model in some speech spaces, thus affecting the overall performance of Vietnamese speech recognition.
[0006] The technical solution of this invention is: a method for filtering Vietnamese speech recognition data based on acoustic spatial distribution constraints, the specific steps of which are as follows:
[0007] Step 1: Collect existing publicly available Vietnamese speech datasets and unlabeled Vietnamese speech data. The publicly available Vietnamese speech datasets are used as manually labeled Vietnamese speech text corpus, and the unlabeled Vietnamese speech data are used as unlabeled corpus.
[0008] Step 2, Preprocessing of speech and text: The speech is denoised using a Vietnamese speech denoising model, and the Vietnamese text data, i.e., the tags, is Unicode-standardized.
[0009] Step 3: Convert the text into a phoneme sequence, and then use the Vietnamese speech-text alignment model to align the phoneme sequence with the speech of the unlabeled corpus to obtain the aligned pseudo-labeled corpus.
[0010] Step 4: Initialize a teacher model from the existing wav2vec2 pre-trained model, fine-tune the initialized teacher model, and use the fine-tuned teacher model to obtain pseudo-labeled corpus from unlabeled corpus data.
[0011] Step 5: The aligned pseudo-labeled corpus obtained in Step 3 and the pseudo-labeled corpus obtained in Step 4 are filtered using the tone contribution of acoustic spatial distribution constraints. The student model is then self-trained using the filtered aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus.
[0012] Step 6: By iterating through the self-training process in Step 5, the optimal Vietnamese speech recognition model is obtained, and experimental verification is performed using a Vietnamese speech test set.
[0013] Furthermore, in Step 4, the wav2vec2 pre-trained model is used as the teacher model. Since wav2vec2 has been pre-trained unsupervised on speech datasets of other languages, the model has already learned some acoustic information. Then, it is fine-tuned using labeled data, and finally, the fine-tuned model is used to obtain pseudo-labeled corpus through unlabeled speech data.
[0014] Furthermore, the specific steps of Step 4 are as follows:
[0015] Initialize a teacher model from an existing wav2vec2 pre-trained model, and fine-tune it on other existing public datasets to obtain a fine-tuned teacher model. The fine-tuning process minimizes l. ctc The loss function is:
[0016] l ctc =l(y i ,f(x i ,θ t )) (1)
[0017] Among them l ctc It is the loss function, y i For the i-th real label, θ t These are the parameters of the initialized teacher model, x i It is the input Vietnamese speech, f(x) i ,θ t ) is the predicted output of the initialized teacher model;
[0018] Unlabeled speech data The input is fed into the fine-tuned teacher model to obtain the pseudo-labels predicted by the fine-tuned teacher model. The specific formula is as follows:
[0019]
[0020] θ t1 These are the parameters of the fine-tuned teacher model.
[0021] Furthermore, in Step 5, the screening of the aligned pseudo-labeled corpus obtained in Step 3 and the pseudo-labeled corpus obtained in Step 4 using the tone contribution of acoustic spatial distribution constraints specifically includes the following:
[0022] Using aligned pseudo-labeled corpora, the top 80% of each sentence in the pseudo-labeled corpora are selected as the training subset based on their tone contribution under acoustic spatial distribution constraints in descending order; the formula for calculating the tone contribution under acoustic spatial distribution constraints is as follows:
[0023]
[0024] Where p i The score represents the coverage factor of the i-th tone in n sentences. i This indicates the contribution of the i-th tone in the current sentence;
[0025] Furthermore, in Step 5, the self-training of the student model using the filtered, aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus specifically includes the following:
[0026] When the aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus were sequentially input into the student model for self-training, a dynamic weight was used to combine the loss of the Vietnamese speech-text corpus and pseudo-labeled data. The total loss function is shown in the following formula.
[0027]
[0028] Where θ s These are the parameters of the current student model. and It consists of manually labeled Vietnamese speech-text corpora and their associated speech data and tags. and It consists of aligned pseudo-labeled corpora after filtering, speech data within the pseudo-labeled corpora, and pseudo-labels. It is the loss function for manually labeled Vietnamese speech-text corpus data. It consists of the filtered, aligned pseudo-labeled corpus, the pseudo-label loss function, and α. t For label confidence, Let α be the weight parameter at the current time step t. T The weight parameters are for the total time step T.
[0029] The beneficial effects of this invention are:
[0030] This invention utilizes label confidence to constrain the contribution of pseudo-label weak supervision signals to the model training process; it adds a pseudo-data filtering method during self-training to effectively improve the recognition performance of Vietnamese speech recognition under low resource conditions; and it demonstrates the effectiveness of the proposed training method on the Vietnamese public dataset VIVOS through a series of ablation experiments. Attached Figure Description
[0031] Figure 1 This is a flowchart from the present invention. Detailed Implementation
[0032] Example 1: As Figure 1 As shown, a method for filtering Vietnamese speech recognition data based on acoustic spatial distribution constraints is described, and the specific steps of the method are as follows:
[0033] Step 1: Collect existing publicly available Vietnamese speech datasets and unlabeled Vietnamese speech data. The publicly available Vietnamese speech datasets are used as manually labeled Vietnamese speech text corpus, and the unlabeled Vietnamese speech data are used as unlabeled corpus.
[0034] Step 2, Preprocessing of speech and text: The speech is denoised using a Vietnamese speech denoising model, and the Vietnamese text data, i.e., the tags, is Unicode-standardized.
[0035] Step 3: Convert the text into a phoneme sequence, and then use the Vietnamese speech-text alignment model to align the phoneme sequence with the speech of the unlabeled corpus to obtain the aligned pseudo-labeled corpus.
[0036] Step 4: Initialize a teacher model from the existing wav2vec2 pre-trained model, fine-tune the initialized teacher model, and use the fine-tuned teacher model to obtain pseudo-labeled corpus from unlabeled corpus data.
[0037] Furthermore, in Step 4, the wav2vec2 pre-trained model is used as the teacher model. Since wav2vec2 has been pre-trained unsupervised on speech datasets of other languages, the model has already learned some acoustic information. Then, it is fine-tuned using labeled data, and finally, the fine-tuned model is used to obtain pseudo-labeled corpus through unlabeled speech data.
[0038] The specific steps of Step 4 are as follows:
[0039] Initialize a teacher model from an existing wav2vec2 pre-trained model, and fine-tune it on other existing public datasets to obtain a fine-tuned teacher model. The fine-tuning process minimizes l. ctc The loss function is:
[0040] l ctc =l(y i ,f(x i ,θ t )) (1)
[0041] Among them l ctc It is the loss function, y i For the i-th real label, θ t These are the parameters of the initialized teacher model, x i It is the input Vietnamese speech, f(x) i ,θ t ) is the predicted output of the initialized teacher model;
[0042] Unlabeled speech data The input is fed into the fine-tuned teacher model to obtain the pseudo-labels predicted by the fine-tuned teacher model. The specific formula is as follows:
[0043]
[0044] θ t1 These are the parameters of the fine-tuned teacher model.
[0045] Step 5: The aligned pseudo-labeled corpus obtained in Step 3 and the pseudo-labeled corpus obtained in Step 4 are filtered using the tone contribution of acoustic spatial distribution constraints. The student model is then self-trained using the filtered aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus.
[0046] Using acoustic spatial distribution constraints to filter data enables the model to learn more semantic knowledge even with low resource requirements. When the model selects a subset from the pseudo-labeled corpus datasets in Step 3 and Step 4 as training corpus, it aims to cover as much Vietnamese tone information as possible and ensure a broad acoustic spatial distribution. Therefore, a coverage factor is introduced to select appropriate speech-text pairs to form the training subset while ensuring the quality of the pseudo-labeled speech-text pairs. In Step 5, the aligned pseudo-labeled corpus obtained in Step 3 and Step 4 are filtered using the tone contribution of acoustic spatial distribution constraints, specifically including the following:
[0047] Using aligned pseudo-labeled corpora, the top 80% of each sentence in the pseudo-labeled corpora are selected as the training subset based on their tone contribution under acoustic spatial distribution constraints in descending order; the formula for calculating the tone contribution under acoustic spatial distribution constraints is as follows:
[0048]
[0049] Where p i The score represents the coverage factor of the i-th tone in n sentences. i This indicates the contribution of the i-th tone in the current sentence;
[0050] Furthermore, regarding the label confidence weight, which is a dynamic weight involved in this invention: Intuitively, pseudo-labeled data contains more noise than manually labeled data. In order to make the posterior probability of the pseudo-label as consistent as possible with that of the correct label during self-training, a dynamic weight is designed to combine the losses of manually labeled data and pseudo-labeled data. Specifically, in Step 5, the self-training of the student model using the filtered and aligned pseudo-labeled corpus, the pseudo-labeled corpus, and the manually labeled Vietnamese speech-text corpus includes the following:
[0051] When the aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus were sequentially input into the student model for self-training, a dynamic weight was used to combine the loss of the Vietnamese speech-text corpus and pseudo-labeled data. The total loss function is shown in the following formula.
[0052]
[0053] Where θ s These are the parameters of the current student model. and It consists of manually labeled Vietnamese speech-text corpora and their associated speech data and tags. and It consists of aligned pseudo-labeled corpora after filtering, speech data within the pseudo-labeled corpora, and pseudo-labels. It is the loss function for manually labeled Vietnamese speech-text corpus data. It consists of the filtered, aligned pseudo-labeled corpus, the pseudo-label loss function, and α. t For label confidence, Let α be the weight parameter at the current time step t. T Let α be the weighting parameter for the total time step T. T Take 1.
[0054] Step 6: By iterating through the self-training process in Step 5, the optimal Vietnamese speech recognition model is obtained, and experimental verification is performed using a Vietnamese speech test set.
[0055] To illustrate the effectiveness of this invention, the following experiments were conducted:
[0056] (a) Experimental setup and evaluation metrics
[0057] The pre-trained model in this method is wav2vec 2 base. This invention uses PyTorch 1.13 to build the model, employing label-smoothed cross-entropy loss as the objective function. The label smoothing rate is 0.1, the gradient clipping is 20, the pre-encoder convolutional kernel size is 3*3, the encoder-decoder hidden layer vectors are 768-dimensional, the encoder has 6 self-attention layers, the decoder embedding dimension is 512, the decoder has 6 self-attention layers, and the label smoothing rate is 0.1. During training, the Adam optimizer is used for optimization, with the learning rate dynamically adjusted. The evaluation metric for the recognition model is the word error rate (WER), which is the edit distance between the predicted text and the real text divided by the length of the real text. A lower WER value indicates a higher recognition accuracy.
[0058] (b) Comparative Experiment
[0059] To compare the performance of Vietnamese language recognition under low resource conditions with existing technologies, we selected the classic Tdnn-LSTM structure and the latest Conformer structure as the most supervised methods in current end-to-end language recognition. The pre-trained model used was the open-source Whisper work from OpenAI in 2022. To ensure a fair comparison, all Vietnamese language recognition models were modeled at the Vietnamese character level, trained for 50 epochs, and their word error rate (WER) on the VIVOS test set was compared.
[0060] Table 1 shows the comparative experiments of different models on the VivoOS test set.
[0061] Tdnn-lstm+ctc Supervised 92.96％ Conformer+ctc Supervised 30.22％ Whisper fine-tuning Pre-training + Fine-tuning 24.48％ Wav2vec2 fine-tuning Pre-training + Fine-tuning 24.21％ This invention Pre-training + Fine-tuning 11.38％
[0062] As shown in Table 1, the traditional Tdnn-1stm supervised model, trained under low-resource conditions, exhibits severe underfitting in Vietnamese speech recognition, demonstrating that general methods are prone to underfitting when corpora are scarce. This invention, in contrast to the pre-training plus fine-tuning paradigm, demonstrates through experiments that introducing pseudo-label data effectively improves the acoustic spatial representation of Vietnamese, further enhancing the performance of Vietnamese speech recognition.
[0063] (c) Ablation test
[0064] To investigate the benefits of aligned pseudo-label data for fine-tuning Vietnamese speech models, this ablation experiment used a dataset that included the entire VIVOS labeled training dataset and a subset of aligned pseudo-labels selected based on tone distribution. The experiment explored whether to use pseudo-label aligned data, whether to use a selection method, and whether to use label confidence weights to fine-tune the model. The experimental results are shown in Table 2.
[0065] Pseudo-label ablation experiment: To investigate the impact of pseudo-label data on the fine-tuning process of the Vietnamese language recognition model, Table 2 shows the Vietnamese word error rates when using the pseudo-label method, when not using the pseudo-label method, and when not using pseudo-label fine-tuning.
[0066] Table 2 shows the comparison of pseudo-label ablation experiments.
[0067] This method 12.38％ - Pseudo-tag filtering method 15.81％ - Pseudo-label data 24.21％
[0068] As shown in Table 3, this experiment sequentially removed the pseudo-labels and used pseudo-label data. The table shows that pseudo-label data improved both the accuracy and performance of the recognition system. However, relying entirely on pseudo-label data leads to a bias in the recognition model due to the large amount of biased tone information within it, resulting in weaker generalization ability. Using the filtering method further improves the Vietnamese language recognition performance. The effectiveness of the filtering method is demonstrated, and it is shown that filtering the model enhances its tone perception and improves its generalization ability.
[0069] Label confidence weight ablation experiment: To investigate the impact of label confidence weights on the fine-tuning process, Table 3 shows the ablation experiment for label confidence weights. This involved removing the confidence weights while maintaining the pseudo-label selection method and the participation of pseudo-labels in training. For the confidence weights, the model initially relies more heavily on the pseudo-label dataset. Since the pseudo-label data is larger than the real label data, the reliance on pseudo-labels needs to be reduced in the later stages of model training. Removing label confidence weights simply involves setting both the pseudo-label weights and the supervised label weights to 1.
[0070] Table 3 shows the comparison of confidence level ablation experiments.
[0071] This method 12.38％ -Label confidence weight 14.20％
[0072] As shown in Table 3, the Vietnamese speech recognition system's performance decreased by 14% after removing the label confidence weights. Because the pseudo-label data contains some erroneous information, over-reliance on pseudo-label data can confuse the model, further leading to a decrease in recognition accuracy.
[0073] Speech denoising and ablation experiments: To investigate whether speech denoising has a robust improvement effect on speech recognition, the design is shown in Table 4. The pseudo-label data selection method and label confidence weighting method were maintained throughout the experiments.
[0074] Table 4 shows the comparison of ablation test results.
[0075] This method 13.56％ +Single-channel speech noise reduction based on time-frequency information 18.28％ +Joint training fine-tuning 11.38
[0076] As shown in Table 4, introducing pseudo-label data improves the Vietnamese speech recognition model compared to not introducing pseudo-label data. Even pseudo-label data contains a large amount of weak supervision signals, and introducing pseudo-label data can enhance the convergence of the recognition model. However, directly using the speech denoising model led to a sharp increase in the model's word error rate of 34%. Due to the difference between the output space of the denoising model and the input space of speech recognition, this difference is not so obvious to the human ear, but for computers, the data domain mismatch leads to a worse recognition effect after denoising. The generative denoising model essentially removes noise information, which inevitably leads to internal noise in the denoised frequencies, preventing the recognition system from generalizing. In this table, the trained denoising model and the recognition model are jointly trained and fine-tuned, and the weights of the denoising model are frozen, allowing the Vietnamese speech recognition model to adapt to the acoustic feature distribution after denoising. Experiments show that the model obtained through joint training has a word error rate that is 16% lower than the model without speech denoising. Experiments also demonstrate that using pseudo-label selection methods, label confidence weight constraints, and speech denoising methods can improve the performance of Vietnamese speech recognition under conditions of low resource availability and the diversity of Vietnamese tones.
[0077] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for filtering Vietnamese speech recognition data based on acoustic spatial distribution constraints, characterized in that, The specific steps of the method are as follows: Step 1: Collect existing publicly available Vietnamese speech datasets and unlabeled Vietnamese speech data. The publicly available Vietnamese speech datasets are used as manually labeled Vietnamese speech text corpus, and the unlabeled Vietnamese speech data are used as unlabeled corpus. Step 2, Preprocessing of speech and text: The speech is denoised using a Vietnamese speech denoising model, and the Vietnamese text data, i.e., the tags, is Unicode-standardized. Step 3: Convert the text into a phoneme sequence, and then use the Vietnamese speech-text alignment model to align the phoneme sequence with the speech of the unlabeled corpus to obtain the aligned pseudo-labeled corpus. Step 4: Initialize a teacher model from the existing wav2vec2 pre-trained model, fine-tune the initialized teacher model, and use the fine-tuned teacher model to obtain pseudo-labeled corpus from unlabeled corpus data. Step 5: The aligned pseudo-labeled corpus obtained in Step 3 and the pseudo-labeled corpus obtained in Step 4 are filtered using the tone contribution of acoustic spatial distribution constraints. The student model is then self-trained using the filtered aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus. Step 6: By iterating through the self-training process in Step 5, the optimal Vietnamese speech recognition model is obtained, and experimental verification is performed using a Vietnamese speech test set. In Step 5, the screening of the aligned pseudo-labeled corpus obtained in Step 3 and the pseudo-labeled corpus obtained in Step 4 using the tone contribution of acoustic spatial distribution constraints specifically includes the following: Using aligned pseudo-labeled corpora, the top 80% of each sentence in the pseudo-labeled corpora are selected as the training subset based on their tone contribution under acoustic spatial distribution constraints in descending order; the formula for calculating the tone contribution under acoustic spatial distribution constraints is as follows: (1)； in In the nth sentence The coverage factor of each tone, This indicates that the first word in the current sentence is... The contribution of each tone.
2. The Vietnamese speech recognition data filtering method based on acoustic spatial distribution constraints according to claim 1, characterized in that: The step 4 mentioned using the wav2vec2 pre-trained model as the teacher model. Since wav2vec2 has been pre-trained unsupervised on speech datasets of other languages, the model has already learned some acoustic information. Then, it is fine-tuned using labeled data, and finally, the fine-tuned model is used to obtain pseudo-labeled corpus through unlabeled speech data.
3. The Vietnamese speech recognition data filtering method based on acoustic spatial distribution constraints according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Initialize a teacher model from an existing wav2vec2 pre-trained model, and fine-tune it on other existing public datasets to obtain a fine-tuned teacher model. The fine-tuning process minimizes... The loss function is: (2)； in It is a loss function. For the first A real label, These are the parameters for initializing the teacher model. It is the input Vietnamese voice. This is the predicted output of the initialized teacher model; Unlabeled speech data The input is fed into the fine-tuned teacher model to obtain the pseudo-labels predicted by the fine-tuned teacher model. The specific formula is as follows: (3)； These are the parameters of the fine-tuned teacher model.
4. The Vietnamese speech recognition data filtering method based on acoustic spatial distribution constraints according to claim 1, characterized in that: In Step 5, the self-training of the student model using the filtered, aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus specifically includes the following: When the aligned pseudo-labeled corpus, pseudo-labeled corpus, and manually labeled Vietnamese speech-text corpus were sequentially input into the student model for self-training, a dynamic weight was used to combine the loss of the Vietnamese speech-text corpus and pseudo-labeled data. The total loss function is shown in the following formula. （4）； in These are the parameters of the current student model. and It consists of manually labeled Vietnamese speech-text corpora and their associated speech data and tags. and It consists of aligned pseudo-labeled corpora after filtering, speech data within the pseudo-labeled corpora, and pseudo-labels. It is the loss function for manually labeled Vietnamese speech-text corpus data. It consists of the filtered, aligned pseudo-labeled corpus and the pseudo-label loss function. For label confidence, For the current time step The weight parameters below, Total time steps The weight parameters below.