Tibetan language conbar dialect speech recognition system based on self-reinforcement learning

Abstract

The invention discloses a Tibetan language conbar dialect speech recognition system based on self-reinforcement learning, and relates to the technical field of speech recognition, and the system comprises an unlabeled speech data collection module, a speech synthesis-based pseudo audio data collection module, an initial model training module, a model reasoning module, a data selection and enhancement module, and a model iteration training module. According to the Tibetan language conbar dialect speech recognition system based on self-reinforcement learning provided by the invention, the scale of a training data set is expanded through unlabeled speech data and pseudo-audio synthesized by speech, a speech recognition model is iteratively optimized by using the idea of self-reinforcement learning, and finally better performance is obtained in the aspect of extremely-low-resource conbar dialect speech recognition; according to the method, the speech recognition performance of the Tibetan Kangba dialect is improved, and a batch of available speech recognition data sets with high-accuracy labels are obtained; the method can be popularized to voice recognition training of other languages, especially low-resource languages, and has high application value.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a Tibetan Kham dialect speech recognition system based on self-enhancement learning. Background Art

[0002] In recent years, Automatic Speech Recognition (ASR) technology has been widely used in many fields. However, for minority languages, such as Tibetan, which are resource-scarce languages, the performance of speech recognition systems still faces many challenges. Among the three major Tibetan dialects, the existing public data sets have the largest scale of annotated speech data sets for the U-Tsang dialect, followed by the Amdo dialect, while the amount of data for the Kham dialect is the most limited, less than 6 hours, and it is an extremely low-resource language. Therefore, current research on Tibetan speech recognition is mainly focused on the U-Tsang dialect and the Amdo dialect. Due to the lack of data, relevant research on the Kham dialect is very rare, resulting in a lack of high-performance and reliable Kham dialect speech recognition models. This situation has severely restricted the application and development of the Kham dialect in speech recognition technology.

[0003] In order to improve the speech recognition performance of extremely low-resource languages, current research has the following two methods:

[0004] 1. Methods for expanding the dataset, including:

[0005] 1.1 Use data enhancement techniques, such as adding noise and spectrum expansion, to improve the model's adaptability to extremely low-resource languages ​​by expanding the training data set;

[0006] 1.2 Add manually annotated data to expand the training dataset.

[0007] 2. Improve model methods, including:

[0008] 2.1 Pre-train the speech recognition model based on a large amount of speech data in other languages, and then fine-tune it with a small amount of target language data to quickly improve the speech recognition effect of extremely low-resource languages ​​and reduce dependence on labeled data;

[0009] 2.2 Based on self-supervised learning and semi-supervised learning, the model is trained through unlabeled data so that it can still learn speech features in resource-scarce scenarios and improve the generalization ability of the model.

[0010] 2.3 Through the multi-language / multi-dialect joint training method, the target language and other language data are jointly trained, and the common features between different languages ​​are utilized to improve the model's speech recognition ability for the target language.

[0011] However, for the Kham dialect, the expansion of manually annotated data requires the participation of a large number of Kham dialect native speakers, which is time-consuming and costly, limiting the expansion of dataset resources. Therefore, there is currently a lack of research on improving speech recognition performance through manual annotation.

[0012] To this end, the present invention proposes a Tibetan Kham dialect speech recognition system based on self-reinforcement learning. Summary of the invention

[0013] The purpose of the present invention is to solve the defects in the prior art and to propose a Tibetan Kham dialect speech recognition system based on self-reinforcement learning.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] A Tibetan Kham dialect speech recognition system based on self-reinforcement learning includes an unlabeled speech data collection module, a pseudo audio data collection module based on speech synthesis, an initial model training module, a model reasoning module, a data selection and enhancement module, and a model iteration training module.

[0016] Furthermore, the unlabeled voice data collection module is used to collect unlabeled Kangba voice data from public network resources;

[0017] The unlabeled speech data collection module comprises:

[0018] Web crawler module, used to crawl Kangba voice data from the public network;

[0019] A manual segmentation module, connected to the web crawler module, for segmenting the crawled voice data at the sentence level;

[0020] A data preprocessing module, connected to the manual segmentation module, for preprocessing the voice data and regularizing the data;

[0021] The data integration module is connected to the data preprocessing module and is used to uniformly mark and store all the data and integrate them into a final unlabeled Kham dialect speech data set.

[0022] Furthermore, the pseudo audio data collection module based on speech synthesis is used to generate pseudo audio data of the Kham dialect based on Tibetan text through speech synthesis technology.

[0023] Furthermore, the initial model training module trains an initial Kangba dialect speech recognition model based on an existing manually annotated data set.

[0024] Furthermore, the model reasoning module is connected to the unlabeled speech data collection module, the pseudo audio data collection module based on speech synthesis and the initial model training module, and is used to recognize the unlabeled speech data and the synthesized pseudo speech as Tibetan text based on the trained speech recognition model.

[0025] Furthermore, the data selection and enhancement module is connected to the model reasoning module, and is used to select reliable speech-text pairs from unlabeled speech data and synthetic pseudo-speech, and expand them together with the manually labeled speech recognition data set into a training data set.

[0026] Furthermore, the model iteration training module is connected to the model reasoning module and the data selection and enhancement module, and is used to iteratively enhance the training of speech recognition model performance based on the expanded training data set, and ultimately obtain a high-accuracy Tibetan Kham dialect speech recognition model.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The Tibetan Kham dialect speech recognition system based on self-reinforcement learning provided in the present invention expands the size of the training data set through unlabeled speech data and pseudo audio of speech synthesis, and iteratively optimizes the speech recognition model using the idea of ​​self-reinforcement learning, and finally obtains better performance in the extremely low-resource Kham dialect speech recognition;

[0029] 2. The present invention not only improves the speech recognition performance of the Tibetan Kham dialect, but also obtains a batch of available speech recognition data sets with high-accuracy labels;

[0030] 3. The present invention can be extended to speech recognition training of other languages, especially low-resource languages, and has high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0032] Figure 1 It is a schematic diagram of the system composition of the present invention;

[0033] Figure 2 It is a schematic diagram of the composition of the unlabeled voice data collection module in the present invention.

[0034] In the figure: 1 unlabeled speech data collection module, 11 web crawler module, 12 manual segmentation module, 13 data preprocessing module, 14 data integration module;

[0035] 2 Pseudo audio data collection module based on speech synthesis, 3 Initial model training module, 4 Model reasoning module, 5 Data selection and enhancement module, 6 Model iterative training module. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention;

[0037] Reference Figure 1-2 , a Tibetan Kham dialect speech recognition system based on self-reinforcement learning,

[0038] It includes an unlabeled speech data collection module 1, a pseudo audio data collection module based on speech synthesis 2, an initial model training module 3, a model reasoning module 4, a data selection and enhancement module 5, and a model iterative training module 6.

[0039] Furthermore, the unlabeled speech data collection module 1 is used to collect unlabeled Kangba speech data from public network resources;

[0040] The unlabeled speech data collection module 1 includes:

[0041] A network crawler module 11 is used to crawl Kangba voice data from the public network;

[0042] The manual segmentation module 12 is connected to the web crawler module 11 and is used to segment the crawled voice data at the sentence level;

[0043] The data preprocessing module 13 is connected to the manual segmentation module 12 and is used to preprocess the speech data and regularize the data;

[0044] The data integration module 14 is connected to the data preprocessing module 13 and is used to uniformly mark and store the data and integrate them into the final unlabeled Kham dialect speech data set.

[0045] Furthermore, the pseudo audio data collection module 2 based on speech synthesis is used to generate pseudo audio data of the Kham dialect according to Tibetan text through speech synthesis technology.

[0046] Furthermore, the initial model training module 3 trains an initial Kham dialect speech recognition model based on an existing manually annotated data set.

[0047] Furthermore, the model reasoning module 4 is connected to the unlabeled speech data collection module 1, the speech synthesis-based pseudo audio data collection module 2 and the initial model training module 3, and is used to recognize the unlabeled speech data and the synthesized pseudo speech as Tibetan text based on the trained speech recognition model.

[0048] Furthermore, the data selection and enhancement module 5 is connected to the model reasoning module 4 to select reliable speech-text pairs from unlabeled speech data and synthetic pseudo-speech, and expand them together with the manually labeled speech recognition data set into a training data set.

[0049] Furthermore, the model iteration training module 6 is connected to the model reasoning module 4 and the data selection and enhancement module 5, and is used to iteratively enhance the training of the speech recognition model performance based on the expanded training data set, and finally obtain a high-accuracy Tibetan Kham dialect speech recognition model.

[0050] The above-mentioned Tibetan Kham dialect speech recognition system based on self-reinforcement learning first collects unlabeled Tibetan Kham dialect speech data from the public network and generates pseudo audio from the trained Tibetan Kham dialect speech synthesis system, and then uses a small amount of manually labeled Kham dialect speech recognition data set to train the initial Kham dialect speech recognition model based on the pre-training-fine-tuning structure, and then uses the model to infer the above two unlabeled data and the synthesized pseudo audio data to obtain text labels. After data screening and enhancement, an expanded training data set is obtained. Iterative training is performed on this data set to gradually optimize the speech recognition performance. Finally, a reliable Tibetan Kham dialect speech recognition model is obtained to improve the accuracy of the model. In practical application, the present invention can be applied to speech recognition training of other low-resource or extremely low-resource languages, and has high practical value.

[0051] In one embodiment of the present invention, a Tibetan Kham dialect speech recognition system based on self-reinforcement learning is proposed. Figure 1As shown, this embodiment includes: an unlabeled speech data collection module 1, which is used to collect unlabeled Kham speech data from public network resources; a pseudo audio data collection module 2 based on speech synthesis, which is used to generate pseudo audio data of Kham dialect according to Tibetan text through speech synthesis technology; an initial model training module 3, which is used to train an initial Kham dialect speech recognition model based on an existing manually labeled data set; a model reasoning module 4, which is connected to the unlabeled speech data collection module 1, the pseudo audio data collection module 2 based on speech synthesis and the initial model training module 3, and is used to recognize the unlabeled speech data and the synthesized pseudo speech based on the trained speech recognition model. Tibetan text; data selection and enhancement module 5, connected to the source model reasoning module 4, is used to select reliable speech-text pairs from unlabeled speech data and synthetic pseudo-speech, and expand them together with the manually labeled speech recognition data set into a training data set; model iteration training module 6, connected to the source model reasoning module 4 and the data selection and enhancement module 5, is used to enhance the performance of the training speech recognition model based on the expanded training data set, and then, based on the trained enhanced speech recognition model, re-reason the data from the unlabeled speech data collection module 1 and the pseudo-audio data collection module 2 based on speech synthesis, update and expand the expanded training set, and iterate. In this process, the reliability of the expanded training data set is gradually improved, and the speech recognition performance is gradually improved.

[0052] In this embodiment, the unlabeled speech data collection module 1 is used to collect unlabeled Kangba speech data from public network resources. The unlabeled speech data collection module 1 includes a network crawler module 11, which is used to crawl Kangba speech data from the public network; a manual segmentation module 12, which is used to segment the crawled speech data at the sentence level; a data preprocessing module 13, which is used to preprocess the speech data and regularize the data; a data integration module 14, which is used to uniformly mark and store all data and integrate them into the final unlabeled Kangba dialect speech data set.

[0053] The web crawler module 11 is used to crawl Kham Tibetan voice data from the public network. First, a web page or article containing Kham dialect audio or time frequency is searched from public network resources, and then a request is sent through request to obtain a set of link addresses of the target web page. After that, for each article, the splinter-based crawler technology is used to obtain the Kham Tibetan audio or video files in each web page. For video files, the audio content is extracted through a Python program, and finally an audio file containing the Kham dialect is obtained.

[0054] The manual segmentation module 12 is connected to the web crawler module 11 and is used to segment the crawled voice data at the sentence level. For each audio file, the Praat software is used to segment the sentence level. (Praat is a professional phonetics software, which is mainly used for analyzing, annotating, processing and synthesizing voice signals. It is commonly used for voice annotation and voice analysis.) Generally speaking, the pauses between sentences are relatively long. Therefore, in the segmentation process, the method of segmentation based on pauses is adopted, and the participation of native speakers of the Tibetan Kham dialect is not required to save time and cost.

[0055] The data preprocessing module 13 is connected to the manual segmentation module 12, and is used to preprocess the speech data and regularize the data. First, the audio type conversion is performed. Since there are audios in MP3 and WAV formats in the data set, all audio types are uniformly converted to WAV types. Then, the audio is resampled. Due to different data sources, the sampling rates of different audio files are different. Therefore, all audios are resampled and the sampling frequencies of all audios are unified to 16kHz. Finally, amplitude normalization is performed. The difference in strength between audio signals from different sources is large, and normalization is used to regularize the data amplitude to between -1 and +1. That is, for each audio, s = [s1, s2, ..., s L ], L represents the number of sampling points in the audio, and the maximum value of the amplitude is calculated The normalization process can be expressed as Among them, l=1,2,...,L.

[0056] The data integration module 14 is connected to the data preprocessing module 13, and is used to uniformly mark and store all data and integrate them into the final unlabeled Kham dialect speech data set. First, rename the preprocessed audio file. The naming method is "speaker serial number_audio serial number.wav", where the speaker serial number is used to distinguish different speakers, and is identified by three numeric characters, such as "001", and the audio serial number represents different sentences segmented by the same speaker, and is identified by four numeric characters, such as "0001". The number of numeric characters changes with the number of segmented sentences. If each speaker's sentence is represented by more than four numeric characters, five numeric characters can be used. Then, all audio files of the same speaker are placed in the same folder, and the folder name is represented by the speaker serial number. Finally, these speaker audio folders are placed in a folder, and this folder is the collected unlabeled Tibetan Kham dialect speech data set.

[0057] The pseudo audio data collection module 2 based on speech synthesis is used to generate pseudo audio data of the Kham dialect according to the Tibetan text through speech synthesis technology. First, collect relevant texts for speech synthesis of the Tibetan Kham dialect. Since different Tibetan dialects are unified in Tibetan in writing form, it is only necessary to select from the Tibetan text. In practice, these texts are mainly selected from the Tibetan part of the Tibetan-Chinese machine translation data set, and the content covers multiple fields. For longer sentences or paragraphs, the sentences are manually processed according to the inter-sentence symbols such as single vertical symbols and double vertical symbols. After the processing is completed, the sentences are named according to the consecutive numbers and stored as txt files. Subsequently, the above text is input into the open source Kham dialect speech synthesis tool to generate a pseudo audio file of the corresponding sentence. The generated audio file will be saved in wav format and named in accordance with the sentence sequence number of the text, thereby constructing a speech recognition data set of pseudo audio-text pairs. It should be noted that the Kham dialect audio generated by speech synthesis may have problems such as misreading or missing reading. These audios with pronunciation errors will be screened and eliminated later to ensure the quality of the data set. In addition, the speech synthesis data only contains the voice of a single speaker. To ensure the diversity and balance of the speakers' voices, the number of sentences should not be too large. The maximum number of sentences of all speakers in the unlabeled speech data collection module 1 can be taken to ensure that the scale of the data set generated based on speech synthesis after pseudo-audio screening is reasonable and balanced.

[0058] The initial model training module 3 is used to train the initial Kangba dialect speech recognition model based on the existing manually annotated data set. In order to improve the training efficiency and model performance, the present invention adopts a method of directly fine-tuning the public high-resource language pre-training model instead of training from scratch. First, in view of the fact that Mandarin and Tibetan belong to the Sino-Tibetan language family, Mandarin is selected as the basic language in the pre-training stage. The existing high-resource language (such as Mandarin, English, etc.) speech recognition model has a strong generalization ability and mature pre-training results. Therefore, the use of a public and well-trained Mandarin pre-training model as a basis can significantly reduce the time and resource overhead of model training, and provide a good starting point for the speech recognition task of the Kangba dialect. Then, through fine-tuning, training is performed on a small amount of manually annotated data sets of the Kangba dialect. During the fine-tuning process, due to the small amount of manually annotated Kangba method speech recognition training data, the weights of the first several layers of the model are frozen during the fine-tuning process, and only the weights of the few layers behind the model are adjusted. The number of layers for adjusting the weights changes with the size of the data set, and the optimal value is selected through multiple experiments. This strategy enables the model to retain the general feature extraction capabilities of high-resource languages ​​while learning language patterns unique to the Kham dialect, such as phonemes, vocabulary, and syntactic features, through the decoding layer. Finally, an initial Kham dialect speech recognition model was obtained after a small amount of annotation and fine-tuning.

[0059] The model reasoning module 4 is connected to the unlabeled speech data collection module 1, the pseudo audio data collection module 2 based on speech synthesis and the initial model training module 3, and is used to recognize the unlabeled speech data and the synthesized pseudo speech as Tibetan text based on the trained speech recognition model. First, this part uses the speech recognition model fine-tuned in the initial model training module 3 to automatically reason about two data sets from the unlabeled speech data collection module 1 and the pseudo audio data collection module 2 based on speech synthesis, and generates preliminary Tibetan text labels, i.e., pseudo labels. Although these labels do not have the accuracy of manual annotation, they can play an effective role in data expansion and help the model to be further optimized. Afterwards, the model is gradually optimized through self-enhancement learning, and the optimized speech recognition is used for reasoning again, which can continuously improve the accuracy of the label. This process realizes the gradual expansion of reliable data sets, and continuously improves the speech recognition performance by iteratively updating the training model.

[0060] The data selection and enhancement module 5 is connected to the model reasoning module 4 and is used to select reliable speech-text pairs from unlabeled speech data and synthetic pseudo-speech, and expand them together with the manually labeled speech recognition data set into a training data set. First, data selection is performed based on the pseudo-labels output by the model reasoning module 4. This step includes two branches according to different data sets:

[0061] (1) Screening of unlabeled speech data: The goal is to remove pseudo-labels that are obviously mismatched and ensure that the pseudo-labels generated from unlabeled data have a high degree of credibility. For pseudo-labels from unlabeled speech data, the ratio of sentence length to word count is used to judge the degree of match between the text label and the speech. If the generated text label has a reasonable correspondence between sentence length and speech duration, and the word count ratio meets the preset standard, the speech-text pair is judged to be more reliable. The ratio is obtained by calculating the average value of the ratio of speech length to text word count in the manually annotated dataset. The qualified speech data and pseudo-labels constitute a valid Kham dialect speech recognition data.

[0062] (2) Screening of synthetic pseudo-audio: The goal is to filter out mispronounced or inaccurate synthetic audio to ensure the quality of the dataset. For pseudo-audio data generated based on speech synthesis, the consistency of the synthesized text and the label generated by the model is compared to determine the consistency between the two. If the synthesized text has a high consistency rate with the pseudo-label generated by reasoning, the speech-text pair is considered to be highly reliable, and the data that meets the criteria will be selected into the training set.

[0063] After completing the data screening of the two branches, the module integrates the existing small amount of manually annotated Kham dialect speech recognition data sets, the screened unlabeled data with pseudo-labels, and the synthetic pseudo-audio data to form a larger training data set, thereby providing richer and more comprehensive training data for the speech recognition model.

[0064] The model iteration training module 6 is connected to the source model reasoning module 4 and the data selection and enhancement module 5, and is used to strengthen the training of the speech recognition model performance based on the expanded training data set. After that, based on the trained enhanced speech recognition model, the data from the unlabeled speech data collection module 1 and the pseudo audio data collection module 2 based on speech synthesis are re-inferred, and the expanded training set is updated and expanded for iteration. In this process, the reliability of the expanded training data set is gradually improved, and the speech recognition performance is gradually improved. At the beginning of each round of training, the module first uses the extended training data set from pseudo labels of manually labeled data, unlabeled data, and synthesized pseudo audio to strengthen the training of the speech recognition model. Due to different data sources, there are differences in the quality and reliability of the data. In order to balance the contribution of various types of data to the model, a weighted cross entropy loss function is adopted to assign different weights to different data. The loss function is designed as follows:

[0065]

[0066] Among them, w i Represents the weight of each sample, which is adjusted according to the data quality. For manually annotated data, w i is larger, while for pseudo-labeled data, w i In this way, the model can pay more attention to high-quality manually labeled data during training, while using pseudo-label data to enrich training samples. is the true label y of the i-th sample i With the model prediction value The cross entropy loss between .

[0067] After the model is trained once, the model iteration training module 6 reuses the trained enhanced speech recognition model to infer the data from the unlabeled speech data collection module 1 and the pseudo audio data collection module 2 based on speech synthesis. The model performs new inference on these data and generates updated pseudo labels. After the new inference generates pseudo labels, the data selection and enhancement module 5 screens the new speech-text pairs again, removes unreliable labels, and retains data with high consistency and high accuracy. Through this screening process, the quality of pseudo labels in the training data set is gradually improved, and the expanded data set is becoming more and more reliable. After completing the inference and data screening, the module re-adds the new data to the training data set, and the model iteration training module 6 continues to perform a new round of enhanced training on the speech recognition model. This process is repeated through multiple rounds of iterations, and each iteration model is trained on the expanded and optimized training data set, and the performance is further improved. With the improvement of label quality, the quality and scale of the data set are steadily improved, and the speech recognition ability of the model is gradually optimized and converged, eventually forming a high-performance Kangba dialect speech recognition system.

[0068] The present invention proposes a Tibetan Kangba dialect speech recognition system based on self-enhancement learning, which is written in Python. In addition, the present invention can be applied to computer terminals, handheld mobile devices or other forms of mobile devices.

Claims

1. A Tibetan Kham dialect speech recognition system based on self-reinforcement learning, characterized in that: The method comprises an unlabeled speech data collection module (1), a pseudo audio data collection module based on speech synthesis (2), an initial model training module (3), a model inference module (4), a data selection and enhancement module (5), and a model iteration training module (6).

2. A Tibetan Kham dialect speech recognition system based on self-enhancement learning according to claim 1, characterized in that: The unlabeled speech data collection module (1) is used to collect unlabeled Kangba speech data from public network resources; The unlabeled speech data collection module (1) comprises: A web crawler module (11), used for crawling Kangba voice data from the public network; A manual segmentation module (12), connected to the web crawler module (11), for segmenting the crawled voice data at the sentence level; A data preprocessing module (13), connected to the manual segmentation module (12), is used to preprocess the speech data and regularize the data; The data integration module (14) is connected to the data preprocessing module (13) and is used to uniformly mark and store all the data and integrate them into a final unlabeled Kangba dialect speech data set.

3. A Tibetan Kham dialect speech recognition system based on self-enhancement learning according to claim 2, characterized in that: The pseudo audio data collection module (2) based on speech synthesis is used to generate pseudo audio data in the Kham dialect based on Tibetan text by using speech synthesis technology.

4. A Tibetan Kham dialect speech recognition system based on self-enhancement learning according to claim 3, characterized in that: The initial model training module (3) trains an initial Kangba dialect speech recognition model based on an existing manually annotated data set.

5. A Tibetan Kham dialect speech recognition system based on self-reinforcement learning according to claim 4, characterized in that: The model inference module (4) is connected to the unlabeled speech data collection module (1), the speech synthesis-based pseudo audio data collection module (2) and the initial model training module (3), and is used to recognize the unlabeled speech data and the synthesized pseudo speech as Tibetan text based on the trained speech recognition model.

6. A Tibetan Kham dialect speech recognition system based on self-reinforcement learning according to claim 5, characterized in that: The data selection and enhancement module (5) is connected to the model inference module (4) and is used to select reliable speech-text pairs from unlabeled speech data and synthetic pseudo-speech, and expand them together with the manually labeled speech recognition data set into a training data set.

7. A Tibetan Kham dialect speech recognition system based on self-enhancement learning according to claim 6, characterized in that: The model iteration training module (6) is connected to the model reasoning module (4) and the data selection and enhancement module (5) and is used to iteratively enhance the training of speech recognition model performance based on the expanded training data set, and ultimately obtain a Tibetan Kham dialect speech recognition model with high accuracy.

Images