Zhuang language speech recognition method based on pseudo tag generation and model migration

Through pseudo-label generation and model migration technology, the IPA labeling model and acoustic model are trained using unlabeled Zhuang audio data, and the encoder decoder architecture combined with the attention mechanism has solved the problems of insufficient data and lack of dictionaries in Zhuang speech recognition. It realizes efficient training and recognition effects of speech recognition system.

CN118887948APending Publication Date: 2024-11-01GUANGXI UNIV FOR NATITIES
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053578.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

Zhuangyu speech recognition technology faces the problem of insufficient training data of small language models, inability to make full use of unlabeled data and lack of pronunciation dictionary, resulting in poor recognition effect.

Method used

Using a method based on pseudo-label generation and model migration, a pseudo-label is generated for unlabeled data by training an IPA annotation model from audio to phoneme, and an acoustic model is constructed using the Conformer model, and a speech recognition model is performed by combining the encoder decoder architecture of attention mechanism. Training, using a large amount of labelless data for model training.

Benefits of technology

With a small amount of high-quality Zhuangyu voice data and a large amount of unlabeled data, an effective voice recognition system is efficiently trained, reducing hardware requirements and recognition speed and improving recognition accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887948A_ABST
    Figure CN118887948A_ABST
Patent Text Reader

Abstract

The invention provides a Zhuang language speech recognition method based on pseudo tag generation and model migration, and the method comprises the two steps: 1, carrying out the training of an acoustic model, including the generation of IPA pseudo tag data and the training of the acoustic model; and step 2, training of a speech recognition system, including migration of an acoustic model and training of a speech recognition model of an encoder-decoder architecture based on an attention mechanism. According to the method, by means of a pseudo label generation method, the problem that a large amount of unlabeled strong speech audio data cannot be fully utilized is solved, and the acoustic model is trained according to the unlabeled strong speech audio data. And through an acoustic model migration method, the problem that a small amount of labeled data is difficult to train an available Zhuang language speech recognition model is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention provides a Zhuang language speech recognition method based on pseudo-label generation and model migration, belonging to the technical field of speech recognition. Background Art

[0002] Automatic Speech Recognition (ASR) is a technology whose goal is to convert spoken speech signals into corresponding text through a computer program. This technology has been widely used in various applications, including voice assistants, voice search, voice command control, etc.

[0003] Traditional speech recognition methods: In the early days of speech recognition, traditional models such as Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) were mainly used. These models usually require manually designed acoustic features such as Mel-Frequency Cepstral Coefficients (MFCCs) and language models to process speech signals.

[0004] With the rise of deep learning, deep neural networks (DNNs) have been introduced to replace the traditional GMM-HMM structure. DNNs can better capture the abstract representation of speech signals, improve recognition accuracy, and reduce reliance on manually designed features. In recent years, end-to-end speech recognition methods have gradually emerged. This method uses deep learning models, such as long short-term memory networks (LSTM) or convolutional neural networks (CNN), directly from raw audio data to text output, omitting the intermediate steps in traditional speech recognition systems.

[0005] In order to handle long-term temporal dependencies in speech signals, the Connectionist Temporal Classification (CTC) loss function and the attention mechanism are introduced. CTC allows the model to learn the mapping from audio to text, while the attention mechanism helps the model focus on the relevant parts of the input sequence. With the increase of large-scale speech datasets and the improvement of cloud computing capabilities, researchers can train more complex models, namely large speech models. Large speech models are based on a large amount of training data and a network structure with a huge number of parameters, which can improve the performance and generalization ability of the recognition system.

[0006] Zhuang is a minority language in my country, widely distributed in Guangxi Zhuang Autonomous Region, Hunan, Guangdong and other places, with a population of about 700,000. In foreign countries, there are also people who speak Zhuang in Vietnam, Laos, Myanmar and other places. However, due to its small population, Zhuang has not received enough resources and attention, which has led to the relative lag in the research of Zhuang speech recognition. There is a lack of large-scale speech data sets and related research, and traditional speech recognition systems are difficult to achieve ideal results in this context. As a minority language in China, Zhuang has important cultural value. The development of Zhuang speech recognition technology will help protect and inherit the culture of the Yao people and promote national unity. At the same time, improving the application of Zhuang in the fields of intelligent voice assistants and voice translation will bring convenience to the lives of the Yao people, improve their quality of life, provide more intelligent and convenient voice interaction tools for the Zhuang community, and promote the inheritance and development of the language.

[0007] By analyzing the current status of research in the field of Zhuang language speech processing at home and abroad, it can be found that there is a relative lack of research on Zhuang language speech processing. Internationally, most speech recognition research focuses on high-resource languages, while research on low-resource languages ​​is relatively limited. In China, although Zhuang is one of China's minority languages, research on its speech processing is relatively limited, and no systematic technical support has yet been formed.

[0008] With the development of deep learning technology, speech recognition technology for low-resource languages ​​has made significant progress, especially the emergence of large speech models, which provide more abundant means. By utilizing existing advanced technologies and methods and combining the characteristics of Zhuang language, it is possible to achieve efficient recognition of Zhuang language.

[0009] The monolingual speech recognition technology based on acoustic model and connectionist time classification is the current mainstream speech recognition technology. This technology builds an acoustic model based on speech encoder, uses connectionist time classification for model training, and uses beam search for speech recognition.

[0010] The so-called acoustic model based on speech encoder is a neural network model built on the basis of deep neural network. The original audio signal is processed by filter group to obtain the sound information sequence or further processed to obtain the inverse of Mel spectrum as the input of the model. The acoustic model accepts these acoustic information inputs in frames, maps them to high-dimensional feature space through a series of deep neural network layers, and outputs a high-dimensional feature vector sequence. Then, the prediction result aligned with the frame is calculated through the feedforward neural network. In order to align with the target text sequence of speech recognition, the connectionist time classification function is used as the loss function to calculate the gap between the prediction result and the true target, which will guide the learning and training of the model.

[0011] Beam search is a search algorithm for sequence generation problems. Its core idea is to consider multiple possible candidates at each step of sequence generation, and these candidates are organized into a beam. By calculating the score of each candidate, beam search selects a part of the candidates with the highest score as the candidate set for the next step of sequence generation. This process is iterated continuously, and through continuous pruning and screening, the generation result of the entire sequence is finally obtained. Beam search effectively copes with the uncertainty and diversity in sequence generation tasks by balancing computational efficiency and the breadth of the search space.

[0012] However, the training of this method requires a large amount of annotated data in the target language. That is, speech fragment data and its corresponding text. However, because there are fewer users and researchers of small languages, their training data is not abundant. In particular, there is currently no public large amount of annotated training data resources for Zhuang language, the application target of this patent. If there is a lack of sufficient training samples, it is difficult to train a usable automatic speech recognition system based on the single language speech recognition technology based on acoustic model and connectionist time classification, and there is often a high error rate problem.

[0013] In addition, a speech recognition system is built based on the Whisper model. First, the Whisper model is initialized through pre-training parameters, and then a small amount of collected and labeled Zhuang language data is used to fine-tune the model, and finally a usable speech recognition system is obtained.

[0014] However, if this method wants to use a small amount of data to fine-tune the whisper model so that it can recognize Zhuang language, it is necessary to use a version of the whisper model with a larger parameter scale. This will result in higher hardware specifications and slower recognition speed when the speech recognition system is actually deployed. Summary of the invention

[0015] The present invention provides a Zhuang language speech recognition method based on pseudo-label generation and model migration to solve the following technical problems:

[0016] (1) Insufficient training data for minority language models

[0017] As a minority language, Zhuang does not currently have a rich resource of annotated speech data sets like English, Chinese and other languages. According to the information I have collected, there are only a few hours of annotated speech data for Zhuang, and it is not a common speech scenario used in daily life.

[0018] (2) Unable to fully utilize unlabeled Zhuang audio data

[0019] The training of speech recognition models is generally supervised training, which requires the use of labeled data for training. At this time, compared with the difficult-to-obtain labeled data, the easier-to-collect unlabeled data (such as crawling and collating Zhuang language news videos and Zhuang language entertainment videos publicly available on the Internet) is difficult to be effectively used.

[0020] (3) How to use the trained acoustic model

[0021] The current mainstream multilingual speech recognition model generally first trains an acoustic model, which mainly captures the acoustic features of the input audio data, and then combines the speech model and pronunciation dictionary to obtain the final text sequence through WFST decoding. However, Zhuang lacks a pronunciation dictionary and cannot be decoded through WFST. Therefore, how to better justify the acoustic model is a problem that needs to be solved.

[0022] The purpose of the present invention is to address the deficiencies of the prior art and to provide a Zhuang language speech recognition method based on pseudo-label generation and model transfer. The technology generates IPA pseudo-labels for a large amount of unlabeled audio by training a labeling model using a large speech representation model, and then uses a model transfer learning method to make full use of the pseudo-labeled dataset to assist in the training of the speech recognition model.

[0023] The technical solution for realizing the present invention is:

[0024] A Zhuang language speech recognition method based on pseudo-label generation and model transfer includes two steps:

[0025] The first step is to train the acoustic model, including generating IPA pseudo-labels for unlabeled Zhuang audio data and training the acoustic model.

[0026] It includes the following sub-steps:

[0027] Train an audio-to-phoneme IPA annotation model to generate correct IPA pronunciation annotations; the training process of the model: use the Phonetisaurus tool to generate IPA annotations for multi-national audio data in commonvoice11; obtain audio-phoneme pair training data from multiple countries; use the training data to select the pre-trained Hubert model as the base model of the IPA annotation model, and fine-tune the training based on the CTC loss function to obtain a multi-national common audio-to-phoneme annotation model; use the phoneme annotation model to generate IPA annotations for Zhuang unlabeled audio; and based on the Zhuang audio dataset obtained by the annotation, use the Conformer model to build an acoustic model and train it.

[0028] The specific training process of the phoneme annotation model is as follows: let the input be a speech feature sequence X of T frames after downsampling = {x 1 ,x2 ,…,x T};

[0029] After Hubert encoding, the encoded feature vector sequence is obtained:

[0030] H enc =Hubert(X)

[0031] The loss function calculation formula is:

[0032]

[0033] Among them, Y is the correct phoneme sequence for model training, and π is a possible alignment method of Y obtained based on the CTC decoding method, namely the alignment path. The CTC decoding method allows the output of blank symbols during the recognition process, and all predicted sequences that are consistent with Y after removing the blank symbols are called possible alignment paths; Represents mapping the label sequence Y to the set of all possible alignments, and P(π|X) is the conditional probability of the alignment path π given the input sequence X;

[0034] After obtaining the Zhuang speech-IPA pair dataset, the Conformer model structure is used as the encoder, and the Zhuang speech-IPA pair dataset is used as training data to train the acoustic model; the input of the acoustic model is the fbank feature extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

[0035] The second step is the training of the speech recognition system, including the migration of the acoustic model and the training of the speech recognition model based on the encoder-decoder architecture of the attention mechanism.

[0036] The training of the speech recognition model includes the following sub-steps:

[0037] The speech recognition model uses an encoder-decoder architecture based on the attention mechanism. The encoder uses the same Conformer architecture as the acoustic model structure, and uses the pre-trained acoustic model parameters to initialize the encoder structure, so that the acoustic model is migrated and integrated into the complete speech recognition model. The decoder uses the Transformer Decoder structure and is randomly initialized.

[0038] Use labeled Zhuang language data, namely Zhuang audio-Zhuang text pairs, to train a speech recognition model based on the Conformer-Transformer architecture.

[0039] The encoder-decoder architecture includes an encoder and a decoder; the encoder generates a higher-level representation feature vector based on the input fbank feature sequence, and the feature vector generated by the encoder can better capture the key information in the audio signal; the decoder receives the feature vector output by the encoder and gradually generates a text sequence; when generating a text sequence, the decoder considers the language model and prior knowledge, as well as part of the text generated previously; at the same time, based on the attention mechanism, the decoder can pay attention to the decoder information that is more effective for the current time step when generating at each time step; during the training process, the model receives known audio-text pairing data; by minimizing the gap between the predicted text and the actual text, the parameters of the model are adjusted so that the model can convert the audio signal into the corresponding text.

[0040] The technical solution of the present invention has the following technical effects:

[0041] (1) A speech recognition system with good results can be efficiently trained with only a small amount of high-quality Zhuang speech datasets and a large amount of unlabeled Zhuang speech data.

[0042] (2) The resulting speech recognition system does not use a pre-trained model with too many parameters, thus ensuring the speed of speech recognition and reducing the hardware requirements for actual applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is the phoneme annotation model training process of the present invention;

[0044] Figure 2 It is a structural diagram of the Conformer model of the present invention;

[0045] Figure 3 is the phoneme annotation model structure of the present invention;

[0046] Figure 4 This is the speech recognition model structure of the present invention. DETAILED DESCRIPTION

[0047] The specific technical solution of the present invention is described in conjunction with the accompanying drawings.

[0048] The present invention mainly includes two steps. The first step is to train the acoustic model, including the generation of IPA pseudo-label data and the training of the acoustic model. The second step is to train the speech recognition system, including the migration of the acoustic model and the training of the speech recognition model based on the encoder-decoder architecture of the attention mechanism. Specifically:

[0049] The first step is the training of the acoustic model.

[0050] First, in order to utilize a large amount of unlabeled Zhuang audio data, it is necessary to train an audio-to-phoneme annotation model to generate the most accurate IPA pronunciation annotations for it. The training process of this model is as follows: Figure 1 As shown in the figure, the Phonetisaurus tool is used to generate IPA annotations for the multinational data in commonvoice11. The Phonetisaurus tool can generate annotations from morphemes to phonemes by loading the annotation model of the corresponding language. The annotation model used can be downloaded from https: / / github.com / uiuc-sst / g2ps. Finally, audio-phoneme pair training data of multiple countries can be obtained. Using this training data, the pre-trained Hubert model is selected as the encoder, and decoding is performed based on the CTC function to obtain the required multinational common audio to phoneme annotation model.

[0051] The specific training process is as follows: Assume that the input is a speech feature sequence X = {x 1 ,x 2 ,…,x T}

[0052] After Hubert encoding, the encoded feature vector sequence is obtained:

[0053] H enc =Hubert(X)

[0054] The loss function calculation formula is:

[0055]

[0056] Among them, Y is the target correct phoneme sequence for model training, π is a possible alignment method (alignment path) of Y, where, It represents mapping the label sequence Y to the set of all possible alignments, and P(π|X) is the conditional probability of the alignment path π given the input sequence X.

[0057] Using the trained multi-national universal phoneme annotation model, a large number of unlabeled Zhuang audios were annotated to obtain a Zhuang speech-IPA pair dataset. Then, the Conformer model structure was used as the encoder, and the Zhuang speech-IPA pair dataset was used as training data to train the Zhuang acoustic model; the model structure is as follows Figure 3 As shown in the figure, the input of the acoustic model is the fbank feature extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

[0058] The second step is to train the speech recognition model.

[0059] Speech recognition model selection Figure 4 The encoder-decoder architecture based on the attention mechanism is shown in the figure. In the encoder part, the Conformer architecture with the same acoustic model structure is used, and the encoder structure is initialized with the pre-trained acoustic model parameters, thereby successfully integrating the acoustic model migration into the complete speech recognition model.

[0060] Using labeled Zhuang language data (Zhuang audio-Zhuang text pairs), a speech recognition model based on the Conformer-Transformer architecture is trained. The encoder-decoder architecture mainly consists of two parts: the encoder and the decoder. The encoder generates a higher-level representation feature vector based on the input fbank feature sequence. The feature vector generated by the encoder can better capture the key information in the audio signal. The decoder receives the feature vector output by the encoder and gradually generates a text sequence. When generating a text sequence, the decoder considers the language model and prior knowledge, as well as part of the previously generated text; at the same time, based on the attention mechanism, the decoder can better pay attention to the decoder information that is more effective for the current time step when generating at each time step. During the training process, the model inputs audio-text pairing data. The training goal is to minimize the gap between the predicted text and the actual text (usually using the cross entropy loss function). The parameters of the model will be adjusted so that the model can more accurately convert the audio signal into the corresponding text.

[0061] The present invention generates IPA phoneme annotations with the help of Hubert, making full use of unannotated Zhuang language audio data.

[0062] In order to enable the acoustic model to better capture the pronunciation features in the audio, and to enhance the data for multi-language mixed training, the IPA annotation sequence is often used as the training target of the acoustic model. However, due to the lack of basic research work related to the Zhuang language, there is no available Zhuang IPA annotation model. In order to solve this problem, the present invention uses the pre-trained Hubert model, and by using the currently available multi-national audio-phoneme annotation data, fine-tunes the Hubert model so that it can have a passing phoneme annotation capability. Then, the fine-tuned Hubert model is used to perform pronunciation annotation for a large number of unannotated Zhuang audios. Then, the Conformer model is used to construct an acoustic model, and the IPA-annotated Zhuang audio data set obtained by annotation is used to perform acoustic model training.

[0063] The present invention is also based on model migration and utilizes a trained acoustic model to reduce the difficulty of training a speech recognition model.

[0064] Generally, after obtaining a trained acoustic model, it will be combined with a speech model and a pronunciation dictionary, and the final text sequence will be obtained through WFST decoding. However, the IPA annotation of Zhuang language is obtained through a trained annotation model, and there is no available pronunciation dictionary. Therefore, it is impossible to directly use the acoustic model to obtain the final text sequence. In order to use the pronunciation dictionary, one approach is to connect a linear neural network layer on the basis of the acoustic model, use the labeled data and CTC loss function to fine-tune the model, and obtain a speech recognition model. However, in this approach, the external linear layer is too simple and cannot make good use of the output information of the acoustic model. At the same time, the training target of the acoustic model is phonemes, and fine-tuning is directly performed on this basis. Because the training tasks are inconsistent, it often does not achieve very good results. The present invention uses a method of migrating the acoustic model as the encoder part of the encoder-decoder speech recognition model, and achieves the goal of using a trained acoustic model to reduce the difficulty of speech recognition model training.

[0065] By using the acoustic model as an encoder, we can fully utilize the acoustic model's ability to capture the pronunciation features in the sound signal. With an additional decoder, with the decoder's strong decoding ability, we can fully utilize the pronunciation information output by the encoder, take into account the language model and prior knowledge, and some of the previously generated text.

Claims

1. A Zhuang language speech recognition method based on pseudo-label generation and model transfer, characterized in that: There are two steps: The first step is to train the acoustic model, including generating IPA pseudo-labels for unlabeled Zhuang audio data and training the acoustic model. The second step is the training of the speech recognition system, including the migration of the acoustic model and the training of the speech recognition model based on the encoder-decoder architecture of the attention mechanism.

2. According to claim 1, a Zhuang language speech recognition method based on pseudo-label generation and model transfer is characterized in that: The first step, training of the acoustic model, specifically includes the following sub-steps: Train an audio-to-phoneme IPA annotation model to generate correct IPA pronunciation annotations; the training process of the model: use the Phonetisaurus tool to generate IPA annotations for multi-national audio data in commonvoice11; obtain audio-phoneme pair training data from multiple countries; use the training data to select the pre-trained Hubert model as the base model of the IPA annotation model, and fine-tune the training based on the CTC loss function to obtain a multi-national common audio-to-phoneme annotation model; use the phoneme annotation model to generate IPA annotations for Zhuang unlabeled audio; and based on the Zhuang audio dataset obtained by the annotation, use the Conformer model to build an acoustic model and train it.

3. The Zhuang language speech recognition method based on pseudo-label generation and model transfer according to claim 2 is characterized in that: The specific training process of the phoneme annotation model is as follows: let the input be a speech feature sequence X = {x1, x2, ..., x T }; After Hubert encoding, the encoded feature vector sequence is obtained: H enc =Hubert(X) The loss function calculation formula is: Among them, Y is the correct phoneme sequence for model training, and π is a possible alignment method of Y obtained based on the CTC decoding method, namely the alignment path. The CTC decoding method allows the output of blank symbols during the recognition process, and all predicted sequences that are consistent with Y after removing the blank symbols are called possible alignment paths; Represents mapping the label sequence Y to the set of all possible alignments, and P(π|X) is the conditional probability of the alignment path π given the input sequence X; After obtaining the Zhuang speech-IPA pair dataset, the Conformer model structure is used as the encoder, and the Zhuang speech-IPA pair dataset is used as training data to train the acoustic model; the input of the acoustic model is the fbank feature extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

4. The Zhuang language speech recognition method based on pseudo-label generation and model transfer according to claim 1 is characterized in that: The second step, training of the speech recognition model, includes the following sub-steps: The speech recognition model uses an encoder-decoder architecture based on the attention mechanism. The encoder uses the same Conformer architecture as the acoustic model structure, and uses the pre-trained acoustic model parameters to initialize the encoder structure, so that the acoustic model is migrated and integrated into the complete speech recognition model. The decoder uses the Transformer Decoder structure and is randomly initialized. Use labeled Zhuang language data, namely Zhuang audio-Zhuang text pairs, to train a speech recognition model based on the Conformer-Transformer architecture.

5. The Zhuang language speech recognition method based on pseudo-label generation and model transfer according to claim 4 is characterized in that: The encoder-decoder architecture includes an encoder and a decoder; the encoder generates a higher-level representation feature vector based on the input fbank feature sequence, and the feature vector generated by the encoder can better capture the key information in the audio signal; The decoder receives the feature vector output by the encoder and gradually generates a text sequence. When generating a text sequence, the decoder considers the language model and prior knowledge, as well as some of the previously generated text. At the same time, based on the attention mechanism, the decoder can pay attention to the decoder information that is more effective for the current time step when generating at each time step. During training, the model receives known audio-text pairs; the model’s parameters are adjusted so that it can convert the audio signal into its corresponding text by minimizing the gap between the predicted text and the actual text.