End-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology

Abstract

The invention discloses an end-to-end efficient Chinese-Yao language speech recognition system based on a cold fusion technology. The system constructs and updates a Chinese-Yao language dictionary through a word segmentation device training module, and performs word segmentation and regularization processing on a text through a data arrangement module. The core lies in a voice model training module, the module utilizes an end-to-end ASR model of an attention mechanism and integrates a large language model through a cold fusion technology for auxiliary training, and the recognition efficiency and accuracy are remarkably improved. And the model evaluation module comprehensively evaluates the quality of the identification result, including the accuracy rate, the recall rate and the F1 score. According to the invention, fine tuning and integrated language model decoding are carried out on the pre-trained language model, a voice recognition model with a good effect can be obtained under the condition that only a small amount of voice data is needed, and advantages of different models can be integrated through a cold fusion technology integrated large language model decoding method. And the identification capability and robustness of the model are further improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to an end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology. Background Art

[0002] Speech recognition technology is an automatic speech conversion technology that combines signal processing, machine learning and natural language processing to achieve the conversion from spoken speech to text. It has undergone the evolution from model-based, statistical and neural network-based. At present, speech recognition technology based on neural networks has been widely used and become the mainstream technology due to its high performance, high accuracy and high efficiency. Traditional speech recognition requires multiple modules including feature extraction, pronunciation dictionary, acoustic model, language model, etc. Speech recognition technology based on neural networks consists of three basic components:

[0003] Acoustic model: Typical systems are mostly modeled based on the first-order hidden Markov model.

[0004] Language model: The language model models the language that the system is targeting. In theory, various language models including regular languages ​​and context-free grammars can be used as language models, but currently various systems generally use the statistical n-gram model and its variants.

[0005] Pronunciation dictionary: The pronunciation dictionary contains the vocabulary set that the system can handle and its pronunciation. The pronunciation dictionary actually provides a mapping between the acoustic model modeling unit and the language model modeling unit.

[0006] The language model can be based on grammatical rules or statistical methods.

[0007] The language model based on grammar rules uses the language structure and grammar specifications defined by the implementation. Such grammar rules can be designed manually or extracted from linguistic knowledge. This language model can provide stronger prior knowledge to ensure that the text generated by the model conforms to the grammar and semantic rules of a specific field. This method has high requirements on the accuracy of professional terms and can generally only constrain sentences within a limited field.

[0008] The language model based on statistical methods uses statistical probability to represent the relationship between vocabulary and language structure. It processes a large amount of text corpus to obtain the probability distribution of a given word sequence and finally calculates the combination probability. Common statistical language models such as the n-gram model can estimate the probability of the next word based on the previous n-1 words. Statistical language models are widely used in many fields of natural language processing, including language recognition, machine translation, text proofreading, etc.

[0009] Traditional speech recognition models need to be composed of multiple parts, namely acoustic models, language models, pronunciation dictionaries, etc. The components need to be trained separately, and the quality of the entire model is often affected by poor training results of some components. The end-to-end speech recognition model integrates the three major components of the traditional speech recognition system, namely the acoustic model, language model, and pronunciation dictionary, into a network model, directly realizing the conversion from input speech to output text. This reduces the possibility of error propagation.

[0010] Even though the current ASR model size and the scale of speech supervision datasets have increased, problems such as domain mismatch and limited text corpus utilization still exist when using ASR models. Therefore, an end-to-end efficient Chinese-Yao speech recognition system based on cold fusion technology is urgently needed. Summary of the invention

[0011] The technical problem to be solved by the present invention is to overcome the above technical defects and provide an end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology.

[0012] In order to solve the above problems, the technical solution of the present invention is to include the following modules:

[0013] A word segmenter training module, wherein the word segmenter training module is used to construct a Chinese-Yao language dictionary and train a word segmenter;

[0014] A data sorting module, the data sorting module is used to use the word segmenter to segment the Chinese-Yao bilingual text and perform text regularization processing;

[0015] A speech model training module, which is used for end-to-end automatic speech recognition model training based on an attention mechanism and integrates a large language model for auxiliary training through cold fusion technology;

[0016] A model evaluation module is used to evaluate the quality of speech recognition decoding results.

[0017] Furthermore, the data sorting module includes a text regularization submodule, which is used to perform operations such as uppercase and lowercase conversion and punctuation processing on the Chinese-Yao bilingual text.

[0018] Further, the cold fusion submodule is implemented by the following steps:

[0019] S1. In the pre-training stage, the large language model and the model component based on the attention mechanism are pre-trained in the projection layer and the softmax layer.

[0020] S2, in the parameter freezing stage, freezing the parameters in the large language model;

[0021] S3. In the auxiliary training stage, the large language model with frozen parameters is integrated into the decoder of the ASR model as an internal model, and the auxiliary ASR model is trained from scratch.

[0022] Furthermore, the system also includes a model optimization submodule, which is used to optimize model performance by adjusting model structure or parameters.

[0023] Furthermore, the calculation formula of the cold fusion submodule is as follows:

[0024]

[0025] Furthermore, the word segmenter training module also includes:

[0026] A dictionary construction submodule, which is used to construct a Chinese-Yao language dictionary by using manual annotation, literature review or crawler technology;

[0027] The dictionary updating submodule is used to regularly update and expand the dictionary to maintain the timeliness and practicality of the dictionary.

[0028] Furthermore, the speech model training module includes:

[0029] A pre-trained language model fine-tuning submodule, wherein the pre-trained language model fine-tuning submodule is used to select a suitable pre-trained language model for fine-tuning;

[0030] A cold fusion submodule is used to integrate the large language model with the end-to-end ASR model based on the attention mechanism through cold fusion, wherein the parameters of the large language model are frozen after the projection layer and the softmax layer are combined with the model components based on the attention mechanism, so as to assist the end-to-end ASR model to be trained from scratch.

[0031] Furthermore, the model evaluation module also includes:

[0032] An accuracy calculation submodule, wherein the accuracy calculation submodule is used to calculate the accuracy of the model recognition result;

[0033] A recall rate calculation submodule, wherein the recall rate calculation submodule is used to calculate the recall rate of the model recognition result;

[0034] The F1 score calculation submodule is used to calculate the F1 score of the model recognition result.

[0035] Furthermore, the calculation formula of the speech model training module is as follows:

[0036]

[0037] The advantages of the present invention compared with the prior art are:

[0038] 1. The present invention provides an end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology. By fine-tuning the pre-trained language model and decoding the integrated language model, a speech recognition model with better effect can be obtained with only a small amount of speech data;

[0039] 2. The present invention provides an end-to-end efficient Chinese-Yao speech recognition system based on cold fusion technology. By integrating a large language model decoding method through cold fusion technology, the advantages of different models can be combined, further improving the recognition ability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a system diagram of the present invention.

[0041] Figure 2 It is an end-to-end ASR model diagram based on the attention mechanism in the speech model training module of the present invention.

[0042] Figure 3 This is a model diagram of the cold fusion submodule of the present invention. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] The end-to-end efficient Chinese-Yao speech recognition system based on cold fusion technology includes the following modules:

[0046] A word segmenter training module, which is used to construct a Chinese-Yao language dictionary and train a word segmenter;

[0047] A data sorting module is used to use a word segmenter to segment the Chinese-Yao bilingual text and perform text regularization processing;

[0048] Speech model training module: The speech model training module is used for end-to-end automatic speech recognition model training based on the attention mechanism, and integrates a large language model for auxiliary training through cold fusion technology;

[0049] Model evaluation module,The model evaluation module is used to evaluate the quality of speech recognition decoding results.

[0050] Furthermore, the word segmenter training module also includes:

[0051] A dictionary construction submodule is used to construct a Chinese-Yao language dictionary by using manual annotation, literature review or crawler technology;

[0052] Dictionary update submodule,The dictionary update submodule is used to regularly update and expand the dictionary to maintain the timeliness and practicality of the dictionary.

[0053] Furthermore, the data sorting module includes a text regularization submodule, which is used to perform operations such as uppercase and lowercase conversion and punctuation processing on the Chinese-Yao bilingual text.

[0054] Furthermore, the speech model training module includes:

[0055] A pre-trained language model fine-tuning submodule is used to select a suitable pre-trained language model for fine-tuning;

[0056] Cold fusion submodule,The cold fusion submodule is used to integrate the large language model with the end-to-end ASR model based on the attention mechanism through cold fusion, where the parameters of the large language model are frozen after the projection layer and softmax layer are combined with the model components based on the attention mechanism, assisting the end-to-end ASR model to be trained from scratch.

[0057] Furthermore, the cold fusion submodule is implemented by the following steps:

[0058] S1. In the pre-training stage, the large language model and the model component based on the attention mechanism are pre-trained in the projection layer and the softmax layer.

[0059] S2, in the parameter freezing stage, freezing the parameters in the large language model;

[0060] S3. In the auxiliary training stage, the large language model with frozen parameters is integrated into the decoder of the ASR model as an internal model, and the auxiliary ASR model is trained from scratch.

[0061] Furthermore, the model evaluation module also includes:

[0062] The accuracy calculation submodule is used to calculate the accuracy of the model recognition result;

[0063] A recall rate calculation submodule, which is used to calculate the recall rate of the model recognition result;

[0064] The F1 score calculation submodule is used to calculate the F1 score of the model recognition result.

[0065] Furthermore, the system also includes a model optimization submodule, which is used to optimize model performance by adjusting model structure or parameters.

[0066] Furthermore, the calculation formula of the speech model training module is as follows:

[0067]

[0068] Furthermore, the calculation formula of the cold fusion submodule is as follows:

[0069]

[0070] refer to Figures 1 to 3 As shown in the figure, in order to train the word segmenter, it is necessary to build a dictionary first. In the process of building the Chinese-Yao dictionary, we can use methods such as manual annotation and literature review to build the dictionary, or we can consider using crawler technology to collect more words from social media, forums, audio and video and other channels. Of course, if we can find an existing bilingual data set, the efficiency will be higher. After completing the construction of the dictionary, we can also regularly update and expand the dictionary, add new words and expressions to maintain the timeliness and practicality of the dictionary, and use the obtained Chinese-Yao word segmenter to segment the Chinese-Yao bilingual text pairs and convert them into data acceptable to the model. At the same time, a set of text regularization rules suitable for Yao language can be formulated, including case conversion, punctuation processing, etc. Through continuous adjustment and optimization, the accuracy of the word segmenter is improved, and a suitable pre-trained language model is selected for fine-tuning. Here we use an end-to-end ASR model based on the attention mechanism (Attention-based Encoder-Decode, AED). Under the action of the attention mechanism, the AED model models the entire sentence. Compared with the connectionist temporal classification model (CTC) and the recurrent neural network transformer (RNN-T) model, the AED model does not require the output sequence and the input sequence to be aligned in time order, so it is easier to train and implement. Its calculation formula is:

[0071]

[0072] The encoder module is based on the input acoustic features x 1:T Generating high-level representations The attention module uses the output of the encoder and the output of the decoder at the previous time step Calculate the context vector c t According to the context vector c t and the target label y at the previous time step t-1 Calculate and get the hidden state output of the decoder Finally, the final output y is obtained through mapping and softmax layer t .

[0073] When training the speech model, we integrate the existing large language model into the speech model. Here we use the cold fusion method. Cold fusion proposes to first pre-train the LM and attention-based model components in the projection layer and softmax layer combination, and then freeze the parameters in the LM to assist the end-to-end ASR model to start training from scratch. The calculation formula of the cold fusion model is as follows:

[0074]

[0075] in is the logit output of the external LM, and [;] represents the concatenation of two vectors. We can use a variety of large language models currently open source, such as LLaMA

[11] 、G l aM

[12] , Bloom

[13] Etc., integrate them with the speech model and compare the optimization effects of different large language models on the speech model.

[0076] Evaluate the quality of speech recognition decoding results. Use multiple indicators such as accuracy, recall, F1 score, etc. to comprehensively evaluate model performance. We select a benchmark model with stable performance as the comparison object and analyze the improvements of the new model. The comparison is not limited to the overall performance, but also analyzes the performance in specific scenarios and details to find out the advantages of the model and areas for improvement.

[0077] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0079] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. An end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology, characterized by: Includes the following modules: A word segmenter training module, wherein the word segmenter training module is used to construct a Chinese-Yao language dictionary and train a word segmenter; A data sorting module, the data sorting module is used to use the word segmenter to segment the Chinese-Yao bilingual text and perform text regularization processing; A speech model training module, which is used for end-to-end automatic speech recognition model training based on an attention mechanism and integrates a large language model for auxiliary training through cold fusion technology; A model evaluation module is used to evaluate the quality of speech recognition decoding results.

2. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 1 is characterized in that: The word segmenter training module also includes: A dictionary construction submodule, which is used to construct a Chinese-Yao language dictionary by using manual annotation, literature review or crawler technology; The dictionary updating submodule is used to regularly update and expand the dictionary to maintain the timeliness and practicality of the dictionary.

3. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 1 is characterized in that: The data sorting module includes a text regularization submodule, which is used to perform operations such as uppercase and lowercase conversion and punctuation processing on the Chinese-Yao bilingual text.

4. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 1 is characterized in that: The speech model training module includes: A pre-trained language model fine-tuning submodule, wherein the pre-trained language model fine-tuning submodule is used to select a suitable pre-trained language model for fine-tuning; A cold fusion submodule is used to integrate the large language model with the end-to-end ASR model based on the attention mechanism through cold fusion, wherein the parameters of the large language model are frozen after the projection layer and the softmax layer are combined with the model components based on the attention mechanism, so as to assist the end-to-end ASR model to be trained from scratch.

5. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 4 is characterized in that: The cold fusion submodule is implemented by the following steps: S1. In the pre-training stage, the large language model and the model component based on the attention mechanism are pre-trained in the projection layer and the softmax layer. S2, in the parameter freezing stage, freezing the parameters in the large language model; S3. In the auxiliary training stage, the large language model with frozen parameters is integrated into the decoder of the ASR model as an internal model, and the auxiliary ASR model is trained from scratch.

6. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 1 is characterized in that: The model evaluation module also includes: An accuracy calculation submodule, wherein the accuracy calculation submodule is used to calculate the accuracy of the model recognition result; A recall rate calculation submodule, wherein the recall rate calculation submodule is used to calculate the recall rate of the model recognition result; The F1 score calculation submodule is used to calculate the F1 score of the model recognition result.

7. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 1 is characterized by: The system also includes a model optimization submodule, which is used to optimize model performance by adjusting model structure or parameters.

8. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 1 is characterized in that: The calculation formula of the speech model training module is as follows:

9. The end-to-end efficient Chinese-Yao language speech recognition system based on cold fusion technology according to claim 4 is characterized in that: The calculation formula of the cold fusion submodule is as follows:

Images