Automobile industry term speech recognition method and device, and storage medium

Abstract

The invention relates to the field of speech recognition, and discloses an automobile industry term speech recognition method and device, and a storage medium. The method comprises the following steps: constructing a Chinese word-phone pronunciation dictionary and an automobile terminology BPE-phone pronunciation dictionary; training the initialized basic speech recognition model by using Chinese speech data; and carrying out parameter weight migration from the basic speech recognition model, and training the target speech recognition model by using the Chinese speech data on the migrated model. According to the embodiment of the invention, a voice recognition system with a good effect can be obtained through efficient training under the condition that only a small amount of high-quality automobile terminology voice data sets and a large amount of unlabeled automobile terminology voice data are needed. The finally obtained speech recognition system does not use a pre-training model with overlarge parameter quantity, so that the speech recognition speed is ensured, and the requirement on hardware in practical application is reduced.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a method and device for speech recognition of automotive industry terminology, and a storage medium. Background Art

[0002] The monolingual speech recognition technology based on acoustic model and connectionist time classification is the current mainstream speech recognition technology. This technology builds an acoustic model based on speech encoder, uses connectionist time classification for model training, and uses beam search for speech recognition.

[0003] The so-called acoustic model based on speech encoder is a neural network model built on the basis of deep neural network. The original audio signal is processed by filter group to obtain the sound information sequence or further processed to obtain the inverse of Mel spectrum as the input of the model. The acoustic model accepts these acoustic information inputs in frames, maps them to high-dimensional feature space through a series of deep neural network layers, and outputs a high-dimensional feature vector sequence. Then, the prediction result aligned with the frame is calculated through the feedforward neural network. In order to align with the target text sequence of speech recognition, the connectionist time classification function is used as the loss function to calculate the gap between the prediction result and the true target, which will guide the learning and training of the model.

[0004] Beam search is a search algorithm for sequence generation problems. Its core idea is to consider multiple possible candidates at each step of sequence generation, and these candidates are organized into a beam. By calculating the score of each candidate, beam search selects a part of the candidates with the highest score as the candidate set for the next step of sequence generation. This process is iterated continuously, and the generation result of the entire sequence is finally obtained through continuous pruning and screening. Beam search effectively copes with the uncertainty and diversity in sequence generation tasks by balancing computational efficiency and the breadth of the search space.

[0005] The training of this method requires a large amount of annotated data in the target language. That is, speech fragment data and its corresponding text. However, the automotive industry terminology has few users and researchers, so its training data is not abundant. There are currently no large amounts of publicly available annotated training data resources. If there are not enough training samples, it is difficult to train a usable automatic speech recognition system based on the monolingual speech recognition technology based on acoustic models and connectionist time classification, and there is often a high error rate. In addition, a pronunciation dictionary of the target language is also required. The pronunciation of professional terms sometimes has dialects, so the production of a pronunciation dictionary requires expert knowledge to guide it, and it is very time-consuming and labor-intensive to complete it manually. Summary of the invention

[0006] The purpose of the present invention is to provide a method and device for speech recognition of automotive industry terminology, and a storage medium, so as to overcome the defects of the prior art in that the error rate is high and the time and energy consumption is very great.

[0007] To achieve this object, the present invention adopts the following technical solutions: A method for speech recognition of automotive industry terminology, comprising: Construct Chinese word-phone pronunciation dictionary and automotive professional terminology BPE-phone pronunciation dictionary; Use Chinese speech data to train the initialized basic speech recognition model; The parameter weights are transferred from the basic speech recognition model, and the target speech recognition model is trained using Chinese speech data on the transferred model.

[0008] Optionally, the method for constructing the Chinese word-phone pronunciation dictionary includes: Generate a Chinese word list: perform word statistics from a Chinese text data set to generate a Chinese word list; the text data set includes a training set, a test set, and a validation set; Pronunciation of Chinese words: Generate automotive professional terminology phonetic IPA annotations for Chinese words to obtain a Chinese word-phoneme pronunciation dictionary.

[0009] Optionally, the method for constructing the automobile professional terminology BPE-phone pronunciation dictionary includes: Step 1: Set the maximum number of word segmentation dictionaries and initialize a dictionary; Step 2: Cut all the text in the corpus into single characters and add them to the dictionary, and also add special characters to the dictionary; Step 3: For the corpus that has been cut into characters, globally count the frequency of two consecutive character combinations in one round; Step 4: Take the combination with the highest frequency, merge the two characters into a whole, add this whole to the dictionary, and replace the two characters in the corpus with this new whole at the same time, treating it as a word; Step 5: Repeat step 3 and step 4 until the maximum number of word segmentation dictionaries is reached or no more merging is possible; Step 6: Generate a word segmentation encoding model file from the final dictionary, and then use this word segmentation dictionary to segment and encode words to generate a BPE list for Mandarin; Step 7: Perform IPA phonetic notation on the BPE list to form a BPE-phone pronunciation dictionary for automotive professional terms.

[0010] Optionally, the training method of the basic speech recognition model includes: Assume that the input is a speech feature sequence X = {x1,x2,…,x T}, After Hubert encoding, the encoded feature vector sequence is obtained: , The loss function calculation formula is: ; Among them, Y is the target correct phoneme sequence for model training, is a possible alignment of Y, represents the set of all possible alignments that map Y to. is the alignment path given the input sequence X The conditional probability of .

[0011] Optionally, the training method of the basic speech recognition model further includes: After obtaining the automotive professional terminology speech-IPA pair dataset, the Conformer model structure is used as the encoder, and the multi-national speech phoneme pair data including the automotive professional terminology speech-IPA pair data is used as training data to train the acoustic model; the input of the acoustic model is the fbank feature extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

[0012] Optionally, the migrating parameter weights from the basic speech recognition model and using Chinese speech data on the migrated model to train the target speech recognition model includes: The target speech recognition model selects an encoder-decoder architecture based on the attention mechanism; The encoder part uses the same Conformer architecture as the acoustic model structure and uses the pre-trained acoustic model parameters to initialize the encoder structure, thereby successfully migrating the acoustic model into the complete target speech recognition model.

[0013] An automotive industry terminology speech recognition device, comprising a memory and a processor; The memory is used to store instructions; The processor is used to execute the instructions in the memory to implement any one of the above automotive industry terminology speech recognition methods.

[0014] A computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute any one of the above-mentioned automotive industry terminology speech recognition methods.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The embodiment of the present invention can efficiently train a speech recognition system with good effect while only requiring a small amount of high-quality automotive professional terminology speech data set and a large amount of unlabeled automotive professional terminology speech data. The speech recognition system finally obtained does not use a pre-training model with too many parameters, which ensures the speed of speech recognition and reduces the hardware requirements in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flow chart of a method for constructing a BPE-phone pronunciation dictionary of automotive professional terms provided in an embodiment of the present invention.

[0018] Figure 2 This is a structural diagram of the Conformer model provided in an embodiment of the present invention.

[0019] Figure 3 A diagram showing the structure of a phoneme annotation model provided by an embodiment of the present invention; Figure 4 An encoder-decoder structure based on an attention mechanism is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] For ease of understanding, the embodiments of the present invention provide the following definitions of abbreviations and key terms: ASR: Automatic Speech Recognition, automatic speech recognition, refers to a technology that can automatically recognize the corresponding text from audio information Basic model: refers to the model trained by source data in transfer learning.

[0022] Target model: refers to the model trained by target data in transfer learning.

[0023] Source data: refers to the data required for pre-training of the model, and in this patent specifically refers to Thai speech data.

[0024] Target data: refers to the data required for parameter optimization of the target model, specifically Mandarin speech data in this patent Phoneme: Phoneme is the smallest unit of speech, and is the smallest sound segment in speech that can distinguish the meaning of a word. Each language has its own phoneme system, and different phonemes are combined into words.

[0025] BPE: Byte Pair Encoder (BPE) is a data compression algorithm used to implement variable-length subwords in a fixed-size vocabulary.

[0026] Subword: A subword is a text unit that is between a word and a character.

[0027] Acoustic model: The task of the acoustic model is to predict the corresponding phoneme, word or other speech unit according to the input speech signal. In the present invention, the task of the acoustic model is to predict the phoneme corresponding to the speech signal.

[0028] Pronunciation dictionary: A pronunciation dictionary is a vocabulary that contains pronunciation information of words. It maps words to their phoneme sequences or other units of pronunciation to describe how to pronounce these words correctly. In applications such as speech recognition and speech synthesis, pronunciation dictionaries are crucial for converting text to corresponding phoneme sequences or converting phoneme sequences back to text.

[0029] BPE pronunciation dictionary: contains the pronunciation information of BPE subwords, which maps BPE subwords to corresponding phoneme sequences or other pronunciation units to describe how to pronounce these words correctly.

[0030] CTC function: Connectionist Temporal Classification (CTC) is a loss function used for sequence learning tasks (such as speech recognition, handwriting recognition, etc.). It allows incomplete alignment between the input sequence and the output sequence, that is, the length of the input sequence can be inconsistent with the length of the output sequence, and there can be repeated characters and blank characters in the output sequence. The goal of CTC is to maximize the match between the conditional probability of a given input sequence and the corresponding output sequence.

[0031] Fbank feature: FBank feature is a commonly used feature representation method in the field of speech signal processing, also known as filter bank features. FBank feature performs a series of filter bank filtering operations on the speech signal and performs logarithmic operations on the output of each filter bank to finally obtain the feature vector of each time window.

[0032] In view of the shortcomings of the prior art, the embodiment of the present invention provides a BPE-phone migration automobile professional terminology recognition technology, which mainly includes three steps: The first step is to construct a pronunciation dictionary, including a Chinese word-phone pronunciation dictionary and a BPE-phone pronunciation dictionary for automotive professional terms; The second step is to train the basic speech recognition model, using Chinese speech data to train the initialized model; The third step is to train the target speech recognition model, migrate the parameter weights from the basic model, and use Chinese speech data for training on the migrated model.

[0033] The above three steps are described in detail below.

[0034] The first step is the construction of a pronunciation dictionary.

[0035] 1) The construction of the Chinese word-phone pronunciation dictionary is as follows: 1-1) Generate a Chinese word list: Perform word statistics from the Chinese text dataset (including training set, test set, and validation set) to generate a Chinese word list.

[0036] 1-2) Pronunciation of Chinese words: Use the Phonetisaurus tool to generate IPA annotations for Chinese words. The Phonetisaurus tool can generate annotations from words to phonemes by loading the annotation model of the corresponding language. The annotation model used can be downloaded from https: / / github.com / uiuc-sst / g2ps. Finally, a Chinese word-phoneme pronunciation dictionary can be obtained.

[0037] 2) Construction of the BPE-phone pronunciation dictionary for automotive terminology, see Figure 1 , the process is: 2-1) Generate a BPE list of automotive terminology: Step 1: Set the maximum number of word segmentation dictionaries to vocab size = 500 and initialize a dictionary.

[0038] Step 2: Cut all the text in the corpus into single characters and add them to the dictionary, and remove special characters (<eos> ， <bos> ， <unk>, space characters, etc.) are also added to the dictionary.

[0039] Step 3: For the corpus that has been cut into characters, globally count the frequency of the combination of two consecutive characters in one round.

[0040] Step 4: Take the combination with the highest frequency, merge the two characters into a whole, add this whole to the dictionary, and simultaneously replace the two characters in the corpus with this new whole as a word.

[0041] Step 5: Repeat step 3 and step 4 until the vocab size is reached or it cannot be merged anymore.

[0042] Step 6: Generate a segmentation encoding model file from the final dictionary, such as tokenizer.model, and then use this segmentation dictionary to segment and encode words to generate a BPE list for Mandarin.

[0043] Step 7: Ask experts in automotive terminology to perform IPA phonetic notation on the BPE list of terms to form a BPE-phone pronunciation dictionary for automotive terminology.

[0044] The second step is to train the basic speech recognition model The specific training process is as follows: Assume that the input is a speech feature sequence X = {x1,x2,…,x T }; After Hubert encoding, the encoded feature vector sequence is obtained: , The loss function calculation formula is: ; Among them, Y is the target correct phoneme sequence for model training, is a possible alignment of Y (alignment path), represents the set of all possible alignments that map Y to. is the alignment path given the input sequence X The conditional probability of .

[0045] After obtaining the automotive professional terminology speech-IPA pair dataset, use the attached figure Figure 2 The Conformer model structure shown in the figure is used as an encoder, and multi-national speech phoneme pair data including automotive professional terminology speech-IPA pair data is used as training data to train the acoustic model. The overall process is shown in the figure Figure 3 As shown in the figure, the input of the acoustic model is the fbank feature extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

[0046] It should be pointed out that the basic speech recognition model in this embodiment is standard Mandarin, and there will be some dialects after the acoustic model training is completed, such as Zhuang, Hakka, Minnan, etc.

[0047] The third step is the training of the target speech recognition model.

[0048] The target speech recognition model is selected as shown in the attached figure Figure 4 The encoder-decoder architecture based on the attention mechanism is shown in the figure. In the encoder part, the Conformer architecture with the same acoustic model structure is used, and the encoder structure is initialized with the pre-trained acoustic model parameters, thereby successfully integrating the acoustic model migration into the complete speech recognition model.

[0049] Using annotated automotive terminology data (automotive terminology audio-automotive terminology text pairs), a speech recognition model based on the Conformer-Transformer architecture is trained. The encoder-decoder architecture mainly consists of two parts: the encoder and the decoder. The encoder generates a higher-level representation feature vector based on the input fbank feature sequence. The feature vector generated by the encoder can better capture the key information in the audio signal. The decoder receives the feature vector output by the encoder and gradually generates a text sequence. When generating a text sequence, the decoder considers the language model and prior knowledge, as well as part of the previously generated text; at the same time, based on the attention mechanism, the decoder can better pay attention to the decoder information that is more effective for the current time step when generating at each time step. During the training process, the model receives known audio-text pairing data. By minimizing the gap between the predicted text and the actual text (usually using the cross entropy loss function), the parameters of the model are adjusted so that the model can more accurately convert the audio signal into the corresponding text.

[0050] In summary, the embodiments of the present invention make improvements in the following aspects: (1) Generate IPA phoneme annotations with the help of Hubert to achieve multi-language mixed training acoustic model In order to enable the acoustic model to better capture the pronunciation features in the audio, and to enhance the data for multi-language mixed training, the IPA annotation sequence is often used as the training target of the acoustic model. However, due to the lack of basic research work related to Mandarin, there is no available IPA annotation model for automotive terminology. In order to solve this problem, the present invention uses the pre-trained Hubert model to fine-tune the Hubert model by using the currently available multi-national audio-phoneme annotation data so that it can have a passing phoneme annotation capability. Then, the fine-tuned Hubert model is used to perform pronunciation annotation for a large amount of unlabeled Mandarin audio. Then, the Conformer model is used to construct an acoustic model, and the acoustic model training is performed based on multi-language data including automotive terminology.

[0051] (2) Based on model transfer, using the trained acoustic model to reduce the difficulty of speech recognition model training Generally, after obtaining a trained acoustic model, it will be combined with a speech model and a pronunciation dictionary, and the final text sequence will be obtained through WFST decoding. However, the IPA annotations of automotive professional terms are obtained through a trained annotation model, and there is no available pronunciation dictionary. Therefore, it is impossible to directly use the acoustic model to obtain the final text sequence. In order to use the pronunciation dictionary, one approach is to connect a linear neural network layer on the basis of the acoustic model, and use labeled data and CTC loss function to fine-tune the model to obtain a speech recognition model. However, in this approach, the external linear layer is too simple and cannot make good use of the output information of the acoustic model. At the same time, the training target of the acoustic model is phonemes, and fine-tuning is directly performed on this basis. Because the training tasks are inconsistent, very good results are often not achieved. The present invention uses a method of migrating the acoustic model as the encoder part of the encoder-decoder speech recognition model, which achieves the goal of using a trained acoustic model to reduce the difficulty of speech recognition model training.

[0052] By using the acoustic model as an encoder, we can fully utilize the acoustic model's ability to capture the pronunciation features in the sound signal. With an additional decoder, with the decoder's strong decoding ability, we can fully utilize the pronunciation information output by the encoder, take into account the language model and prior knowledge, and some of the previously generated text.

[0053] Based on the same concept, an embodiment of the present invention provides a cleaning control device applied to a cleaning robot. The cleaning control device includes: a control module, which is used to control the cleaning robot to rotate itself during the cleaning operation of the cleaning robot.

[0054] Based on the same concept, an embodiment of the present invention provides an automotive industry terminology speech recognition device, which includes a memory and a processor. The memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the automotive industry terminology speech recognition method provided by the embodiment of the present invention.

[0055] Based on the same concept, an embodiment of the present invention provides a computer-readable storage medium, in which at least one instruction is stored. The instruction is loaded and executed by a processor to implement the automotive industry terminology speech recognition method provided by an embodiment of the present invention.

[0056] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0057] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / unk> < / bos> < / eos>

Claims

1. A method for speech recognition of automotive industry terminology, characterized in that: include: Construct Chinese word-phone pronunciation dictionary and automotive professional terminology BPE-phone pronunciation dictionary; Use Chinese speech data to train the initialized basic speech recognition model; The parameter weights are transferred from the basic speech recognition model, and the target speech recognition model is trained using Chinese speech data on the transferred model.

2. The automotive industry terminology speech recognition method according to claim 1, characterized in that: The method for constructing the Chinese word-phone pronunciation dictionary comprises: Generate a Chinese word list: perform word statistics from a Chinese text data set to generate a Chinese word list; the text data set includes a training set, a test set, and a validation set; Pronunciation of Chinese words: Generate IPA annotations for Chinese words and obtain a Chinese word-phoneme pronunciation dictionary.

3. The automotive industry terminology speech recognition method according to claim 2, characterized in that: The method for constructing the automobile professional term BPE-phone pronunciation dictionary comprises: Step 1: Set the maximum number of word segmentation dictionaries and initialize a dictionary; Step 2: Cut all the text in the corpus into single characters and add them to the dictionary, and also add special characters to the dictionary; Step 3: For the corpus that has been cut into characters, globally count the frequency of two consecutive character combinations in one round; Step 4: Take the combination with the highest frequency, merge the two characters into a whole, add this whole to the dictionary, and replace the two characters in the corpus with this new whole at the same time, treating it as a word; Step 5: Repeat step 3 and step 4 until the maximum number of word segmentation dictionaries is reached or no more merging is possible; Step 6: Generate a word segmentation encoding model file from the final dictionary, and then use this word segmentation dictionary to segment and encode words to generate a BPE list for Mandarin; Step 7: Perform phonetic notation of automotive professional terms in the BPE list to form a BPE-phone pronunciation dictionary of automotive professional terms.

4. The automotive industry terminology speech recognition method according to claim 3, characterized in that: The training method of the basic speech recognition model includes: Assume that the input is a speech feature sequence X = {x1,x2,…,x T }, After Hubert encoding, the encoded feature vector sequence is obtained: ， The loss function calculation formula is: ； Among them, Y is the target correct phoneme sequence for model training, is a possible alignment of Y, represents the set of all possible alignments that map Y to. is the alignment path given the input sequence X The conditional probability of .

5. The method for speech recognition of automotive industry terminology according to claim 4, characterized in that: The training method of the basic speech recognition model also includes: After obtaining the automotive professional terminology speech-IPA pair dataset, the Conformer model structure is used as the encoder, and the multi-national speech phoneme pair data including the automotive professional terminology speech-IPA pair data is used as training data to train the acoustic model; the input of the acoustic model is the fbank feature extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

6. The method for speech recognition of automotive industry terminology according to claim 5, characterized in that: The step of migrating parameter weights from the basic speech recognition model and training the target speech recognition model using Chinese speech data on the migrated model includes: The target speech recognition model selects an encoder-decoder architecture based on the attention mechanism; The encoder part uses the same Conformer architecture as the acoustic model structure and uses the pre-trained acoustic model parameters to initialize the encoder structure, thereby successfully migrating the acoustic model into the complete target speech recognition model.

7. A speech recognition device for automotive industry terminology, characterized in that: including memory and processor; The memory is used to store instructions; The processor is used to execute the instructions in the memory to implement the automotive industry terminology speech recognition method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, enable the computer to execute the method for speech recognition of automotive industry terms as claimed in any one of claims 1 to 6.

Images