A method and device for recognizing terms in the automobile industry, and a storage medium

Abstract

The application relates to the field of speech recognition and discloses a vehicle industry term speech recognition method and device and a storage medium. The method comprises the following steps: constructing a Chinese word-phone pronunciation dictionary and a vehicle professional term BPE-phone pronunciation dictionary; training an initialized basic speech recognition model by using Chinese speech data; and migrating the parameter weight from the basic speech recognition model, and training a target speech recognition model by using the Chinese speech data on the migrated model. According to the embodiment of the application, a speech recognition system with good effect can be efficiently trained under the condition that only a small amount of high-quality vehicle professional term speech data set and a large amount of unlabeled vehicle professional term speech data are needed. The finally obtained speech recognition system does not use a pre-training model with excessive parameter quantity, ensures the speed of speech recognition, and reduces the requirement on hardware in actual application.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method, device, and storage medium for automotive industry terminology. Background Technology

[0002] Monolingual speech recognition technology based on acoustic models and connectionist temporal classification is currently a mainstream speech recognition technique. This technology constructs an acoustic model based on a speech encoder, trains the model using connectionist temporal classification, and performs speech recognition using bundle search.

[0003] The so-called acoustic model based on a speech encoder is a neural network model built on a deep neural network. The raw audio signal is processed through a filter bank to obtain a sequence of sound information, or further processed to obtain the inverse of the Mel-frequency spectrum, which serves as the model's input. The acoustic model accepts this frame-by-frame acoustic information input and maps it to a high-dimensional feature space through a series of deep neural network layers, outputting a sequence of high-dimensional feature vectors. Then, a feedforward neural network is used to calculate the prediction result aligned with the frame. To align with the target text sequence for speech recognition, a connectionist temporal classification function is used as the loss function to calculate the gap between the prediction result and the true target; this gap guides the model's learning and training.

[0004] Beam search is a search algorithm for sequence generation problems. Its core idea is to consider multiple possible candidates at each step of sequence generation, organizing these candidates into a beam. By calculating the score of each candidate, beam search selects the subset of candidates with the highest scores as the candidate set for the next step. This process iterates continuously, pruning and filtering, until the final generated sequence is obtained. Beam search effectively addresses the uncertainty and diversity in sequence generation tasks by balancing computational efficiency and the breadth of the search space.

[0005] Training this method requires a large amount of labeled data in the target language, namely speech segments and their corresponding texts. However, automotive industry terminology has relatively few users and researchers, resulting in limited training data. Currently, there are no publicly available resources of large amounts of labeled training data. Without sufficient training samples, monolingual speech recognition techniques based on acoustic models and connectionist temporal classification struggle to train usable automatic speech recognition systems, often resulting in high error rates. Furthermore, a pronunciation dictionary for the target language is needed. The pronunciation of technical terms sometimes carries dialectal elements, so creating such a dictionary requires expert guidance and is extremely time-consuming and labor-intensive if done manually. Summary of the Invention

[0006] The purpose of this invention is to provide a speech recognition method, device, and storage medium for automotive industry terminology, in order to overcome the shortcomings of existing technologies, such as high error rates and high time and effort consumption.

[0007] To achieve this objective, the present invention adopts the following technical solution:

[0008] A speech recognition method for automotive industry terminology includes:

[0009] Construct a Chinese word-phone pronunciation dictionary and an automotive terminology BPE-phone pronunciation dictionary;

[0010] The initial basic speech recognition model was trained using Chinese speech data;

[0011] The parameter weights are transferred from the basic speech recognition model, and the target speech recognition model is trained on the transferred model using Chinese speech data.

[0012] Optionally, the method for constructing the Chinese word-phone pronunciation dictionary includes:

[0013] Generate a Chinese word list: Perform word statistics from a Chinese text dataset to generate a Chinese word list; the text dataset includes a training set, a test set, and a validation set;

[0014] Phonetic annotation for Chinese words: Generate IPA annotations for automotive terminology for Chinese words, resulting in a Chinese word-phoneme pronunciation dictionary.

[0015] Optionally, the method for constructing the BPE-phone pronunciation dictionary for automotive terminology includes:

[0016] Step 1: Set the maximum number of words in the word segmentation dictionary and initialize a dictionary;

[0017] Step 2: Cut all text in the corpus into individual character form and add them to the dictionary, and also add special characters to the dictionary;

[0018] Step 3: For the corpus that has been segmented into characters, globally calculate the frequency of two consecutive character combinations in one round;

[0019] Step 4: Take the combination with the highest frequency, merge the two characters into a whole, add this whole to the dictionary, and simultaneously replace both characters in the corpus with this new whole as a word;

[0020] Step 5: Repeat steps 3 and 4 until the maximum number of word segmentation dictionaries is reached or no further merging is possible;

[0021] Step 6: Generate a word segmentation and encoding model file from the final dictionary, and then use this word segmentation dictionary to segment and encode words, thereby generating a list of BPEs for Mandarin.

[0022] Step 7: Add IPA phonetic annotations to the BPE list to create a BPE-phone pronunciation dictionary for automotive terminology.

[0023] Optionally, the training method for the basic speech recognition model includes:

[0024] Suppose the input is a speech feature sequence X = {x1,x2,…,x} consisting of T downsampled frames. T},

[0025] After Hubert encoding, the encoded feature vector sequence is obtained:

[0026] ,

[0027] The formula for calculating its loss function is:

[0028] ;

[0029] Where Y is the target correct phoneme sequence during model training. This is one possible alignment of Y. This represents mapping Y to the set of all possible alignments. Given an input sequence X, the alignment path is... The conditional probability.

[0030] Optionally, the training method for the basic speech recognition model further includes:

[0031] After obtaining the BPE-phone pronunciation dictionary for automotive terminology, a Conformer model structure is used as the encoder. Multilingual speech phoneme pairs, including the BPE-phone pronunciation dictionary, are used as training data to train the acoustic model. The input of the acoustic model is the fbank features extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

[0032] Optionally, the transfer of parameter weights from the basic speech recognition model, and the training of the target speech recognition model on the transferred model using Chinese speech data, includes:

[0033] The target speech recognition model adopts an encoder-decoder architecture based on an attention mechanism;

[0034] In the encoder part, the same Conformer architecture as the acoustic model structure is used, and the encoder structure is initialized with the pre-trained acoustic model parameters, thus successfully transferring and integrating the acoustic model into the complete target speech recognition model.

[0035] A speech recognition device for automotive industry terminology includes a memory and a processor;

[0036] The memory is used to store instructions;

[0037] The processor is configured to execute the instructions in the memory to implement the automotive industry terminology speech recognition method described above.

[0038] A computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the automotive industry terminology speech recognition method described in any of the above embodiments.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] This invention enables efficient training of a high-performance speech recognition system with only a small amount of high-quality automotive terminology speech data and a large amount of unlabeled automotive terminology speech data. The final speech recognition system avoids using a pre-trained model with excessively large parameters, ensuring speech recognition speed and reducing hardware requirements for practical applications. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the method for constructing a BPE-phone pronunciation dictionary for automotive terminology provided in this embodiment of the invention.

[0043] Figure 2 The diagram shows the structure of the Conformer model provided in this embodiment of the invention.

[0044] Figure 3 This is a structural diagram of the phoneme annotation model provided in an embodiment of the present invention;

[0045] Figure 4 The encoder-decoder structure based on the attention mechanism is provided in the embodiments of the present invention. Detailed Implementation

[0046] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] For ease of understanding, the embodiments of the present invention provide the following definitions of abbreviations and key terms involved:

[0048] ASR stands for Automatic Speech Recognition, referring to a technology that can automatically extract corresponding text from audio information.

[0049] Base model: refers to the model obtained by training the source data in transfer learning.

[0050] Target model: refers to the model obtained by training on target data in transfer learning.

[0051] Source data: refers to the data required for model pre-training, and in this patent, it specifically refers to Thai speech data.

[0052] Target data: refers to the data required for parameter optimization of the target model; in this patent, it specifically refers to Mandarin speech data.

[0053] Phoneme: A phoneme is the smallest unit of speech, the smallest segment of sound that can distinguish the meaning of a word. Every language has its own phoneme system, and words are formed by combining different phonemes.

[0054] BPE: Byte Pair Encoder is a data compression algorithm used to implement variable-length subwords in a fixed-size vocabulary.

[0055] Subword: A subword is a smaller piece of text that is between a word and a character.

[0056] Acoustic Model: The task of an acoustic model is to predict the corresponding phonemes, words, or other speech units based on the input speech signal. In this invention, the task of the acoustic model is to predict the phonemes corresponding to the speech signal.

[0057] A pronunciation dictionary is a vocabulary that contains pronunciation information for words. It maps words to their phoneme sequences or other phonetic units to describe how to pronounce those words correctly. In applications such as speech recognition and speech synthesis, pronunciation dictionaries are crucial for converting text into corresponding phoneme sequences or vice versa.

[0058] BPE Pronunciation Dictionary: Contains pronunciation information for BPE subwords, mapping BPE subwords to corresponding phoneme sequences or other pronunciation units to describe how to pronounce these words correctly.

[0059] The Connectionist Temporal Classification (CTC) function is a loss function used for sequence learning tasks such as speech recognition and handwriting recognition. It allows for imperfect alignment between the input and output sequences; that is, the lengths of the input and output sequences can differ, and the output sequence can contain repeated characters and whitespace. The goal of CTC is to maximize the match between the conditional probability of a given input sequence and its corresponding output sequence.

[0060] F-bank features: F-bank features are a commonly used feature representation method in the field of speech signal processing, also known as filter bank features. F-bank features are obtained by performing a series of filter bank operations on the speech signal and performing logarithmic operations on the output of each filter bank, finally obtaining the feature vector for each time window.

[0061] To address the shortcomings of existing technologies, this invention provides a BPE-phone migration technology for recognizing automotive terminology, which mainly includes three steps:

[0062] The first step is to build a pronunciation dictionary, including a Chinese word-phone pronunciation dictionary and an automotive terminology BPE-phone pronunciation dictionary;

[0063] The second step is to train the basic speech recognition model, using Chinese speech data to train the initialized model;

[0064] The third step is to train the target speech recognition model, which involves transferring parameter weights from the base model and then training the transferred model using Chinese speech data.

[0065] The three steps described above are described in detail below.

[0066] The first step is to build a pronunciation dictionary.

[0067] 1) The construction process of the Chinese word-phone pronunciation dictionary is as follows:

[0068] 1-1) Generate a list of Chinese words: Perform word statistics from the Chinese text dataset (including the training set, test set, and validation set) to generate a list of Chinese words.

[0069] 1-2) Phonetic annotation of Chinese words: Use the Phonetisaurus tool to generate IPA annotations for Chinese words. The Phonetisaurus tool can generate word-to-phoneme annotations by loading the annotation model of the corresponding language. The annotation model can be downloaded online. Finally, a Chinese word-phoneme pronunciation dictionary can be obtained.

[0070] 2) For the construction of the BPE-phone pronunciation dictionary for automotive terminology, please refer to [link / reference]. Figure 1 The process is as follows:

[0071] 2-1) Generate a list of automotive terminology BPE:

[0072] Step 1: Set the maximum number of words in the word segmentation dictionary (vocab size=500) and initialize a dictionary.

[0073] Step 2: Cut all text in the corpus into individual character form and add it to the dictionary, and include special characters ( <eos> ， <bos> ， <unk>Characters such as spaces are also added to the dictionary.

[0074] Step 3: For the corpus that has been segmented into characters, globally calculate the frequency of two consecutive character combinations in a round.

[0075] Step 4: Take the combination with the highest frequency, merge the two characters into a whole, add this whole to the dictionary, and simultaneously replace both characters in the corpus with this new whole as a word.

[0076] Step 5: Repeat steps 3 and 4 until the vocab size is reached or no more merging is possible.

[0077] Step 6: Generate a word segmentation and encoding model file from the final dictionary, such as tokenizer.model. Then use this word segmentation dictionary to segment and encode words, thereby generating a list of BPEs for Mandarin.

[0078] Step 7: Ask automotive terminology experts to annotate the BPE terminology list with IPA phonetic transcription, creating a BPE-phone pronunciation dictionary for automotive terminology.

[0079] The second step is training the basic speech recognition model.

[0080] The specific training process is as follows: assuming the input is a speech feature sequence X = {x1,x2,…,x} consisting of T downsampled frames. T };

[0081] After Hubert encoding, the encoded feature vector sequence is obtained:

[0082] ,

[0083] The formula for calculating its loss function is:

[0084] ;

[0085] Where Y is the target correct phoneme sequence during model training. It is one possible alignment method (alignment path) for Y. This represents mapping Y to the set of all possible alignments. Given an input sequence X, the alignment path is... The conditional probability.

[0086] After obtaining the BPE-phone pronunciation dictionary for automotive terminology, use it as shown in the attached image. Figure 2 The Conformer model structure shown serves as the encoder. Multilingual phoneme pairs, including a dictionary of BPE-phone pronunciations for automotive terminology, are used as training data to train the acoustic model. The overall process is illustrated in the attached diagram. Figure 3 As shown, the input to the acoustic model is the fbank features extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function for model training.

[0087] It should be noted that the basic speech recognition model in this embodiment is standard Mandarin. After the acoustic model is trained, some dialects will be used, such as Zhuang, Hakka, and Minnan.

[0088] The third step is training the target speech recognition model.

[0089] The target speech recognition model selection is shown in the attached figure. Figure 4 The encoder-decoder architecture based on the attention mechanism is shown. The encoder part uses the same Conformer architecture as the acoustic model, and initializes the encoder structure with pre-trained acoustic model parameters, thus successfully transferring and integrating the acoustic model into the complete speech recognition model.

[0090] A speech recognition model based on a Conformer-Transformer architecture is trained using labeled automotive terminology data (audio-text pairs of automotive terminology). The encoder-decoder architecture consists of two main parts: an encoder and a decoder. The encoder generates higher-level representation feature vectors based on the input fbank feature sequence. These feature vectors better capture key information in the audio signal. The decoder receives the feature vectors output by the encoder and progressively generates text sequences. When generating text sequences, the decoder considers the language model, prior knowledge, and previously generated text. Simultaneously, based on an attention mechanism, the decoder pays more attention to decoder information that is more relevant to the current time step during generation. During training, the model receives known audio-text pairing data. By minimizing the difference between the predicted and actual text (typically using a cross-entropy loss function), the model's parameters are adjusted to more accurately convert audio signals into corresponding text.

[0091] In summary, the embodiments of the present invention have made improvements in the following aspects:

[0092] (1) By using Hubert to generate IPA phoneme annotations, multilingual mixed training of acoustic models can be achieved.

[0093] To enable acoustic models to better capture pronunciation features in audio and to enhance multilingual training, IPA-annotated sequences are often used as training targets. However, due to limited basic research on Mandarin, there are no readily available IPA annotation models for automotive terminology. To address this issue, this invention leverages a pre-trained Hubert model, fine-tuning it using currently available multilingual audio-phoneme annotation data to achieve adequate phoneme annotation capabilities. Then, the fine-tuned Hubert model is used to annotate pronunciations in a large volume of unannotated Mandarin audio. Finally, an acoustic model is constructed using a Conformer model and trained on multilingual data, including automotive terminology.

[0094] (2) Based on model transfer, the training difficulty of speech recognition model is reduced by utilizing the pre-trained acoustic model.

[0095] Generally, after obtaining a trained acoustic model, it is combined with a speech model and a pronunciation dictionary to obtain the final text sequence through WFST decoding. However, the IPA annotations for automotive terminology are obtained through a trained annotation model, without a usable pronunciation dictionary. Therefore, the acoustic model cannot be directly used to obtain the final text sequence. To utilize a pronunciation dictionary, one approach is to add a linear neural network layer to the acoustic model, fine-tuning the model using labeled data and a CTC loss function to obtain a speech recognition model. However, this approach results in an overly simple linear layer that cannot effectively utilize the output information of the acoustic model. Furthermore, the training objective of the acoustic model is phonemes; directly fine-tuning it based on this often fails to achieve good results due to the inconsistent training tasks. This invention uses a method of transferring the acoustic model as the encoder part of an encoder-decoder speech recognition model, achieving the goal of reducing the training difficulty of the speech recognition model by utilizing a trained acoustic model.

[0096] By using an acoustic model as an encoder, its ability to capture pronunciation features in sound signals can be fully utilized. An additional decoder, leveraging its powerful decoding capabilities, can further enhance the pronunciation information output by the encoder while also taking into account the language model, prior knowledge, and previously generated text.

[0097] Based on the same concept, embodiments of the present invention provide a speech recognition device for automotive industry terminology. The device includes a memory and a processor. The memory stores at least one instruction, which is loaded and executed by the processor to implement the speech recognition method for automotive industry terminology provided in this embodiment of the invention.

[0098] Based on the same concept, embodiments of the present invention provide a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the speech recognition method for automotive industry terminology provided in embodiments of the present invention.

[0099] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0100] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / unk> < / bos> < / eos>

Claims

1. A speech recognition method for automotive industry terminology, characterized in that, include: Construct a Chinese word-phone pronunciation dictionary and an automotive terminology BPE-phone pronunciation dictionary; The initial basic speech recognition model was trained using Chinese speech data; The parameter weights are transferred from the basic speech recognition model, and the target speech recognition model is trained on the transferred model using Chinese speech data. The method for constructing the BPE-phone pronunciation dictionary for automotive terminology includes: Step 1: Set the maximum number of words in the word segmentation dictionary and initialize a dictionary; Step 2: Cut all text in the corpus into individual character form and add them to the dictionary, and also add special characters to the dictionary; Step 3: For the corpus that has been segmented into characters, globally calculate the frequency of two consecutive character combinations in one round; Step 4: Take the combination with the highest frequency, merge the two characters into a whole, add this whole to the dictionary, and simultaneously replace both characters in the corpus with this new whole as a word; Step 5: Repeat steps 3 and 4 until the maximum number of word segmentation dictionaries is reached or no further merging is possible; Step 6: Generate a word segmentation and encoding model file from the final dictionary, and then use this word segmentation dictionary to segment and encode words, thereby generating a list of BPEs for Mandarin. Step 7: Perform IPA phonetic transcription of the BPE list to form a BPE-phone pronunciation dictionary for automotive terminology; After obtaining the BPE-phone pronunciation dictionary for automotive terminology, a Conformer model structure is used as the encoder. Multilingual speech phoneme pairs, including the BPE-phone pronunciation dictionary, are used as training data to train the acoustic model. The input of the acoustic model is the fbank features extracted from the original audio signal, and the target output is the phoneme sequence corresponding to the audio. The CTC loss function is used as the objective function to train the model.

2. The speech recognition method for automotive industry terminology according to claim 1, characterized in that, The method for constructing the Chinese word-phone pronunciation dictionary includes: Generate a Chinese word list: Perform word statistics from a Chinese text dataset to generate a Chinese word list; the text dataset includes a training set, a test set, and a validation set; Phonetic annotation for Chinese words: Generate IPA annotations for Chinese words to obtain a Chinese word-phone pronunciation dictionary.

3. The speech recognition method for automotive industry terminology according to claim 2, characterized in that, The training method for the basic speech recognition model includes: Suppose the input is a speech feature sequence X = {x1,x2,…,x} consisting of T downsampled frames. T }, After Hubert encoding, the encoded feature vector sequence is obtained: ， The formula for calculating its loss function is: ； Where Y is the target correct phoneme sequence during model training. This is one possible alignment of Y. This represents mapping Y to the set of all possible alignments. Given an input sequence X, the alignment path is... The conditional probability.

4. The speech recognition method for automotive industry terminology according to claim 3, characterized in that, The transfer of parameter weights from the basic speech recognition model, followed by training the target speech recognition model on the transferred model using Chinese speech data, includes: The target speech recognition model adopts an encoder-decoder architecture based on an attention mechanism; In the encoder part, the same Conformer architecture as the acoustic model structure is used, and the encoder structure is initialized with the pre-trained acoustic model parameters, thus successfully transferring and integrating the acoustic model into the complete target speech recognition model.

5. A speech recognition device for automotive industry terminology, characterized in that, Including memory and processor; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to implement the automotive industry terminology speech recognition method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The instruction, when executed on a computer, causes the computer to perform the automotive industry terminology speech recognition method according to any one of claims 1 to 4.

Images