Speech instruction recognition method based on BERT pre-training language model

Abstract

The invention relates to the technical field of voice recognition, and discloses a voice instruction recognition method based on a BERT pre-training language model, and the method comprises the steps: receiving a to-be-recognized voice, and converting the to-be-recognized voice into a text sequence through an automatic voice recognition model; fine-tuning the BERT pre-training language model, and performing text classification on the text sequence by using the fine-tuned BERT pre-training language model to obtain an instruction category corresponding to the to-be-recognized voice; performing model compression on the fine-tuned BERT pre-training language model by utilizing knowledge distillation to obtain a sequence classification model; according to the method, the situation that the audio data is more difficult to obtain than the text data is considered, the BERT is selected to pre-train the language model, the classification accuracy is improved, the knowledge distillation is utilized to perform model compression, the instruction category label of the to-be-recognized voice is recognized, and the voice recognition accuracy is improved. And data support is provided for man-machine interaction.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech command recognition method based on a BERT pre-trained language model. Background Art

[0002] Short voice command recognition is widely used in life, office, military and other fields. The higher the recognition accuracy, the more convenient and practical the human-computer interaction can be. Voice commands are short, highly repetitive, and lack context. Traditional end-to-end voice command recognition methods mainly extract spectral features of audio. When faced with commands that have never appeared but have the same meaning, they lack understanding of semantic relevance, so it is difficult to generalize and apply them universally.

[0003] At present, the rapid development of smart mobile devices and cloud computing has greatly promoted the advancement of artificial intelligence, making it a symbol of the field of science and technology. Artificial intelligence has had a profound impact on our daily lives and military activities. Today, smart robots, smart homes, augmented reality devices, self-driving cars, automated digital industries, unmanned logistics and smart medical care have become an indispensable part of human life. Language is a bridge for interpersonal communication, and rich semantic information promotes mutual understanding and collaboration among people. In the field of artificial intelligence, the convenience of human-computer interaction is the key to user experience, and the core of all this is language communication. The premise for robots to accurately execute commands is to fully understand the commands. The ability to recognize commands with different timbres and different expressions but the same meaning and accurately execute them is the core of speech recognition technology. Therefore, speech recognition technology has become a key research field.

[0004] Short voice command recognition technology is an important branch of speech recognition technology, focusing on identifying specific commands or requests from short voice clips provided by users. Voice technology plays a key role in various fields, including mobile devices, home, medical and military. In the field of mobile devices, voice-based applications such as Apple's Siri, Huawei's Xiaoyi, Xiaomi's Xiaoai, as well as voice search and SMS dictation functions have penetrated into daily life, all thanks to advanced voice recognition technology. For smart homes, users can control home appliances with simple commands, such as "turn on the air conditioner" or "turn off the TV", and devices such as Xiaoai speakers have become standard in many families. In the medical field, voice-controlled devices provide convenience for people with disabilities and mobility impairments, and improve operational efficiency. In military applications, short voice control commands provide convenient operation for equipment such as drones.

[0005] As an audio classification task, command recognition usually uses a single model that takes audio data as input and directly outputs the corresponding category. This method has some problems in the scenario of short voice command recognition. Voice commands are used to convey relatively deterministic information, and are usually short, repetitive, and lack context. In addition, the size of the data set is limited. Directly training a single model cannot obtain a full understanding of the audio semantics, and the generalization ability is weak. Summary of the invention

[0006] In view of this, the present invention provides a voice command recognition based on the BERT pre-trained language model to solve the problem that a single model is used for voice recognition in the prior art, which has the problem of being unable to fully understand the audio semantics and having weak generalization ability.

[0007] In a first aspect, the present invention provides a method for voice command recognition based on a BERT pre-trained language model, the method comprising:

[0008] Receive the speech to be recognized, and convert the speech to be recognized into a text sequence through an automatic speech recognition model;

[0009] Fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to classify the text sequence to obtain the instruction category corresponding to the speech to be recognized;

[0010] Use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model;

[0011] The sequence classification model is used to extract features from the text sequence to obtain the instruction category label corresponding to the speech to be recognized.

[0012] The present invention automatically recognizes the speech to be recognized through an automatic speech recognition model, converts the speech to be recognized into a text sequence, and then uses the BERT pre-trained language model to perform text classification on the text sequence, so as to fully consider the fact that audio data is more difficult to obtain than text data, and selects the BERT pre-trained language model instead of the speech recognition model to improve the classification accuracy, and uses knowledge distillation to compress the model into a smaller model, so as to reduce the model size and improve the recognition efficiency while ensuring the accuracy, and uses the sequence classification model to extract features from the text sequence to accurately identify the instruction category label corresponding to the speech to be recognized, thereby completing the voice instruction recognition and providing data support for human-computer interaction.

[0013] In an optional implementation, the automatic speech recognition model is a Whisper automatic speech recognition model, which uses an encoder-decoder Transformer model. The automatic speech recognition model converts the speech to be recognized into a text sequence, including:

[0014] The encoder-decoder Transformer model is used to convert the speech to be recognized into the corresponding text sequence.

[0015] The present invention recognizes speech by adopting the Transformer model of the encoder-decoder in the Whisper automatic speech recognition model, and converts the speech to be recognized into a text sequence, so as to give full play to the accuracy and robustness of the Whisper automatic speech recognition model in speech recognition to reach a level close to that of humans.

[0016] In an optional implementation, the fine-tuned BERT pre-trained language model is used to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized, including:

[0017] Split the text sequence into token sequences and add special markers before the token sequences;

[0018] Convert the token sequence into word embeddings and input them into the BERT pre-trained language model. Use the Transformer encoding block in the BERT pre-trained language model to extract features and obtain the deep representation corresponding to each token sequence.

[0019] The output features corresponding to the special tags are used as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

[0020] The present invention splits the text sequence and inputs the token sequence obtained by the split into a BERT pre-trained language model for feature extraction to determine the instruction category of the speech to be recognized, thereby facilitating the recognition of the instruction of the speech to be recognized.

[0021] In an optional implementation, knowledge distillation is used to perform model compression on the fine-tuned BERT pre-trained language model, including:

[0022] Knowledge distillation is performed on the fine-tuned BERT pre-trained language model to compress the teacher model into a student model.

[0023] The present invention compresses the BERT pre-trained language model into a smaller model by utilizing knowledge distillation, thereby reducing the delay of speech recognition and improving the speed of speech recognition.

[0024] In an optional implementation, after compressing the teacher model into a student model, the method further includes:

[0025] The loss function of the compressed student model is calculated, and the model parameters of the student model are adjusted using the calculation results until the model loss is within the loss threshold range.

[0026] The present invention adjusts the student model according to the loss function calculation result of the compressed student model, so that the model loss is within the loss threshold range, thereby improving the recognition ability of the model and improving the generalization ability of the model.

[0027] In an optional implementation, the loss function is calculated according to the following formula:

[0028]

[0029] in, is the loss function, is the cross entropy loss corresponding to the hard target, is the cross entropy loss corresponding to the soft target, y is the category label of sample x, and y k is the k-th category label, K is the total number of categories, q is the output distribution of the student model obtained by the softmax function (T=1), α∈[0,1], and T is the temperature parameter.

[0030] The present invention calculates the loss function so as to adjust the model according to the loss function, thereby improving the generalization ability of the model.

[0031] In a second aspect, the present invention provides a voice command recognition device based on a BERT pre-trained language model, the device comprising:

[0032] A conversion module, used for receiving the speech to be recognized and converting the speech to be recognized into a text sequence through an automatic speech recognition model;

[0033] The classification module is used to fine-tune the BERT pre-trained language model and use the fine-tuned BERT pre-trained language model to classify the text sequence to obtain the instruction category corresponding to the speech to be recognized;

[0034] The compression module is used to compress the fine-tuned BERT pre-trained language model using knowledge distillation to obtain a sequence classification model;

[0035] The feature extraction module is used to extract features from the text sequence using the sequence classification model to obtain the instruction category label corresponding to the speech to be recognized.

[0036] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the voice command recognition method based on the BERT pre-trained language model of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the voice command recognition method based on the BERT pre-trained language model of the above-mentioned first aspect or any corresponding embodiment thereof.

[0038] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the voice command recognition method based on the BERT pre-trained language model of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flow chart of a method for voice command recognition based on a BERT pre-trained language model according to an embodiment of the present invention;

[0041] Figure 2 is a framework diagram of a voice command recognition system based on a BERT pre-trained language model according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a BERT-based sequence classification model architecture according to an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of the knowledge distillation principle according to an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of training and accuracy changes according to an embodiment of the present invention;

[0045] Figure 6 is a schematic diagram of training and loss changes according to an embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of the test and accuracy change according to an embodiment of the present invention;

[0047] Figure 8 is a schematic diagram of a test and loss change according to an embodiment of the present invention;

[0048] Fig. 9 is a structural block diagram of a voice command recognition device based on a BERT pre-trained language model according to an embodiment of the present invention;

[0049] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] The following is an introduction to the current research status:

[0052] Early speech recognition systems relied on simple pattern matching techniques, and later developed to use Hidden Markov Models (HMM) and Gaussian Mixture Models (GMM). With the rise of machine learning, especially deep learning, speech recognition technology has made revolutionary progress. Modern speech recognition systems generally use deep neural networks, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Transformer models.

[0053] Later, OpenAI open-sourced the Whisper automatic speech recognition model. Whisper is a general speech recognition model that uses a large amount of multi-language and multi-task supervised data for training, and can achieve robustness and accuracy close to human levels in English speech recognition.

[0054] Whisper can also perform tasks such as multilingual speech recognition, speech translation, and language identification. Its architecture is a simple end-to-end method that uses an encoder-decoder Transformer model to convert the input audio into a corresponding text sequence and specify different tasks based on special tags.

[0055] Large language models are mainly based on the Transformer architecture. The key innovation of Transformer is the self-attention mechanism, which allows the model to automatically decide which parts of the input sequence to focus on when processing input data. Since then, multiple high-performance LLMs (Large Language Models) have emerged, including the GPT series, BERT and T5, as well as BART and RoBERTa. These models use a large amount of data to pre-train the models, greatly improving the model's ability to understand the semantic information in the text. They have achieved excellent results in multiple NLP (Natural Language Processing) tasks and refreshed the best results in multiple fields. The bidirectional context understanding of the BERT language model, as well as the pre-training tasks of the Masked Language Model (MLM) and Next Sentence Prediction (NSP) can help it learn deeper language rules and obtain word embedding representations with high semantic relationships.

[0056] Model compression is an important machine learning technology that aims to reduce the size of deep learning models and improve their operating efficiency while trying to maintain or only slightly reduce the performance of the model. This is especially important for deploying advanced artificial intelligence technologies to devices with limited computing resources.

[0057] According to an embodiment of the present invention, an embodiment of a voice instruction recognition method based on a BERT pre-trained language model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0058] In this embodiment, a voice command recognition method based on a BERT pre-trained language model is provided. Figure 1 is a flow chart of a method for voice command recognition based on a BERT pre-trained language model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0059] Step S101, receiving speech to be recognized, and converting the speech to be recognized into a text sequence through an automatic speech recognition model.

[0060] In the embodiment of the present invention, in the currently mature natural language processing (NLP) technology, the large language model is pre-trained on a large text corpus, and a full understanding of the semantics of natural language text is obtained, and the performance of downstream tasks is improved through transfer learning. Therefore, the rich semantic knowledge of the large model can be used to better represent short text instructions. Therefore, in order to take advantage of the pre-trained language model, the embodiment of the present invention adopts the method of first converting the speech to be recognized into a text sequence.

[0061] During the human-computer interaction process, the voice audio input by the user is received. The voice is the voice to be recognized. The function of the automatic speech recognition model is to perform automatic speech recognition, recognize the voice to be recognized through the automatic speech recognition model, and convert the voice to be recognized into a corresponding text sequence to facilitate the analysis of the voice to be recognized.

[0062] Step S102, fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized.

[0063] In an embodiment of the present invention, considering that the pre-trained large language model in natural language processing contains word embedding representations with strong semantic relationships, voice instructions of different statements can be accurately understood. Therefore, automatic speech recognition is combined with the BERT pre-trained language model for speech recognition. The speech to be recognized is first converted into a text sequence, and then the BERT pre-trained language model is fine-tuned. The text sequence is parsed into the corresponding instruction category through the fine-tuned BERT pre-trained language model. After comparing the effects and speeds of different speech recognition models, the model is fine-tuned on a data set designed to simulate real scenarios in specific scenarios.

[0064] It should be noted that, considering that audio data is more difficult to obtain than text data and that text data is richer than audio data in the pre-training stage, a language model is selected rather than a speech recognition model.

[0065] Step S103, using knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model.

[0066] In the embodiment of the present invention, due to the large scale of the pre-trained language model, the inference speed of the pre-trained language model cannot meet the real-time requirements of speech recognition. Therefore, considering the difficulty of model deployment under resource constraints, knowledge distillation is used to compress the fine-tuned BERT pre-trained language model into a smaller model, thereby reducing the model scale and further improving the recognition speed while ensuring the recognition accuracy.

[0067] Step S104: extract features from the text sequence using a sequence classification model to obtain a command category label corresponding to the speech to be recognized.

[0068] In the embodiment of the present invention, a sequence classification model is used to extract features from a text sequence for classification, and an instruction category label is assigned to the speech to be recognized, thereby completing the recognition of the speech to be recognized.

[0069] The voice command recognition method based on the BERT pre-trained language model provided in this embodiment automatically recognizes the voice to be recognized through the automatic speech recognition model, converts the voice to be recognized into a text sequence, and then uses the BERT pre-trained language model to perform text classification on the text sequence, so as to fully consider the fact that audio data is more difficult to obtain than text data, and select the BERT pre-trained language model instead of the speech recognition model to improve the classification accuracy, and use knowledge distillation to compress the model into a smaller model, so as to reduce the model size and improve the recognition efficiency while ensuring the accuracy, and use the sequence classification model to extract features from the text sequence to accurately identify the instruction category label corresponding to the voice to be recognized, thereby completing the voice command recognition and providing data support for human-computer interaction.

[0070] In this embodiment, a voice command recognition method based on a BERT pre-trained language model is provided, and the process includes the following steps:

[0071] Step S201, receiving speech to be recognized, and converting the speech to be recognized into a text sequence through an automatic speech recognition model.

[0072] Specifically, the above step S201 includes:

[0073] Step S2011, using the encoder-decoder Transformer model to convert the speech to be recognized into a corresponding text sequence.

[0074] In the embodiment of the present invention, the automatic speech recognition module adopts the Whisper model, which can quickly and relatively accurately recognize speech information. The original Whisper model includes five versions of different sizes, as well as the FasterWhisper version with improved reasoning speed. After testing the recognition accuracy and recognition speed of these versions under different prompt words, the embodiment of the present invention selects the improved version Faster Whisper-largev2.

[0075] The Whisper model is a general speech recognition model that uses a large amount of multilingual and multi-task supervised data for training, and can achieve near-human-level robustness and accuracy in English speech recognition. The Whisper model can also perform tasks such as multilingual speech recognition, speech translation, and language recognition. The architecture of the Whisper model is a simple end-to-end method that uses an encoder-decoder Transformer model to convert the input speech to be recognized into a corresponding text sequence and specify different tasks based on special tags.

[0076] By adopting the encoder-decoder Transformer model in the Whisper automatic speech recognition model to recognize speech, the speech to be recognized is converted into a text sequence, so as to give full play to the accuracy and robustness of the Whisper automatic speech recognition model in speech recognition that is close to the human level.

[0077] Step S202, fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized.

[0078] Specifically, in the above step S202, the text sequence is classified using the fine-tuned BERT pre-trained language model to obtain the instruction category corresponding to the speech to be recognized, including:

[0079] Step S2021, split the text sequence into token sequences, and add special tags before the token sequences.

[0080] Step S2022, convert the token sequence into word embedding, input it into the BERT language pre-training model, use the Transformer encoding block in the BERT pre-training language model to perform feature extraction, and obtain the deep representation corresponding to each token sequence.

[0081] Step S2023, determining the output feature corresponding to the special mark as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

[0082] In an embodiment of the present invention, the input text sequence is first divided into a token sequence, and a special marker, namely [CLS] token, is added before the sequence. Each token sequence is converted into a word embedding and input into the BERT pre-trained language model. After feature extraction of multiple Transformer blocks of the BERT pre-trained language model, the deep representation of each token is output. Finally, the output feature representation corresponding to [CLS] is used as the input of a fully connected classifier (single linear layer) to obtain the instruction classification result corresponding to the speech to be recognized.

[0083] By splitting the text sequence, the token sequence obtained by the splitting is input into the BERT pre-trained language model for feature extraction to determine the instruction category of the voice to be recognized, thereby facilitating the identification of the instructions of the voice to be recognized.

[0084] Step S203: Use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model.

[0085] Specifically, the above step S203 includes:

[0086] Step S2031, perform knowledge distillation on the fine-tuned BERT pre-trained language model and compress the teacher model into a student model.

[0087] In the embodiment of the present invention, since voice command recognition has certain requirements for real-time performance, scenarios with high real-time requirements require as low a delay as possible. For example, in special scenarios, the result needs to be recognized within 1 second. In order to improve the speed of voice recognition, the sequence classification module is compressed so that it can classify text sequences faster to meet the real-time requirements.

[0088] By using knowledge distillation to compress the BERT pre-trained language model into a smaller model, the delay of speech recognition is reduced and the speed of speech recognition is increased.

[0089] Model compression is an important machine learning technology that aims to reduce the size of deep learning models and improve their operating efficiency while maintaining or only slightly reducing the performance of the model. This is particularly important for deploying advanced artificial intelligence technologies to devices with limited computing resources. Table 1 introduces the main methods of model compression.

[0090] Table 1

[0091]

[0092]

[0093] In order to maintain the performance of the compressed model as much as possible, the embodiment of the present invention selects knowledge distillation.

[0094] Specifically, knowledge distillation in the model compression method is used to transfer the knowledge of the model to a smaller model. The distilled model has faster inference speed and is easier to deploy.

[0095] The models before and after distillation are called Teacher and Student respectively. The network structure of Student is similar to that of Teacher, where the input and output structures are the same as Teacher, but it has fewer hidden layers and smaller hidden layer dimensions, so that it has fewer learnable parameters.

[0096] Step S2032, calculate the loss function of the compressed student model, and use the calculation result to adjust the model parameters of the student model until the model loss is within the loss threshold range.

[0097] In the embodiment of the present invention, the training target of the Student includes two parts: hard labels and soft labels. The hard label is the category label of the data; the soft label is the probability distribution P obtained by the logits output by the Teacher through the softmax function with the temperature parameter T. During training, the predicted distribution p of the soft label output by the Student is also obtained through the softmax function with the temperature parameter T. The specific loss function is:

[0098]

[0099]

[0100] in, is the cross entropy loss corresponding to the hard target, is the cross entropy loss corresponding to the soft target, y is the category label of sample x, and y k is the k-th category label, K is the total number of categories, q is the output distribution obtained by the Student through the softmax function (T = 1), and α∈[0,1] controls the two-part loss ( and ) weight, T is the above temperature parameter.

[0101] The model parameters are adjusted using the loss function calculation results until the model loss is within the loss threshold range (1% to 2%).

[0102] It should be noted that in order to better utilize the knowledge learned by the pre-trained model, the word embedding part of the Student network directly uses the weights of the pre-trained model, and these parameters are fixed and not updated during the model compression stage. When encountering words that have not appeared in the dataset but are included in the pre-trained vocabulary, these words will still be represented by semantically appropriate vectors, thereby obtaining reasonable inference results.

[0103] Step S204: extract features from the text sequence using a sequence classification model to obtain a command category label corresponding to the speech to be recognized.

[0104] Please see Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0105] The voice command recognition method based on the BERT pre-trained language model provided in this embodiment adjusts the student model according to the loss function calculation result of the compressed student model, so that the model loss is within the loss threshold range, thereby improving the recognition ability of the model and improving the generalization ability of the model.

[0106] The data set of the fine-tuned large language model used in the embodiment of the present invention contains 16 types of military tactical action text instructions, which are used to simulate the combat instructions issued by the commander in the battlefield environment. Each type of instruction contains an average of 16 data. Some instructions of the first five categories are shown in Table 2.

[0107] Table 2

[0108]

[0109]

[0110] The speech dataset contains a total of one hundred speech data, which are obtained by transcribing the above text instructions into audio format.

[0111] The process of voice command recognition based on the BERT pre-trained language model mainly includes two parts: voice command recognition and model compression.

[0112] 1. Voice command recognition

[0113] Speech command recognition is an audio classification task, the goal of which is to assign a category label to a piece of speech command data. Figure 2 As shown in the figure, an automatic speech recognition model is used to convert speech input into a text sequence, and then a sequence classification module is used to extract and classify the text features to complete the recognition of the command. The automatic speech recognition module uses Whisper, and the sequence classification module is obtained by fine-tuning the pre-trained language model BERT-base on the text instruction dataset, and then compressing the fine-tuned model through the knowledge distillation method.

[0114] The BERT-base+ classifier structure for sequence classification is as follows Figure 3 As shown in the figure, the input text is first divided into token sequences, and a special [CLS] token is added before the sequence. Each token is converted into a word embedding and input into BERT. After feature extraction by multiple Transformer blocks of BERT, the deep representation of each token is output. Finally, the output feature representation corresponding to [CLS] is used as the input of the fully connected classifier (single linear layer) to obtain the classification result of the instruction.

[0115] Among them, each category in the above-constructed dataset is randomly divided into training set and test set in an 8:2 ratio, and the sequence classification model is fully fine-tuned and partially fine-tuned on this dataset. Partial fine-tuning refers to training only the fully connected layer part of the classifier and fixing the parameters of the pre-trained model.

[0116] 2. Model Compression

[0117] Speech command recognition has certain requirements for real-time performance. Scenarios with high real-time requirements require the lowest possible latency, for example, the result needs to be recognized within 1 second. In order to improve the reasoning speed of the speech command recognition system, the sequence classification module will be compressed so that it can classify text sequences faster to meet the real-time requirements. Specifically, the knowledge distillation in the model compression method is used to transfer the knowledge of the model to a smaller model. The distilled model has a faster reasoning speed and is easier to deploy.

[0118] The models before and after distillation are called Teacher and Student respectively. The network structure of Student is similar to that of Teacher, where the input and output structures are the same as Teacher, but it has fewer hidden layers and smaller hidden layer dimensions, so that it has fewer learnable parameters. The training objectives of Student include two parts: hard labels and soft labels. The hard label is the category label of the data; the soft label is the probability distribution P obtained by the logits output by Teacher through the softmax function with temperature parameter T. During training, the predicted distribution p of the soft label output by Student is also obtained through the softmax function with temperature parameter T, such as Figure 4 shown.

[0119] The voice command recognition method based on the BERT pre-trained language model provided in the embodiment of the present invention was tested, and the test results are as follows:

[0120] 1. Bert-base large model fine-tuning training results

[0121] After 100 rounds of training iterations (each small batch of samples is trained as a step), the training loss and test accuracy curves of full fine-tuning and partial fine-tuning are as follows: Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown, it can be seen that the recognition effect of full fine-tuning is significantly better than that of partial fine-tuning, and the accuracy rate can reach 96% on the test set.

[0122] 2. Automatic speech recognition results

[0123] Table 4 shows the classification accuracy of different size versions of Whisper and Faster Whisper. The prompt parameter represents the initial prompt command parameter of the model, which guides the output of the model. 0, 1, 2, and 3 represent four prompt commands (see Table 3). The prompt words are from few to many, and more guide our dataset. Adding zh means setting the output to Chinese. Only FasterWhisper has this parameter. Time represents the time to recognize a voice command in seconds (this result is obtained by testing NVIDIA P10016 GB GPU). The Accuracy parameter represents the final classification accuracy under 100 voice messages in the test set. Teacher and Student represent the classification accuracy of the original model and the distilled model under 100 voice commands in the test set.

[0124] Table 3

[0125]

[0126]

[0127] Table 4

[0128]

[0129] From the analysis of the table, the reasoning speed and accuracy of different versions of Whisper are significantly affected by the prompt words. The richer the prompt words, the slower the reasoning speed and the higher the accuracy. At the same time, the larger the model, the higher the accuracy but the lower the reasoning speed. The reasoning speed of Faster Whisper becomes slightly slower as the length of the prompt words changes, but the accuracy is significantly improved. Finally, after comprehensively observing the reasoning accuracy of its different versions on Student, we selected FasterWhisper-largev2 as the final version, with an inference speed of around 1 second and an accuracy of 95%.

[0130] 3. Model compression test results

[0131] When compressing the model, the network structure of Student is similar to that of Teacher, but the number of layers of Transformer encoding blocks, the number of multi-head attention heads, and the number of hidden neurons in the fully connected feedforward network are less, as shown in Table 5. Here, α and T are set to 0.3 and 5 respectively during training, the number of training rounds is 100, and the optimizer uses AdamW (β1=0.9, β2=0.999, ε=10 -8 ,λ=10 -2 , lr=5×10 -5), the learning rate increases linearly from 0 to lr in the first 20 rounds, and then decreases linearly to 0 in the next 80 rounds. The Student model obtained by distillation achieves 100% training accuracy and 96.15% test accuracy on the dataset shown in Table 3.1, which is comparable to the performance of Teacher. The number of parameters of Teacher and Student is 113M and 31.6M respectively, of which the number of learnable parameters of Student is 4.7M, and the remaining parameters not involved in training mainly come from the word embedding part. Under the same computing environment, the average inference time of Teacher and Student for each data on the dataset is 23.39ms and 3.01ms respectively. The inference speed of the model after compression is significantly improved.

[0132] Table 5

[0133]

[0134] After obtaining the Student model through knowledge distillation, we also need to add it to Figure 2 The overall performance of audio command classification is evaluated in the voice command recognition system shown in Table 1. Based on the data set shown in Table 1, we recorded the corresponding voice data to evaluate the performance of the system. The system's speech recognition module uniformly uses FasterWhisper-large v2 (and uses the same prompt words), and the sequence classification module achieves accuracies of 96.74% and 95.65% when using Teacher and Student respectively (see Table 5 for the performance of other speech recognition models). The performance loss of the compressed model is small, and the accuracy is still high and maintained above 95%.

[0135] The system's inference speed is affected by the specific GPU / CPU model. We finally chose to test it on the more conventional NVIDIA RTX 2080Ti 11GB GPU. On this device, we achieved an average inference time of less than 1 second (the average inference time was 0.73 seconds), meeting the real-time requirements.

[0136] In this embodiment, a voice command recognition device based on a BERT pre-trained language model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0137] This embodiment provides a voice command recognition device based on a BERT pre-trained language model. Fig. 9 As shown, including:

[0138] The conversion module 901 is used to receive the speech to be recognized and convert the speech to be recognized into a text sequence through an automatic speech recognition model.

[0139] The classification module 902 is used to fine-tune the BERT pre-trained language model and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized.

[0140] The compression module 903 is used to compress the fine-tuned BERT pre-trained language model by using knowledge distillation to obtain a sequence classification model.

[0141] The feature extraction module 904 is used to extract features from the text sequence using a sequence classification model to obtain an instruction category label corresponding to the speech to be recognized.

[0142] In some optional implementations, the conversion module 901 includes:

[0143] The conversion unit is used to convert the speech to be recognized into a corresponding text sequence using the encoder-decoder Transformer model.

[0144] In some optional implementations, the classification module 902 includes:

[0145] The segmentation unit is used to segment the text sequence into a token sequence and add a special marker before the token sequence.

[0146] The input unit is used to convert the token sequence into word embedding, which is input into the BERT pre-trained language model. The Transformer encoding block in the BERT pre-trained language model is used for feature extraction to obtain the deep representation corresponding to each token sequence.

[0147] The classification unit is used to use the output features corresponding to the special mark as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

[0148] In some optional implementations, the compression module 903 includes:

[0149] The compression unit is used to perform knowledge distillation on the fine-tuned BERT pre-trained language model and compress the teacher model into a student model.

[0150] In some optional embodiments, the device further comprises:

[0151] The adjustment module is used to calculate the loss function of the compressed student model and use the calculation result to adjust the model parameters of the student model until the model loss is within the loss threshold range.

[0152] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0153] The voice command recognition device based on the BERT pre-trained language model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0154] The embodiment of the present invention also provides a computer device having the above Fig. 9 The voice command recognition device based on the BERT pre-trained language model is shown.

[0155] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0156] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0157] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0158] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0160] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0161] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, etc. The output device 40 can include a display device, etc.

[0162] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0163] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0164] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope of the present application.

Claims

1. A voice command recognition method based on a BERT pre-trained language model, characterized in that: The method comprises: Receiving speech to be recognized, and converting the speech to be recognized into a text sequence through an automatic speech recognition model; Fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain an instruction category corresponding to the speech to be recognized; Use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model; The sequence classification model is used to extract features of the text sequence to obtain the instruction category label corresponding to the speech to be recognized.

2. The method according to claim 1, characterized in that The automatic speech recognition model is a Whisper automatic speech recognition model, which adopts an encoder-decoder Transformer model. The automatic speech recognition model is used to convert the speech to be recognized into a text sequence, including: The encoder-decoder Transformer model is used to convert the speech to be recognized into a corresponding text sequence.

3. The method according to claim 1, characterized in that The text sequence is classified using the fine-tuned BERT pre-trained language model to obtain the instruction category corresponding to the speech to be recognized, including: Splitting the text sequence into token sequences and adding special markers before the token sequences; Convert the token sequence into word embeddings, input them into the BERT pre-trained language model, and use the Transformer encoding block in the BERT pre-trained language model to perform feature extraction to obtain the deep representation corresponding to each token sequence; The output features corresponding to the special tags are used as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

4. The method according to claim 1, characterized in that: The method of using knowledge distillation to compress the fine-tuned BERT pre-trained language model includes: Knowledge distillation is performed on the fine-tuned BERT pre-trained language model to compress the teacher model into a student model.

5. The method according to claim 4, characterized in that After compressing the teacher model into a student model, the method further comprises: The loss function of the compressed student model is calculated, and the model parameters of the student model are adjusted using the calculation results until the model loss is within the loss threshold range.

6. The method according to claim 5, characterized in that The loss function is calculated according to the following formula: in, is the loss function, is the cross entropy loss corresponding to the hard target, is the cross entropy loss corresponding to the soft target, y is the category label of sample x, y k is the k-th category label, K is the total number of categories, q is the output distribution of the student model obtained by the softmax function (T=1), α∈[0,1], and T is the temperature parameter.

7. A voice command recognition device based on a BERT pre-trained language model, characterized in that: The device comprises: A conversion module, used for receiving the speech to be recognized and converting the speech to be recognized into a text sequence through an automatic speech recognition model; A classification module, used to fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain an instruction category corresponding to the speech to be recognized; The compression module is used to compress the fine-tuned BERT pre-trained language model using knowledge distillation to obtain a sequence classification model; The feature extraction module is used to extract features from the text sequence using a sequence classification model to obtain an instruction category label corresponding to the speech to be recognized.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the voice command recognition method based on the BERT pre-trained language model described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the voice command recognition method based on the BERT pre-trained language model according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the voice instruction recognition method based on the BERT pre-trained language model according to any one of claims 1 to 6.

Images