A voice command recognition method based on BERT pre-trained language model

Abstract

The present invention relates to the field of speech recognition technology and discloses a method for voice command recognition based on a BERT pre-trained language model. The method comprises: receiving a speech to be recognized and converting the speech to be recognized into a text sequence through an automatic speech recognition model; fine-tuning the BERT pre-trained language model and performing text classification on the text sequence using the fine-tuned BERT pre-trained language model to obtain a command category corresponding to the speech to be recognized; performing model compression on the fine-tuned BERT pre-trained language model using knowledge distillation to obtain a sequence classification model; and performing feature extraction on the text sequence using the sequence classification model to obtain a command category label corresponding to the speech to be recognized. Taking into account the fact that audio data is more difficult to obtain than text data, the present invention selects a BERT pre-trained language model to improve classification accuracy, performs model compression using knowledge distillation, recognizes the voice command category label to be recognized, and provides data support for human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech command recognition method based on a BERT pre-trained language model. Background Art

[0002] Short voice command recognition is widely used in everyday life, office work, military affairs, and other fields. The higher the recognition accuracy, the more convenient and practical human-computer interaction can be. Voice commands are short, repetitive, and lack context. Traditional end-to-end voice command recognition methods primarily extract spectral features from audio. However, they lack understanding of semantic relevance when faced with commands that have never appeared before but have the same meaning, making them difficult to generalize and universally apply.

[0003] The rapid development of smart mobile devices and cloud computing has significantly propelled the advancement of artificial intelligence (AI), making it a hallmark of the technological landscape. AI has already had a profound impact on our daily lives and military operations. Intelligent robots, smart homes, augmented reality devices, self-driving cars, automated digital industries, unmanned logistics, and smart healthcare have become integral to human life. Language serves as a bridge for interpersonal communication, its rich semantic information facilitating mutual understanding and collaboration. In the field of AI, convenient human-computer interaction is crucial to the user experience, and at the core of all this lies language. Accurate robot execution requires a thorough understanding of commands. The core of speech recognition technology is the ability to accurately recognize and execute commands with varying timbre and expression, but with the same meaning. Therefore, speech recognition has become a key research area.

[0004] Short voice command recognition technology is a key branch of speech recognition technology, focusing on identifying specific commands or requests from short user-provided voice clips. Voice technology plays a key role in various fields, including mobile devices, home appliances, healthcare, and the military. In the mobile device sector, voice-based applications such as Apple's Siri, Huawei's Xiaoyi, and Xiaomi's Xiaoai, as well as voice search and SMS dictation features, have become deeply integrated into daily life, all thanks to advanced speech recognition technology. In smart homes, users can control appliances with simple commands such as "turn on the air conditioner" or "turn off the TV." Devices like the Xiaoai speaker have become standard equipment in many households. In the medical field, voice-controlled devices provide convenience and improve operational efficiency for people with disabilities and mobility impairments. In military applications, short voice control commands enable convenient operation of equipment such as drones.

[0005] As an audio classification task, command recognition typically uses a single model that takes audio data as input and directly outputs the corresponding category. This approach presents some challenges when recognizing short voice commands. Voice commands, used to convey relatively deterministic information, are typically short, repetitive, and lack context. Furthermore, datasets are limited, so directly training a single model cannot fully understand the audio semantics and has limited generalization capabilities. Summary of the Invention

[0006] In view of this, the present invention provides a voice command recognition based on the BERT pre-trained language model to solve the problem that a single model is used for voice recognition in the prior art, which has the problem of being unable to fully understand the audio semantics and having weak generalization ability.

[0007] In a first aspect, the present invention provides a method for voice command recognition based on a BERT pre-trained language model, the method comprising:

[0008] Receive the speech to be recognized and convert it into a text sequence through an automatic speech recognition model;

[0009] Fine-tune the BERT pre-trained language model and use it to classify the text sequence to obtain the instruction category corresponding to the speech to be recognized;

[0010] Use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model;

[0011] The sequence classification model is used to extract features from the text sequence to obtain the instruction category label corresponding to the speech to be recognized.

[0012] The present invention automatically identifies the speech to be recognized through an automatic speech recognition model, converts the speech to be recognized into a text sequence, and then uses the BERT pre-trained language model to perform text classification on the text sequence. In order to fully consider the fact that audio data is more difficult to obtain than text data, the BERT pre-trained language model is selected instead of the speech recognition model to improve the classification accuracy. The model is compressed by knowledge distillation into a smaller model to reduce the model size and improve the recognition efficiency while ensuring the accuracy rate. The sequence classification model is used to extract features from the text sequence to accurately identify the instruction category label corresponding to the speech to be recognized, thereby completing the voice instruction recognition and providing data support for human-computer interaction.

[0013] In an optional embodiment, the automatic speech recognition model is a Whisper automatic speech recognition model, which uses an encoder-decoder Transformer model to convert the speech to be recognized into a text sequence through the automatic speech recognition model, including:

[0014] The encoder-decoder Transformer model is used to convert the speech to be recognized into the corresponding text sequence.

[0015] The present invention recognizes speech by adopting the Transformer model of the encoder-decoder in the Whisper automatic speech recognition model, and converts the speech to be recognized into a text sequence, so as to give full play to the accuracy and robustness of the Whisper automatic speech recognition model in speech recognition that is close to the human level.

[0016] In an optional embodiment, the fine-tuned BERT pre-trained language model is used to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized, including:

[0017] Split the text sequence into token sequences and add special markers before the token sequences;

[0018] Convert the token sequence into word embeddings and input them into the BERT pre-trained language model. Use the Transformer encoding block in the BERT pre-trained language model to perform feature extraction and obtain the deep representation corresponding to each token sequence.

[0019] The output features corresponding to the special tags are used as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

[0020] The present invention splits the text sequence and inputs the token sequence obtained by the split into a BERT pre-trained language model for feature extraction to determine the instruction category of the speech to be recognized, thereby facilitating the recognition of the instruction of the speech to be recognized.

[0021] In an optional embodiment, knowledge distillation is used to perform model compression on the fine-tuned BERT pre-trained language model, including:

[0022] Knowledge distillation is performed on the fine-tuned BERT pre-trained language model to compress the teacher model into a student model.

[0023] The present invention uses knowledge distillation to compress the BERT pre-trained language model into a smaller model, thereby reducing the delay of speech recognition and improving the speed of speech recognition.

[0024] In an optional embodiment, after compressing the teacher model into a student model, the method further includes:

[0025] Calculate the loss function of the compressed student model and use the calculation results to adjust the model parameters of the student model until the model loss is within the loss threshold range.

[0026] The present invention adjusts the student model according to the loss function calculation result of the compressed student model, so that the model loss is within the loss threshold range, thereby improving the recognition ability of the model and the generalization ability of the model.

[0027] In an optional embodiment, the loss function is calculated according to the following formula:

[0028]

[0029] in, is the loss function, is the cross entropy loss corresponding to the hard target, is the cross entropy loss corresponding to the soft target, y is the category label of sample x, y k is the k-th category label, K is the total number of categories, q is the output distribution of the student model obtained by the softmax function (T = 1), α∈[0,1], and T is the temperature parameter.

[0030] The present invention calculates the loss function so as to adjust the model according to the loss function, thereby improving the generalization ability of the model.

[0031] In a second aspect, the present invention provides a voice command recognition device based on a BERT pre-trained language model, the device comprising:

[0032] A conversion module is used to receive the speech to be recognized and convert the speech to be recognized into a text sequence through an automatic speech recognition model;

[0033] The classification module is used to fine-tune the BERT pre-trained language model and use the fine-tuned BERT pre-trained language model to classify text sequences to obtain the instruction category corresponding to the speech to be recognized;

[0034] The compression module is used to compress the fine-tuned BERT pre-trained language model using knowledge distillation to obtain a sequence classification model;

[0035] The feature extraction module is used to extract features from the text sequence using the sequence classification model to obtain the instruction category label corresponding to the speech to be recognized.

[0036] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the voice command recognition method based on the BERT pre-trained language model of the above-mentioned first aspect or any corresponding embodiment thereof.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the voice command recognition method based on the BERT pre-trained language model of the above-mentioned first aspect or any corresponding embodiment thereof.

[0038] In a fifth aspect, the present invention provides a computer program product comprising computer instructions, which are used to enable a computer to execute the voice command recognition method based on the BERT pre-trained language model of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 1 is a flow chart of a method for voice command recognition based on a BERT pre-trained language model according to an embodiment of the present invention;

[0041] Figure 2 2 is a framework diagram of a voice command recognition system based on a BERT pre-trained language model according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a BERT-based sequence classification model architecture according to an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of the knowledge distillation principle according to an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of training and accuracy changes according to an embodiment of the present invention;

[0045] Figure 6 is a schematic diagram of training and loss changes according to an embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of the test and accuracy changes according to an embodiment of the present invention;

[0047] Figure 8 is a schematic diagram of testing and loss changes according to an embodiment of the present invention;

[0048] Figure 9 2 is a block diagram of a voice command recognition device based on a BERT pre-trained language model according to an embodiment of the present invention;

[0049] Figure 10 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0051] The following is an introduction to the current research status:

[0052] Early speech recognition systems relied on simple pattern matching techniques, later evolving to use Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs). With the rise of machine learning, particularly deep learning, speech recognition technology has achieved revolutionary progress. Modern speech recognition systems commonly employ deep neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTMs), and the Transformer model.

[0053] Later, OpenAI open-sourced the Whisper automatic speech recognition model. Whisper is a general speech recognition model that uses a large amount of multilingual and multi-task supervised data for training, and can achieve robustness and accuracy close to human levels in English speech recognition.

[0054] Whisper can also perform tasks such as multilingual speech recognition, speech translation, and language identification. Its architecture is a simple end-to-end approach that uses an encoder-decoder Transformer model to convert the input audio into a corresponding text sequence and specify different tasks based on special tags.

[0055] Large language models are primarily based on the Transformer architecture. The Transformer's key innovation is the self-attention mechanism, which allows the model to automatically determine which parts of the input sequence to focus on when processing input data. Since then, several high-performance LLMs (Large Language Models) have emerged, including the GPT series, BERT and T5, as well as BART and RoBERTa. These models use large amounts of data for pre-training, greatly improving their ability to understand semantic information in text. They have achieved excellent results in multiple NLP (Natural Language Processing) tasks, breaking state-of-the-art results in multiple fields. The BERT language model's bidirectional contextual understanding, as well as pre-training tasks for the Masked Language Model (MLM) and Next Sentence Prediction (NSP), help it learn deeper linguistic patterns and obtain word embedding representations with high semantic relationships.

[0056] Model compression is an important machine learning technology that aims to reduce the size of deep learning models and improve their operating efficiency while maintaining or slightly reducing the performance of the model. This is particularly important for deploying advanced artificial intelligence technologies to devices with limited computing resources.

[0057] According to an embodiment of the present invention, an embodiment of a method for voice instruction recognition based on a BERT pre-trained language model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0058] In this embodiment, a voice command recognition method based on the BERT pre-trained language model is provided. Figure 1 is a flow chart of a method for voice command recognition based on a BERT pre-trained language model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0059] Step S101: receiving a speech to be recognized and converting the speech to be recognized into a text sequence through an automatic speech recognition model.

[0060] In the embodiment of the present invention, in the currently mature natural language processing (NLP) technology, large language models are pre-trained on large text corpora, gaining a full understanding of the semantics of natural language texts. Transfer learning is used to improve the performance of downstream tasks. Therefore, the rich semantic knowledge of the large model can be used to better represent short text instructions. Therefore, in order to take advantage of the pre-trained language model, the embodiment of the present invention first converts the speech to be recognized into a text sequence.

[0061] During the human-computer interaction process, the voice audio input by the user is received. The voice is the voice to be recognized. The function of the automatic speech recognition model is to perform automatic speech recognition, recognize the voice to be recognized through the automatic speech recognition model, and convert the voice to be recognized into a corresponding text sequence to facilitate the analysis of the voice to be recognized.

[0062] Step S102: fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized.

[0063] In an embodiment of the present invention, considering that the pre-trained large language model in natural language processing contains word embedding representations with strong semantic relationships, it can accurately understand voice instructions of different statements. Therefore, automatic speech recognition is combined with the BERT pre-trained language model for speech recognition. The speech to be recognized is first converted into a text sequence, and then the BERT pre-trained language model is fine-tuned. The text sequence is parsed into the corresponding instruction category by the fine-tuned BERT pre-trained language model. After comparing the effects and speeds of different speech recognition models, the model is fine-tuned on a dataset designed to simulate real-world scenarios in specific scenarios.

[0064] It should be noted that, considering that audio data is more difficult to obtain than text data and text data is richer than audio data in the pre-training stage, a language model is selected instead of a speech recognition model.

[0065] Step S103: Use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model.

[0066] In this embodiment of the present invention, due to the large size of the pre-trained language model, the inference speed of the pre-trained language model cannot meet the real-time requirements of speech recognition. Therefore, considering the difficulty of model deployment under resource constraints, knowledge distillation is used to compress the fine-tuned BERT pre-trained language model into a smaller model. This reduces the model size while maintaining recognition accuracy and further improves recognition speed.

[0067] Step S104: Using a sequence classification model to perform feature extraction on the text sequence, obtain an instruction category label corresponding to the speech to be recognized.

[0068] In an embodiment of the present invention, a sequence classification model is used to extract features from a text sequence for classification, and an instruction category label is assigned to the speech to be recognized, thereby completing the recognition of the speech to be recognized.

[0069] The voice command recognition method based on the BERT pre-trained language model provided in this embodiment automatically recognizes the voice to be recognized through an automatic speech recognition model, converts the voice to be recognized into a text sequence, and then uses the BERT pre-trained language model to perform text classification on the text sequence, taking into full consideration the fact that audio data is more difficult to obtain than text data. The BERT pre-trained language model is selected to replace the voice recognition model to improve classification accuracy, and knowledge distillation is used to compress the model into a smaller model to reduce the model size and improve recognition efficiency while ensuring accuracy. The sequence classification model is used to extract features from the text sequence to accurately identify the instruction category label corresponding to the voice to be recognized, thereby completing voice command recognition and providing data support for human-computer interaction.

[0070] This embodiment provides a method for voice command recognition based on a BERT pre-trained language model. The process includes the following steps:

[0071] Step S201: receiving a speech to be recognized and converting the speech to be recognized into a text sequence through an automatic speech recognition model.

[0072] Specifically, the above step S201 includes:

[0073] Step S2011: Use the encoder-decoder Transformer model to convert the speech to be recognized into a corresponding text sequence.

[0074] In this embodiment of the present invention, the automatic speech recognition module uses the Whisper model, which can quickly and accurately recognize speech information. The original Whisper model includes five large and small versions, as well as a FasterWhisper version with improved inference speed. After testing these versions for recognition accuracy and speed under different prompt words, this embodiment of the present invention selected the improved version, Faster Whisper-largev2.

[0075] The Whisper model is a general-purpose speech recognition model trained using a large amount of multilingual and multi-task supervised data. It achieves near-human-level robustness and accuracy in English speech recognition. The Whisper model can also perform tasks such as multilingual speech recognition, speech translation, and language identification. The Whisper model architecture is a simple end-to-end approach that uses an encoder-decoder Transformer model to convert the input speech to be recognized into a corresponding text sequence, specifying different tasks based on special tags.

[0076] By using the encoder-decoder Transformer model in the Whisper automatic speech recognition model to recognize speech, the speech to be recognized is converted into a text sequence, so as to enable the Whisper automatic speech recognition model to achieve accuracy and robustness close to human levels in speech recognition.

[0077] Step S202: fine-tune the BERT pre-trained language model, and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized.

[0078] Specifically, in step S202 above, the text sequence is classified using the fine-tuned BERT pre-trained language model to obtain the instruction category corresponding to the speech to be recognized, including:

[0079] Step S2021: split the text sequence into token sequences and add special tags before the token sequences.

[0080] In step S2022, the token sequence is converted into word embeddings and input into the BERT language pre-training model. The Transformer encoding block in the BERT pre-training language model is used to perform feature extraction to obtain the deep representation corresponding to each token sequence.

[0081] In step S2023, the output feature corresponding to the special mark is determined as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

[0082] In an embodiment of the present invention, the input text sequence is first divided into a token sequence, and a special marker, namely [CLS] token, is added before the sequence. Each token sequence is converted into a word embedding and input into the BERT pre-trained language model. After feature extraction of multiple Transformer blocks of the BERT pre-trained language model, a deep representation of each token is output. Finally, the output feature representation corresponding to [CLS] is used as the input of a fully connected classifier (a single linear layer) to obtain the instruction classification result corresponding to the speech to be recognized.

[0083] By splitting the text sequence, the token sequence obtained by the split is input into the BERT pre-trained language model for feature extraction to determine the instruction category of the speech to be recognized, thereby facilitating the identification of the instruction of the speech to be recognized.

[0084] In step S203, knowledge distillation is used to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model.

[0085] Specifically, the above step S203 includes:

[0086] Step S2031: perform knowledge distillation on the fine-tuned BERT pre-trained language model to compress the teacher model into a student model.

[0087] In this embodiment of the present invention, voice command recognition requires real-time performance. Scenarios with high real-time requirements require minimal latency. For example, in special scenarios, recognition results may be required within one second. To increase speech recognition speed, the sequence classification module is compressed, enabling it to classify text sequences more quickly to meet real-time requirements.

[0088] By using knowledge distillation to compress the BERT pre-trained language model into a smaller model, the delay of speech recognition is reduced and the speed of speech recognition is increased.

[0089] Model compression is an important machine learning technique that aims to reduce the size of deep learning models and improve their operational efficiency while maintaining or only slightly degrading model performance. This is particularly important for deploying advanced AI technologies on devices with limited computing resources. Table 1 introduces the main model compression methods.

[0090] Table 1

[0091]

[0092]

[0093] In order to maintain the performance of the compressed model as much as possible, the embodiment of the present invention uses knowledge distillation.

[0094] Specifically, knowledge distillation in the model compression method is used to transfer the model's knowledge to a smaller model. The distilled model has faster inference speed and is easier to deploy.

[0095] The models before and after distillation are called Teacher and Student, respectively. The network structure of Student is similar to that of Teacher, with the same input and output structures as Teacher, but with fewer hidden layers and smaller hidden layer dimensions, resulting in fewer learnable parameters.

[0096] Step S2032: Calculate the loss function of the compressed student model and use the calculation result to adjust the model parameters of the student model until the model loss is within the loss threshold range.

[0097] In this embodiment of the present invention, the training objective of the Student consists of two parts: hard labels and soft labels. The hard labels are the category labels of the data; the soft labels are the probability distribution P obtained by applying the softmax function with a temperature parameter T to the logits output by the Teacher. During training, the predicted distribution p of the soft labels output by the Student is also obtained by applying the softmax function with a temperature parameter T. The specific loss function is:

[0098]

[0099]

[0100] in, is the cross entropy loss corresponding to the hard target, is the cross entropy loss corresponding to the soft target, y is the category label of sample x, y k is the k-th category label, K is the total number of categories, q is the output distribution obtained by Student through the softmax function (T = 1), and α∈[0,1] controls the two-part loss ( and ) weight, T is the above temperature parameter.

[0101] The model parameters are adjusted using the loss function calculation results until the model loss is within the loss threshold range (1% to 2%).

[0102] It's important to note that to better leverage the knowledge learned by the pre-trained model, the word embeddings in the Student network directly use the pre-trained model's weights, and these parameters are fixed and not updated during the model compression phase. When encountering words that don't appear in the dataset but are included in the pre-trained vocabulary, these words will still receive semantically appropriate vector representations, enabling reasonable inference results.

[0103] Step S204: Using a sequence classification model to perform feature extraction on the text sequence, obtain an instruction category label corresponding to the speech to be recognized.

[0104] Please see Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0105] The voice command recognition method based on the BERT pre-trained language model provided in this embodiment adjusts the student model according to the loss function calculation result of the compressed student model, so that the model loss is within the loss threshold range, thereby improving the recognition ability of the model and the generalization ability of the model.

[0106] The dataset used to fine-tune the large language model used in this embodiment of the present invention contains 16 categories of military tactical action text instructions, used to simulate combat orders issued by commanders in a battlefield environment. Each category of instructions contains an average of 16 data items. Table 2 shows some of the instructions from the first five categories.

[0107] Table 2

[0108]

[0109]

[0110] The speech dataset contains a total of one hundred speech data, which are obtained by transcribing the above text instructions into audio format.

[0111] The process of voice command recognition based on the BERT pre-trained language model mainly includes two parts: voice command recognition and model compression.

[0112] 1. Voice command recognition

[0113] Speech command recognition is an audio classification task, the goal of which is to assign a category label to a speech command data. Figure 2 As shown, an automatic speech recognition model converts speech input into a text sequence. A sequence classification module then extracts and classifies the text to complete command recognition. The automatic speech recognition module uses Whisper, and the sequence classification module fine-tunes the pre-trained language model BERT-base on a text command dataset and compresses the fine-tuned model using knowledge distillation.

[0114] The BERT-base+classifier structure for sequence classification is as follows Figure 3 As shown, the input text is first segmented into a sequence of tokens, and a special [CLS] token is prepended to the sequence. Each token is converted into a word embedding and input into BERT. After feature extraction through BERT's multiple Transformer blocks, a deep representation of each token is output. Finally, the output feature representation corresponding to [CLS] is used as the input to a fully connected classifier (a single linear layer) to obtain the instruction classification result.

[0115] Among them, each category in the above-constructed dataset is randomly divided into training set and test set according to 8:2, and the sequence classification model is fully fine-tuned and partially fine-tuned on this dataset. Partial fine-tuning refers to training only the fully connected layer part of the classifier and fixing the parameters of the pre-trained model.

[0116] 2. Model Compression

[0117] Speech command recognition has certain real-time requirements. Scenarios with high real-time requirements require the lowest possible latency, for example, requiring recognition results within 1 second. To improve the inference speed of the speech command recognition system, the sequence classification module is compressed, enabling it to classify text sequences more quickly to meet real-time requirements. Specifically, knowledge distillation, a model compression method, is used to transfer the model's knowledge to a smaller model. The resulting distilled model has faster inference speed and is easier to deploy.

[0118] The models before and after distillation are called Teacher and Student respectively. The network structure of Student is similar to that of Teacher, where the input and output structures are the same as Teacher, but it has fewer hidden layers and smaller hidden layer dimensions, so that it has fewer learnable parameters. The training objectives of Student include two parts: hard labels and soft labels. The hard label is the category label of the data; the soft label is the probability distribution P obtained by the logits output by Teacher through the softmax function with temperature parameter T. During training, the predicted distribution p of the soft label output by Student is also obtained through the softmax function with temperature parameter T, such as Figure 4 shown.

[0119] The speech command recognition method based on the BERT pre-trained language model provided by the embodiment of the present invention was tested, and the test results are as follows:

[0120] 1. Bert-base large model fine-tuning training results

[0121] After 100 rounds of training iterations (each small batch of samples is trained as a step), the training loss and test accuracy curves of full fine-tuning and partial fine-tuning are as follows: Figure 5 、 Figure 6 、 Figure 7 and Figure 8 As shown in the figure, it can be seen that the recognition effect of full fine-tuning is significantly better than that of partial fine-tuning, and the accuracy rate can reach 96% on the test set.

[0122] 2. Automatic speech recognition results

[0123] Table 4 shows the classification accuracy of different sizes of Whisper and Faster Whisper. The prompt parameter represents the initial prompt command parameter of the model, which guides the model output. 0, 1, 2, and 3 represent four prompt commands (see Table 3), with prompt words ranging from few to many, and more guiding our dataset. Adding zh means setting the output to Chinese. Only FasterWhisper has this parameter. Time represents the time it takes to recognize a voice command in seconds (this result was tested on an NVIDIA P100 16 GB GPU). The Accuracy parameter represents the final classification accuracy under the test set of 100 voice messages. Teacher and Student represent the classification accuracy of the original model and the distilled model under the test set of 100 voice commands.

[0124] Table 3

[0125]

[0126]

[0127] Table 4

[0128]

[0129] The table shows that the inference speed and accuracy of different versions of Whisper are significantly affected by the cue words. The richer the cue words, the significantly slower the inference speed, while the accuracy improves to a certain extent. Furthermore, the larger the model, the significantly improved the accuracy but also the significantly reduced inference speed. Faster Whisper's inference speed slows slightly with the length of the cue words, but its accuracy improves significantly. Finally, after comprehensively observing the inference accuracy of different versions on Student, we selected FasterWhisper-largev2 as the final version, achieving an inference speed of around 1 second and an accuracy of 95%.

[0130] 3. Model compression test results

[0131] When compressing the model, the network structure of Student is similar to that of Teacher, but the number of layers of Transformer encoding blocks, the number of multi-head attention heads, and the number of hidden neurons in the fully connected feedforward network are smaller, as shown in Table 5. Here, during training, α and T are set to 0.3 and 5 respectively, the number of training rounds is 100, and the optimizer uses AdamW (β1 = 0.9, β2 = 0.999, ε = 10 -8 ,λ=10 -2 , lr=5×10 -5), the learning rate was linearly increased from 0 to lr in the first 20 epochs and then linearly decreased to 0 in the next 80 epochs. The resulting Student model achieved 100% training accuracy and 96.15% test accuracy on the dataset shown in Table 3.1, comparable to the performance of the Teacher. The Teacher and Student models have 113M and 31.6M parameters, respectively, of which 4.7M are learnable parameters for the Student model. The remaining untrained parameters mainly come from word embeddings. Under the same computing environment, the average inference time per data entry for the Teacher and Student models on the dataset was 23.39ms and 3.01ms, respectively, significantly improving the inference speed of the compressed model.

[0132] Table 5

[0133]

[0134] After obtaining the Student model through knowledge distillation, we also need to add it to Figure 2 The overall performance of audio command classification was evaluated in the speech command recognition system shown in Table 1. Based on the dataset shown in Table 1, we recorded corresponding speech data to evaluate the system's performance. The system's speech recognition module uniformly uses FasterWhisper-large v2 (and uses the same prompt words). The sequence classification module achieves accuracies of 96.74% and 95.65% when using Teacher and Student, respectively (see Table 5 for performance details when using other speech recognition models). The compressed model shows minimal performance loss, maintaining a high accuracy rate above 95%.

[0135] The system's inference speed is affected by the specific GPU / CPU model. We ultimately chose to test on the more conventional NVIDIA RTX 2080Ti 11GB GPU. On this device, we achieved an average inference time of less than 1 second (average inference time of 0.73 seconds), meeting the real-time requirements.

[0136] In this embodiment, a voice command recognition device based on a BERT pre-trained language model is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0137] This embodiment provides a speech instruction recognition device based on the BERT pre-trained language model. Figure 9 Shown, including:

[0138] The conversion module 901 is used to receive the speech to be recognized and convert the speech to be recognized into a text sequence through an automatic speech recognition model.

[0139] The classification module 902 is used to fine-tune the BERT pre-trained language model and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized.

[0140] The compression module 903 is used to use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model.

[0141] The feature extraction module 904 is used to extract features from the text sequence using a sequence classification model to obtain a command category label corresponding to the speech to be recognized.

[0142] In some optional embodiments, the conversion module 901 includes:

[0143] The conversion unit is used to convert the speech to be recognized into a corresponding text sequence using the encoder-decoder Transformer model.

[0144] In some optional implementations, the classification module 902 includes:

[0145] The segmentation unit is used to segment the text sequence into a token sequence and add a special marker before the token sequence.

[0146] The input unit is used to convert the token sequence into word embeddings, which are then input into the BERT pre-trained language model. The Transformer encoding block in the BERT pre-trained language model is used for feature extraction to obtain the deep representation corresponding to each token sequence.

[0147] The classification unit is used to use the output features corresponding to the special tags as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

[0148] In some optional implementations, the compression module 903 includes:

[0149] The compression unit is used to perform knowledge distillation on the fine-tuned BERT pre-trained language model and compress the teacher model into a student model.

[0150] In some optional embodiments, the device further comprises:

[0151] The adjustment module is used to calculate the loss function of the compressed student model and use the calculation results to adjust the model parameters of the student model until the model loss is within the loss threshold range.

[0152] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0153] The voice command recognition device based on the BERT pre-trained language model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0154] The embodiment of the present invention also provides a computer device having the above Figure 9 The voice command recognition device shown is based on the BERT pre-trained language model.

[0155] See also Figure 10 , Figure 10 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 10 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 10 A processor 10 is taken as an example.

[0156] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0157] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0158] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0160] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 10 The bus connection is taken as an example.

[0161] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, etc. The output device 40 can include a display device, etc.

[0162] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0163] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0164] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are intended to fall within the scope of this application.

Claims

1. A method for voice command recognition based on a BERT pre-trained language model, characterized in that: The method comprises: Receive the speech to be recognized and convert the speech to be recognized into a text sequence through an automatic speech recognition model; Fine-tuning the BERT pre-trained language model, and using the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain an instruction category corresponding to the speech to be recognized; Use knowledge distillation to compress the fine-tuned BERT pre-trained language model to obtain a sequence classification model; Using a sequence classification model to extract features from the text sequence to obtain a command category label corresponding to the speech to be recognized; The method of using knowledge distillation to compress the fine-tuned BERT pre-trained language model includes: Perform knowledge distillation on the fine-tuned BERT pre-trained language model to compress the teacher model into a student model; After compressing the teacher model into a student model, the method further includes: Calculate the loss function of the compressed student model and use the calculation results to adjust the model parameters of the student model until the model loss is within the loss threshold range; The loss function is calculated according to the following formula: in, is the loss function, is the cross entropy loss corresponding to the hard target, is the cross entropy loss corresponding to the soft target, y For samples x The category label, For the k category labels, K is the total number of categories, q is the output distribution obtained by Student through the softmax function (T=1), , Controls the weight of the cross entropy loss corresponding to hard targets and the cross entropy loss corresponding to soft targets. T is the temperature parameter and P is the probability distribution.

2. The method according to claim 1, characterized in that The automatic speech recognition model is a Whisper automatic speech recognition model, which adopts an encoder-decoder Transformer model. The automatic speech recognition model converts the speech to be recognized into a text sequence, including: The encoder-decoder Transformer model is used to convert the speech to be recognized into a corresponding text sequence.

3. The method according to claim 1, characterized in that The text sequence is classified using the fine-tuned BERT pre-trained language model to obtain the instruction category corresponding to the speech to be recognized, including: Split the text sequence into token sequences and add special markers before the token sequences; Convert the token sequence into word embeddings, input them into the BERT pre-trained language model, and use the Transformer encoding block in the BERT pre-trained language model to perform feature extraction to obtain the deep representation corresponding to each token sequence; The output features corresponding to the special tags are used as the input of the fully connected classifier to obtain the instruction category corresponding to the speech to be recognized.

4. A voice command recognition device based on a BERT pre-trained language model, characterized in that: The method for voice command recognition based on a BERT pre-trained language model according to any one of claims 1 to 3, wherein the device comprises: A conversion module, configured to receive the speech to be recognized and convert the speech to be recognized into a text sequence through an automatic speech recognition model; A classification module is used to fine-tune the BERT pre-trained language model and use the fine-tuned BERT pre-trained language model to perform text classification on the text sequence to obtain the instruction category corresponding to the speech to be recognized; The compression module is used to compress the fine-tuned BERT pre-trained language model using knowledge distillation to obtain a sequence classification model; The feature extraction module is used to extract features from the text sequence using a sequence classification model to obtain an instruction category label corresponding to the speech to be recognized.

5. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the voice command recognition method based on the BERT pre-trained language model according to any one of claims 1 to 3 by executing the computer instructions.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the voice command recognition method based on the BERT pre-trained language model according to any one of claims 1 to 3.

7. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the voice instruction recognition method based on the BERT pre-trained language model according to any one of claims 1 to 3.

Images