Command word recognition method and device and storage medium

Abstract

The invention discloses a command word recognition method and device and a storage medium. The method comprises the steps that real-time audio stream information is acquired; the real-time audio stream information is preprocessed; performing query feature recognition on the preprocessed audio stream information according to a preset feature extraction model to obtain a current query feature, the preset feature extraction model being a model obtained by training a preset student model after combining an embedded feature of a preset teacher model with corresponding classification output through a knowledge distillation method; and performing matching in a preset command word feature library according to the current query feature, and determining a target command word. According to the method, the preset feature extraction model is obtained by combining the knowledge distillation method with the metric learning training, the model feature extraction performance is ensured while the model training cost is reduced, and the target command word can be identified more quickly and accurately.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a command word recognition method, device and storage medium. Background Art

[0002] As an important means of human-computer interaction, command word recognition plays an increasingly important role in smart homes, cars, virtual reality and other fields. Especially when applied to augmented reality (AR) devices that lack traditional interaction methods such as mouse, keyboard or touch screen, in order to meet the user's lightweight, low-power, and long-lasting device usage requirements, voice interaction based on command word recognition has become an important way of interaction. In particular, considering different device usage scenarios, voice interaction is required to run in real time on the low-power platform of the AR device without a network, so the use of an offline command word recognition system has become a way to meet this interaction demand.

[0003] The offline command word recognition system in the industry includes three recognition methods: a method based on speech classification, an automatic speech recognition (ASR) method, and a method of training a feature extraction model. However, since the method based on speech classification requires the targeted collection of a large number of corpora, the cost is relatively high; the algorithm model in the ASR-based method is large and power-hungry, and it is difficult to run efficiently on a platform with lower computing power. Therefore, the method of training a feature extraction model can be used in AR devices for command word recognition. Therefore, in order to improve the accuracy of command word recognition, how to improve the feature extraction capability of the trained model has become a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a command word recognition method, device and storage medium, aiming to solve the technical problem of how to improve the feature extraction capability of the model to improve the accuracy of command word recognition when using a training feature extraction model for voice interaction in an AR device.

[0005] To achieve the above objectives, the present application proposes a command word recognition method, which includes:

[0006] Get real-time audio stream information;

[0007] Preprocessing the real-time audio stream information;

[0008] Performing query feature recognition on the preprocessed audio stream information according to a preset feature extraction model to obtain current query features, wherein the preset feature extraction model is a model obtained by training a preset student model after combining the embedded features of a preset teacher model with corresponding classification outputs through a knowledge distillation method;

[0009] The current query feature is matched in a preset command word feature library to determine the target command word.

[0010] In one embodiment, before the step of performing query feature recognition on the preprocessed audio stream information according to a preset feature extraction model to obtain the current query feature, the step includes:

[0011] Collect voice corpus information;

[0012] Performing data processing on the speech corpus information to obtain a training corpus;

[0013] Input each training sample in the training corpus into the preset teacher model respectively to obtain feature information and classification information output by the preset teacher model;

[0014] Based on the constraint condition of minimizing the angle loss function, combined with the feature information and the classification information, the preset student model is trained according to the training corpus to obtain a preset feature extraction model.

[0015] In one embodiment, the speech corpus information includes a number of conversational speech samples of different language categories, and the step of performing data processing on the speech corpus information to obtain a training corpus includes:

[0016] Performing alignment processing on each of the dialogue speech samples to obtain a cutting timestamp;

[0017] Perform word segmentation and segmentation on each of the dialogue speech samples based on the segmentation timestamp to obtain a command word dataset;

[0018] The command word data set is cleaned to obtain a training corpus.

[0019] In one embodiment, the step of training the preset student model based on the training corpus and obtaining the preset feature extraction model based on the constraint condition of minimizing the angle loss function, combining the feature information and the classification information, includes:

[0020] Performing Fbank feature transformation on each training sample in the training corpus to obtain a number of Fbank feature samples;

[0021] Input each of the Fbank feature samples into a preset student model, and train the preset student model by combining the feature information and the classification information using a knowledge distillation method;

[0022] When the minimization condition of the angle loss function is met, a preset feature extraction model is obtained, and the angle loss function is ArcFaceLoss.

[0023] In one embodiment, before the step of matching the current query feature in a preset command word feature library to determine the target command word, the step includes:

[0024] Acquire speech samples corresponding to each standard command word, extract features of a preset dimension for each speech sample, and normalize the extracted dimensional features, wherein the preset dimension is consistent with the dimension of the embedded feature;

[0025] A plurality of median dimensional features are constructed according to the dimensional features obtained by the normalization process, and a library feature corresponding to each standard command word in the preset command word feature library is determined according to each of the median dimensional features.

[0026] In one embodiment, the step of matching the current query feature in a preset command word feature library to determine the target command word includes:

[0027] Determine a similarity score between the current query feature and each of the library features;

[0028] Comparing each of the similarity scores with the activation threshold of the corresponding library feature, and obtaining a set of candidate command words according to the comparison results;

[0029] In the candidate command word set, the candidate command word corresponding to the maximum similarity score is determined as the target command word.

[0030] In one embodiment, before the step of comparing each of the similarity scores with the activation threshold of the corresponding library feature and obtaining a set of candidate command words according to the comparison results, the following steps are included:

[0031] Obtaining precision information and recall rate information of the standard command words to which each of the library features belongs;

[0032] A preset Pareto optimization algorithm is used to calculate the activation threshold corresponding to each of the library features according to the precision information and the recall rate information.

[0033] In one embodiment, the step of comparing each of the similarity scores with the activation threshold of the corresponding library feature and obtaining a set of candidate command words according to the comparison results includes:

[0034] When the similarity score is greater than the activation threshold corresponding to the library feature, the standard command word corresponding to the library feature is determined as a candidate command word.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a command word recognition device, which includes: a memory, a processor, and a command word recognition program stored in the memory and executable on the processor, and the command word recognition program, when executed by the processor, implements the steps of the command word recognition method described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a command word recognition program is stored, and when the command word recognition program is executed by a processor, the steps of the command word recognition method described above are implemented.

[0037] The present application discloses a method for command word recognition, including: obtaining real-time audio stream information; preprocessing the real-time audio stream information; performing query feature recognition on the preprocessed audio stream information according to a preset feature extraction model to obtain the current query feature, wherein the preset feature extraction model is a model obtained by training a preset student model after combining the embedded features of a preset teacher model with the corresponding classification output through a knowledge distillation method; matching the current query feature in a preset command word feature library to determine the target command word. Since the preset feature extraction model of the preprocessed audio stream information input in the present application is constructed based on a preset learning model obtained by combining knowledge distillation with metric learning training, the knowledge distillation method can reduce the training cost of the model while ensuring the performance of feature extraction, thereby enabling the matching process to more quickly identify the correct command word. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 This is a flow chart of the first embodiment of the command word recognition method of the present application;

[0041] Figure 2 This is a flow chart of the second embodiment of the command word recognition method of the present application;

[0042] Figure 3This is a schematic diagram of the model distillation framework in the command word recognition method of this application;

[0043] Figure 4 This is a flow chart of the third embodiment of the command word recognition method of the present application;

[0044] Figure 5 A schematic diagram of the full-process reasoning technology route of the command word recognition method of this application;

[0045] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the command word recognition method of this application.

[0046] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The present application embodiment provides a command word recognition method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the command word recognition method of the present application. In this embodiment, the method includes: Steps S10 to S40:

[0050] Step S10: Acquire real-time audio stream information.

[0051] It should be noted that the method of this embodiment can be applied to the voice interaction scenario of AR devices, and can also be applied to the voice interaction scenario of other smart devices. The execution subject of the method of this embodiment can be a computer service device with data processing, program running and network communication, such as a mobile phone, smart home device, head-mounted AR device, etc., and can also be other electronic devices that can achieve the same or similar functions as the present application by accessing an offline voice and offline command word recognition system. The following takes a head-mounted AR device (hereinafter referred to as "AR device") as an example to illustrate this embodiment and the following embodiments.

[0052] It is understandable that the real-time audio stream information can be a continuous audio signal collected by the AR device in real time, and the continuous audio signal can include the user's conversational voice and background noise. In order to obtain a more stable conversational voice, Voice Activity Detection (VAD) can be used to identify the user's conversational voice in the audio signal.

[0053] It should be noted that before VAD recognition, the above real-time audio stream information can also be transformed by Fbank (FilterBank) to extract the inverse spectrum information of the audio signal and obtain the Fbank stream. The Fbank feature is based on the Mel scale, which is a simulation of the human auditory system. It scales the frequency in a nonlinear way to better match the human auditory perception. The original audio signal is high-dimensional data, and the computational cost of directly processing this data is high. However, the Fbank transformation can reduce the data dimension and retain important frequency information, which is more helpful for the VAD algorithm to distinguish the user's conversational voice from other background noise.

[0054] In a specific implementation, when a real-time audio stream is collected, the audio stream can first be input into a set of Mel filters and converted into an Fbank stream through frequency domain transformation, where the transformation parameter of the Mel filter can be set to Mel, the number of filters can be set to, for example, 32, the frame length can be set to 32ms, and the frame shift can be set to 10ms.

[0055] Step S20: pre-processing the real-time audio stream information.

[0056] It should be noted that, first, the Fbank stream can be processed by the VAD algorithm to obtain the start and end time points of each conversation voice in the real-time audio signal, and the Fbank features can be extended by 100ms before and after according to the start and end time points, which can prevent the loss of command word phonemes due to inaccurate VAD. Finally, in order to unify the Fbank features of the subsequent model, the Fbank features shorter than 2s can be extended to 2s, and the Fbank features longer than 2s can keep the original length unchanged. This realizes the preprocessing process of real-time audio stream information before feature extraction.

[0057] Step S30: Perform query feature recognition on the preprocessed audio stream information according to a preset feature extraction model to obtain current query features. The preset feature extraction model is a model obtained by training a preset student model after combining the embedded features of the preset teacher model with the corresponding classification output through the knowledge distillation method.

[0058] In some embodiments, the loss function of the preset student model includes an angle loss function.

[0059] It should be understood that in order to obtain a feature extraction model with high feature extraction performance under low data volume requirements and training resource requirements, a distillation learning method can be used to perform model training of a preset feature extraction model. The model training process can also be performed on a cloud server associated with the AR device.

[0060] It is understandable that in order to meet the requirements of the target task, i.e., the requirements of the command word recognition task, the selection of the teacher model can focus on the field similar to the command word recognition, such as the ASR model; and considering the feature extraction capability of the model, a learning model that has been pre-trained by self-supervision can be selected. Specifically, in this embodiment, the WavLM model that has been pre-trained by self-supervision and then fine-tuned by ASR can be selected as the teacher model in the distillation learning.

[0061] It should be understood that in order to obtain a teacher model that is more suitable for command word recognition, the above WavLM model can be fine-tuned to obtain the final preset teacher model, and the following adjustments are made in configuration:

[0062] (1) Fix the first eight encoder modules to prevent fine-tuning from causing large-scale migration of pre-trained underlying features and improve the stability of underlying features;

[0063] (2) In order to match the output features of the base model and the small model, the projection layer features of WavLM are set to 128 dimensions;

[0064] (3) Loss function uses an angular loss function (for example, ArcFaceLoss, a loss function used for deep face recognition tasks) to improve the ability to distinguish between different categories by increasing the inter-class spacing and reducing the intra-class spacing.

[0065] It should be noted that, considering the computing power limitations and power consumption requirements of AR devices, the selection of student models must also meet the requirements of lightweight and high reasoning efficiency. For example, the Mobileone model can be used as the student model in distillation learning. The Mobileone model is based on the CNN network structure, which is extremely optimized for the CPU and GPU on the device side. On the one hand, the re-parameterization technique can further reduce the model parameters after the model training is completed. On the other hand, the straight forward reasoning avoids excessive memory usage, thereby improving the reasoning efficiency in these two aspects.

[0066] It should be understood that in order to further improve the model inference speed, the above Mobileone model can be fine-tuned to obtain the final preset student model, which is adjusted as follows:

[0067] (1) The network is narrowed, so that its computational workload is only 1 / 4 of the original Mobileone model;

[0068] (2) When the sound waveform is transformed into Fbank features, the number of Mel filter banks is set to 32, the frame length is 32ms, and the frame shift is 10ms. In this way, a 2s audio signal can be converted into a 32×197 matrix, which greatly reduces the size of the feature maps of each layer.

[0069] It should be understood that due to the large structural differences between the teacher model and the student model, the output of the embedding layer (i.e., feature extraction layer) and the classification layer can be used to implement multi-level supervision. At the same time, ArcFaceLoss is used to enhance the compactness within the class and increase the distance between classes in the student model, so as to learn features with more distinguishing and generalization capabilities.

[0070] It can be understood that during the model training process, the input of the teacher model is the time domain sound waveform, while the input of the student model is the frequency domain Fbank feature obtained by converting the sound waveform through feature transformation in the front-end preprocessing. Repeat the training until the preset constraints are met, and obtain the trained preset student model, which can be used as the preset feature extraction model.

[0071] In the specific implementation, the preprocessed audio stream information, namely the Fbank stream information, is input into a preset feature extraction model, and several Fbank features in the above Fbank stream are extracted through the preset feature extraction model, and the extracted features are normalized to obtain the final query feature, namely the current query feature (Query).

[0072] Step S40: Match the current query feature in the preset command word feature library to determine the target command word.

[0073] It should be understood that the preset command word feature library may be a pre-set feature library (Gallery) including features corresponding to a number of standard command words, and the library features corresponding to each command word may be normalized features.

[0074] It is understandable that since the current query features and the library features are normalized, the cosine distance can be used to calculate the similarity between the features. The larger the similarity value, the higher the degree of match between the two. The similarity calculation formula is as follows:

[0075]

[0076] Among them, q i Indicates the i-th position of the current query feature, g i Represents the i-th position of the library feature.

[0077] In a specific implementation, the similarity between the current query feature and each library feature in the preset command word feature library can be calculated one by one, and then the standard command word corresponding to the library feature with the greatest similarity is selected as the target command word.

[0078] In this embodiment, real-time audio stream information is obtained; the real-time audio stream information is preprocessed; query feature recognition is performed on the preprocessed audio stream information according to a preset feature extraction model to obtain the current query feature, and the preset feature extraction model is a model obtained by training the preset student model after combining the embedded features of the preset teacher model with the corresponding classification output through the knowledge distillation method; the current query feature is matched in the preset command word feature library to determine the target command word. Since the preset feature extraction model of the preprocessed audio stream information input in this embodiment is constructed based on the preset learning model obtained by the knowledge distillation method combined with the metric learning training, the knowledge distillation method can reduce the training cost of the model while ensuring the performance of feature extraction, thereby enabling the matching process to more quickly identify the correct command word.

[0079] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the command word recognition method of the present application.

[0080] In this embodiment, in order to specifically illustrate the training process of the preset feature extraction model, before step S30, steps S01 to S04 are included:

[0081] Step S01: Collect voice corpus information.

[0082] It should be understood that the speech corpus information may include speech corpora in various scenarios collected from the Internet and open source data sets in the ASR field, and the speech corpus information may include speech information of several sentences of dialogue speech samples in different language categories, for example: long speech in the English category and long speech in the Chinese category.

[0083] Step S02: Process the speech corpus information to obtain a training corpus.

[0084] It should be noted that, in order to normalize the speech corpus information collected from different sources, data processing such as segmentation, alignment and data cleaning can be performed on each sentence of speech information. Specifically, step S02 includes steps S021 to S023:

[0085] Step S021: Align each conversation speech sample to obtain a cutting timestamp.

[0086] It should be understood that an open source timestamp alignment algorithm (such as the WhisperX algorithm) may be used to align each conversation speech sample to obtain a word-level or character-level timestamp.

[0087] Step S022: performing word segmentation and segmentation on each dialogue speech sample based on the segmentation timestamp to obtain a command word dataset.

[0088] In the specific implementation, for long speech in the English category, the word-level timestamp can be used as the command word segmentation granularity; for long speech in the Chinese category, the character-level timestamp is first used to segment it, and then the segmentation result is used as the segmentation granularity. Then, a command word dataset including multiple language categories is constructed.

[0089] Step S023: Clean the command word data set to obtain a training corpus.

[0090] It should be understood that samples with shorter or longer durations can be eliminated. For example, samples with a duration less than 200ms or greater than 5s can be set as abnormal samples, and these abnormal samples can be eliminated from the command word data set. Words or phrases with too few samples can also be eliminated. For example, samples corresponding to words or phrases with less than 20 samples can be set as invalid samples, and these invalid samples can also be eliminated from the command word data set.

[0091] In a specific implementation, the above abnormal samples and invalid samples are removed from the command word data set to complete data cleaning and obtain a training corpus.

[0092] Step S03: Input each training sample in the training corpus into the preset teacher model respectively to obtain the feature information and classification information output by the preset teacher model.

[0093] It should be noted that the feature information can be a fixed-length feature vector representation of each training sample converted by a preset teacher model, namely, an embedded feature (Embedding); the classification information can be a category score (Logits) generated by the Embedding through the fully connected layer.

[0094] Step S04: Based on the constraint condition of minimizing the angle loss function, combined with feature information and classification information, the preset student model is trained according to the training corpus to obtain a preset feature extraction model.

[0095] It should be noted that, during the model training process, the input of the teacher model is the time domain sound waveform, while the input of the student model is the frequency domain Fbank feature. Therefore, data enhancement can be achieved in the time domain and frequency domain respectively, and then the preset student model is trained using each training sample after data enhancement until the constraints of ArcFaceLoss are met, and the preset feature extraction model is obtained.

[0096] It is understandable that time domain enhancement can be performed by random time shift, random loudness adjustment, random speed and random noise. The specific time domain enhancement operation is as follows:

[0097] Random time shift: For fixed input length speech (training samples) containing invalid signals at both ends, the valid speech pieces are randomly shifted along the time axis to generate new speech, improving the performance degradation caused by the time shift deviation of the command word;

[0098] Random loudness adjustment: Randomly adjust the loudness of the voice signal within a certain range to improve the recognition effect of changes in voice volume;

[0099] Random speed: scale the speech speed within a certain range (e.g. 0.5 to 2.0) to improve the recognition effect caused by speech speed changes;

[0100] Random noise: Adding background noise from the usage environment and any scene is helpful to improve the model's noise resistance in real scenes.

[0101] It is understandable that the frequency domain enhancement can be performed in a random mask manner. The specific frequency domain enhancement operation is as follows:

[0102] Random masking: After the sound waveform signal is converted into Fbank features, the spectrum graph is randomly masked, which helps improve the model's recognition performance of different human voice frequency changes.

[0103] It can be understood that the AR device can perform Fbank feature transformation on each training sample in the training corpus to obtain several Fbank feature samples; input each Fbank feature sample into the preset student model, and use the knowledge distillation method to combine feature information and classification information to train the preset student model; when the minimization condition of the angle loss function is met, the preset feature extraction model is obtained, and the angle loss function is ArcFaceLoss.

[0104] Further, please refer to Figure 3 The specific process of using the knowledge distillation method to perform model training to obtain a preset feature extraction model in this embodiment is described. Figure 3 Schematic diagram of the model distillation framework in the command word recognition method of this application.

[0105] exist Figure 3 In the example, the teacher model (teacher network) can adopt the slightly adjusted WavLM model, and the student model (student network) can adopt the slightly adjusted Mobileone model.

[0106] The input of the teacher network is the time domain sound waveform, i.e., the original training samples in the training corpus, while the input of the student network is the frequency domain Fbank features, i.e., the Fbank feature samples obtained by Fbank transformation of each training sample.

[0107] Regarding feature layer distillation: The embeddings of the feature layer outputs of the teacher network and the student network can both be 128 dimensions. The regression loss function L1-Loss is used to ensure that the student network fits the teacher network features during training. The regression loss function is expressed as follows:

[0108]

[0109] in, represents the i-th element of the teacher network output feature, Represents the i-th element of the student network output feature.

[0110] About classification layer distillation: The classification layer classifies the input speech into different command words according to the set categories. Figure 3 On the distillation architecture, firstly, the output of the classification layer of the teacher network and the student network is connected to a Softmax layer to convert Logits into multi-classification probability distribution. KLDivLoss is introduced to measure the difference between the distribution of different categories output by the teacher network and the student network, which is expressed as follows:

[0111]

[0112] Among them, p(x i ) represents the probability value of the i-th output of the teacher network, q(x i ) represents the probability value of the i-th output of the student network.

[0113] Regarding the classification constraints of the student network: Since the input Fbank features of the student network are two-dimensional features, they can be regarded as image recognition from the perspective of data structure representation. Therefore, ArcFaceLoss, which is commonly used in face recognition tasks, can be introduced to minimize the angle difference between the same command words, enhance the differences between different command words, and improve the recognition ability of command words. The specific formula is as follows:

[0114]

[0115] Where s is the scale factor, which indicates the radius of the hypersphere; m is the angle boundary penalty coefficient. The larger the m, the larger the interval between classes. The larger the s and m, the more difficult the training. It can be personalized according to the task requirements. For example, in this embodiment, s=32 and m=0.5 can be set.

[0116] It should be understood that the preset student model is trained based on the above-mentioned distillation architecture in the cloud server associated with the AR device. When the above-mentioned ArcFaceLoss minimization condition is met, the training is considered completed, and the preset feature extraction model is obtained, which can then be deployed in the AR device to achieve subsequent recognition tasks.

[0117] In this embodiment, speech corpus information is collected, and the speech corpus information includes several conversation speech samples of different language categories; each conversation speech sample is aligned to obtain a cutting timestamp; each conversation speech sample is segmented and cut based on the cutting timestamp to obtain a command word data set; the command word data set is cleaned to obtain a training corpus. Multilingual word-level alignment technology is used in corpus construction to improve the sample diversity of the training corpus. In the training process of distillation learning, each training sample in the training corpus is transformed into an Fbank feature to obtain several Fbank feature samples; each Fbank feature sample is input into a preset student model, and the preset student model is trained by combining feature information and classification information using a knowledge distillation method; when the minimization condition of ArcFaceLoss is met, a preset feature extraction model is obtained, thereby realizing the combination of distillation and metric learning in model training, and ensuring the reasoning efficiency of the model while achieving model lightweight.

[0118] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to the above description, and will not be described in detail later. Figure 4 , Figure 4 This is a flow chart of the third embodiment of the command word recognition method of the present application.

[0119] In this embodiment, in order to specifically illustrate the process of constructing the preset command word feature library, before step S30, the following steps are also included: Steps S001 to S002:

[0120] Step S001: Acquire speech samples corresponding to each standard command word, extract features of preset dimensions for each speech sample, and normalize the extracted dimensional features, wherein the preset dimensions are consistent with the dimensions of the embedded features.

[0121] It should be understood that the voice samples corresponding to each standard command word must meet the two requirements of broad representativeness (regardless of factors such as gender, age or region, the voice samples must be well representative) and easy to obtain (due to the limitations of existing conditions, it is difficult to collect actual voice data corresponding to each standard command word in various regions).

[0122] It is understandable that, in order to meet the above two requirements, a text-to-speech (TTS) technology may be used to synthesize a certain number (eg, 40) of speech samples of different styles for each standard command word.

[0123] It should be understood that in order to make the features of each standard command word representative and reduce the impact of abnormal pronunciation and noise in the speech sample, for each standard command word, 128-dimensional features can be extracted one by one, and then the 128-dimensional features of each speech sample can be L2 normalized to eliminate the differences in volume or intensity between different speech samples under the same standard command word.

[0124] Step S002: construct a number of median dimensional features according to the dimensional features obtained through normalization processing, and determine the library features corresponding to each standard command word in the preset command word feature library according to each median dimensional feature.

[0125] It can be understood that for the 128-dimensional features, the same dimension (e.g., the first dimension) of all speech samples under the same standard command word is put together, and then the median of these values ​​is calculated. The median calculation step is repeated until the median of all dimensions is calculated, and then the median of each dimension is used to construct the final 128-dimensional feature vector, which represents the typical features of the corresponding standard command word and is stored in the preset command word feature library as the library features of the standard command word.

[0126] It should be understood that, by generating a corresponding library feature for each standard command word through the above steps, the representativeness of the feature is ensured while effectively reducing the influence of noise, thus providing a stable feature basis for the subsequent command word recognition process.

[0127] Furthermore, considering that the standard command word corresponding to the library feature with the largest similarity value with the current query feature output by the model in the preset command word feature library is directly used as the matching result, since any speech segment intercepted by the VAD will obtain a recognition result, and the speech segment may not be the set standard command word, it is easy to cause a large number of invalid voices to be misactivated and affected by the standard command words that are easy to respond. A corresponding activation threshold can be set for each standard command word, so step S30 specifically includes: steps S301 to S303:

[0128] Step S301: Determine the similarity scores between the current query feature and each library feature.

[0129] Step S302: Compare each similarity score with the activation threshold of the corresponding library feature, and obtain a candidate command word set according to the comparison result.

[0130] It should be noted that since manually setting the activation threshold of each library feature is cumbersome and the effect is poor, considering that there is a certain correlation between each standard command word and their corresponding activation thresholds are independent, the threshold setting can be modeled as a multi-objective optimization task. For example, the Pareto multi-objective optimization algorithm can be used to maximize the precision and recall of each standard command word.

[0131] Specifically, we can first obtain the precision information and recall rate information of the standard command words to which each library feature belongs; then use the preset Pareto optimization algorithm to calculate the activation threshold corresponding to each library feature according to the precision information and recall rate information. The formula description is as follows:

[0132] maximize: [F11(t1),F12(t2),…,F1 N (t N )],

[0133] subject to i ∈[0,1],i=1,2,…,N

[0134] Among them, t i represents the activation threshold of the i-th library feature category; F1 i It represents the F1 score of the i-th library feature category, and the calculation formula is F1=2*(Precision*Recall) / (Precision+Recall). The larger the F1 score, the better the recognition performance.

[0135] It should also be noted that the progress information and recall rate information of the above-mentioned standard command words can be obtained based on the specific data statistics in the aforementioned model training process. The above-mentioned process of calculating the activation threshold corresponding to each library feature can be implemented by calling the Pymoo library for solution in actual operation.

[0136] In a specific implementation, the similarity scores between the current query feature and each library feature are calculated one by one, and each similarity score calculated is compared with the activation threshold corresponding to the library feature to which it belongs. When the similarity score is greater than the activation threshold corresponding to the library feature, the standard command word corresponding to the library feature is determined as a candidate command word. The above steps of similarity score calculation-comparison-candidate command word determination are repeated until all library features in the preset command word feature library are traversed to obtain a candidate command word set containing several candidate command words.

[0137] Step S303: In the candidate command word set, the candidate command word corresponding to the maximum similarity score is determined as the target command word.

[0138] It is understandable that in the above candidate command word set, each candidate command word can be sorted based on the similarity score, and then the candidate command word corresponding to the library vector with the maximum similarity score is obtained and determined as the target command word.

[0139] In addition, you can also refer to Figure 5 The full process description of using the command word recognition method of the present application to obtain the target command word by the AR device is as follows: Figure 5This is a schematic diagram of the full-process reasoning technology route of the command word recognition method of this application.

[0140] Depend on Figure 5 It can be seen that a set of real-time input audio streams needs to go through the following processing steps to output the target command words:

[0141] (1) Frequency domain transformation: real-time audio stream is converted into Fbank stream;

[0142] (2) VAD recognition: The start and end time points of each conversation speech in the real-time audio signal are obtained through the VAD algorithm;

[0143] (3) Preprocessing: Expand the Fbank features forward and backward to adapt to the model input format;

[0144] (4) Feature extraction: Use the preset feature extraction model to extract the preprocessed Fbank features and normalize them to obtain the current query features (Query);

[0145] (5) Feature matching: Calculate the similarity scores between the current query feature (Query) and the features in the preset command word feature library (Gallery);

[0146] (6) Post-processing: First, among the similarity scores of the N library features, K (K∈[0,N]) library features that are greater than the corresponding activation threshold are screened out to obtain K candidate command words. Then, among the K candidate command words, the candidate command word corresponding to the library vector with the maximum similarity score is obtained, which is the target command word.

[0147] In this embodiment, by constructing a preset command word feature library, the similarity score of each library feature corresponding to the current query feature is determined; each similarity score is compared with the activation threshold of the corresponding library feature, and a candidate command word set is obtained according to the comparison result; in the candidate command word set, the candidate command word corresponding to the maximum similarity score is determined as the target command word. The activation threshold can be set for each library feature to greatly reduce misidentification and improve recognition accuracy. The Pareto optimization algorithm can be used to automatically set the activation threshold of each library feature, which is conducive to achieving efficient threshold setting while ensuring the reliability of setting the threshold.

[0148] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the command word recognition method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0149] The present application also provides a command word recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the command word recognition method in the above-mentioned embodiment one.

[0150] Reference below Figure 6 , Figure 6 The device structure diagram of the hardware operating environment involved in the command word recognition method of the present application. The command word recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The command word recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0151] like Figure 6As shown, the command word recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the command word recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the command word recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a command word recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0152] The present application also provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the command word recognition method in the above-mentioned embodiment.

[0153] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0154] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned command word recognition method, and can solve the technical problem of command word recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the command word recognition method provided by the above-mentioned embodiment, and will not be repeated here.

[0155] The above are only some embodiments of the present application, and are not intended to limit the scope of the patent of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or directly / indirectly applied in other related technical fields, can be included in the scope of protection of the patent of the present application.

Claims

1. A command word recognition method, characterized in that: The method comprises: Get real-time audio stream information; Preprocessing the real-time audio stream information; Performing query feature recognition on the preprocessed audio stream information according to a preset feature extraction model to obtain current query features, wherein the preset feature extraction model is a model obtained by training a preset student model after combining the embedded features of a preset teacher model with corresponding classification outputs through a knowledge distillation method; The current query feature is matched in a preset command word feature library to determine the target command word.

2. The method according to claim 1, characterized in that Before the step of performing query feature recognition on the preprocessed audio stream information according to the preset feature extraction model to obtain the current query feature, the step includes: Collect voice corpus information; Performing data processing on the speech corpus information to obtain a training corpus; Input each training sample in the training corpus into the preset teacher model respectively to obtain feature information and classification information output by the preset teacher model; Based on the constraint condition of minimizing the angle loss function, combined with the feature information and the classification information, the preset student model is trained according to the training corpus to obtain a preset feature extraction model.

3. The method according to claim 2, characterized in that The speech corpus information includes a number of conversational speech samples of different language categories, and the step of performing data processing on the speech corpus information to obtain a training corpus includes: Performing alignment processing on each of the dialogue speech samples to obtain a cutting timestamp; Perform word segmentation and segmentation on each of the dialogue speech samples based on the segmentation timestamp to obtain a command word dataset; The command word data set is cleaned to obtain a training corpus.

4. The method according to claim 2, characterized in that The step of training the preset student model based on the training corpus and obtaining the preset feature extraction model based on the constraint condition of minimizing the angle loss function, combining the feature information and the classification information, comprises: Performing Fbank feature transformation on each training sample in the training corpus to obtain a number of Fbank feature samples; Input each of the Fbank feature samples into a preset student model, and train the preset student model by combining the feature information and the classification information using a knowledge distillation method; When the minimization condition of the angle loss function is met, a preset feature extraction model is obtained, and the angle loss function is ArcFaceLoss.

5. The method according to claim 1, characterized in that Before the step of matching the current query feature in the preset command word feature library to determine the target command word, the method includes: Acquire speech samples corresponding to each standard command word, extract features of a preset dimension for each speech sample, and normalize the extracted dimensional features, wherein the preset dimension is consistent with the dimension of the embedded feature; A plurality of median dimensional features are constructed according to the dimensional features obtained by the normalization process, and a library feature corresponding to each standard command word in the preset command word feature library is determined according to each of the median dimensional features.

6. The method according to claim 5, characterized in that The step of matching the current query feature in a preset command word feature library to determine the target command word includes: Determine a similarity score between the current query feature and each of the library features; Comparing each of the similarity scores with the activation threshold of the corresponding library feature, and obtaining a set of candidate command words according to the comparison results; In the candidate command word set, the candidate command word corresponding to the maximum similarity score is determined as the target command word.

7. The method according to claim 6, characterized in that Before the step of comparing each of the similarity scores with the activation threshold of the corresponding library feature and obtaining a set of candidate command words according to the comparison results, the method includes: Obtaining precision information and recall rate information of the standard command words to which each of the library features belongs; A preset Pareto optimization algorithm is used to calculate the activation threshold corresponding to each of the library features according to the precision information and the recall rate information.

8. The method according to claim 6, characterized in that The step of comparing each of the similarity scores with the activation threshold of the corresponding library feature and obtaining a set of candidate command words according to the comparison result comprises: When the similarity score is greater than the activation threshold corresponding to the library feature, the standard command word corresponding to the library feature is determined as a candidate command word.

9. A command word recognition device, characterized in that: The command word recognition device includes: a memory, a processor, and a command word recognition program stored in the memory and executable on the processor. When the command word recognition program is executed by the processor, the steps of the command word recognition method as described in any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: The storage medium stores a command word recognition program, which, when executed by a processor, implements the steps of the command word recognition method according to any one of claims 1 to 8.

Images