Command word confirmation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of voice recognition, in particular to a command word confirmation method and device, equipment and a storage medium, and the method comprises the steps: training a recognition model on a general corpus, then optimizing the recognition model based on a command word corpus, and fixing a model weight to obtain a fixed recognition model; extracting the audio embedding of the input audio based on the fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimension space, and aligning; the aligned audio embedding is processed through a self-attention mechanism, context information in the audio is captured, and the matching effect between the audio embedding and the text embedding is optimized; and performing similarity comparison on the audio embedding and the text embedding of the input text of the custom vocabulary, and if the similarity exceeds a first preset threshold, identifying the command word as the corresponding custom vocabulary. According to the method and the device, the recognition capability of the user-defined vocabularies is remarkably improved, and the recognition accuracy of the personalized instruction is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a command word confirmation method, device, equipment and storage medium. Background Art

[0002] In existing speech recognition systems, especially in smart home and smart terminal scenarios, command word recognition relies on a preset fixed vocabulary, which makes it difficult to cope with user personalized needs or special instructions in specific application scenarios, such as "open my music playlist" or "set reading mode". Although the CTC (Connectionist Temporal Classification) algorithm is fast and memory-efficient when processing sequence tasks, it does not provide sufficient support for non-fixed and custom command words, resulting in frequent misrecognition or unrecognition problems. In addition, the introduction of custom vocabulary faces the challenge of how to ensure accurate recognition without significantly increasing computing resource consumption. Traditional methods require retraining the model or significantly increasing the number of parameters, which is time-consuming, labor-intensive, and increases system complexity.

[0003] Therefore, existing speech recognition systems do not provide sufficient support for custom vocabulary, resulting in low accuracy in recognizing personalized commands and technical issues that affect user experience. Summary of the invention

[0004] The main purpose of this application is to provide a command word confirmation method, device, equipment and storage medium, aiming to solve the technical problem that the existing speech recognition system does not support custom vocabulary enough, resulting in low accuracy of personalized command recognition and affecting user experience.

[0005] In order to achieve the above-mentioned invention object, the present application proposes a command word confirmation method, the method comprising: The recognition model is trained on the general corpus, then optimized based on the command word corpus, and the model weights are fixed to obtain a fixed recognition model. Extract audio embedding of the input audio based on a fixed recognition model, map the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and align them; The aligned audio embedding is processed through the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding. The audio embedding is compared with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary.

[0006] Furthermore, after the step of processing the aligned audio embedding by the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding, the method further comprises: Calculate the similarity between audio embedding and text embedding at the frame level; If there is a certain number of frames where the similarity is lower than a second preset threshold, it is determined to be a misrecognition and the current input audio is rejected.

[0007] Furthermore, after the step of processing the aligned audio embedding by the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding, it also includes: Classify based on the command word list and generate a confusing identification list; When the recognized command word is in the easily confused recognition list, traverse each command word in the easily confused recognition list; Calculate the similarity between the audio embedding and the text embedding of each command word separately; The command word with the highest similarity score is selected as the final recognition result, and the recognition is updated to the corresponding command word.

[0008] Furthermore, the steps of training the recognition model on the general corpus, optimizing the recognition model based on the command word corpus, and fixing the model weights to obtain a fixed recognition model include: Conduct preliminary training of the recognition model on general corpus; Optimizing the preliminarily trained model based on command word data including preset command words; After the optimization is completed, the weight parameters of the model are fixed to form a fixed recognition model.

[0009] Furthermore, the step of extracting audio embedding of the input audio based on the fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and aligning them includes: Extract features from input audio and generate audio embedding based on a fixed recognition model; Convert the input text into an embedding vector and generate text embedding; Map audio embeddings to the same dimensional space as text embeddings; Using the Viterbi algorithm, the audio embeddings and text embeddings are aligned on the same timeline.

[0010] Furthermore, the step of processing the aligned audio embedding through a self-attention mechanism to capture contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding includes: Apply self-attention mechanism to the aligned audio embeddings to capture contextual information in the audio sequence; Based on the audio embedding and text embedding processed by the self-attention mechanism, frame-by-frame similarity calculation is performed in the same dimensional space; The contrast loss function is used to adjust the distance between audio embedding and text embedding to optimize the matching effect between audio embedding and text embedding.

[0011] Furthermore, the step of comparing the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, identifying the command word as the corresponding custom vocabulary, includes: Compare the similarity between audio embedding and text embedding frame by frame; Comparing the calculated similarity with a first preset threshold; If the similarity exceeds a first preset threshold, the command word is recognized as a corresponding custom word, and a recognition result is output.

[0012] The second aspect of the present application provides a command word confirmation device, comprising: The training module is used to train the recognition model on the general corpus, then optimize the recognition model based on the command word corpus, and fix the model weights to obtain a fixed recognition model; A mapping module is used to extract the audio embedding of the input audio based on a fixed recognition model, map the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and align them; The optimization module is used to process the aligned audio embeddings through the self-attention mechanism, capture the contextual information in the audio, and optimize the matching effect between the audio embedding and the text embedding; A comparison module is used to compare the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary.

[0013] The third aspect of the present application also includes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0014] The fourth aspect of the present application also includes a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0015] Beneficial effects: This solution significantly improves the command word recognition system's support for custom vocabulary, thereby greatly improving the recognition accuracy and user experience of personalized instructions. First, the model is optimized based on the general corpus combined with specific command word corpus, and the key weights are fixed to ensure the stability and accuracy of the model when processing command words. Then, by mapping the audio embedding and text embedding to the same dimensional space and aligning them, effective fusion of cross-modal information is achieved, enhancing the system's understanding ability. The introduction of the self-attention mechanism further optimizes the audio embedding, enabling it to capture richer contextual information, solving the problem of misrecognition caused by lack of contextual support in traditional methods. Finally, by comparing the similarity of audio embedding and custom vocabulary text embedding, this solution can efficiently identify user-defined command words under low threshold conditions, greatly improving the flexibility and accuracy of personalized services, effectively solving the problem of insufficient support for custom vocabulary in the prior art, and providing users with a more intelligent and personalized interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a command word confirmation method according to an embodiment of the present application; Figure 2 A schematic block diagram of the structure of a command word confirmation device according to an embodiment of the present application; Figure 3 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "a", "an", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0020] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0021] Reference Figure 1 The embodiment of the present invention provides a command word confirmation method, including steps S1-S4, specifically: S1. Train the recognition model on the general corpus, then optimize the recognition model based on the command word corpus, and fix the model weights to obtain a fixed recognition model; S2, extracting audio embedding of the input audio based on a fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and aligning them; S3, process the aligned audio embedding through the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding; S4: Compare the audio embedding with the text embedding of the input text of the custom vocabulary for similarity. If the similarity exceeds a first preset threshold, identify the command word as the corresponding custom vocabulary.

[0022] The above step S1 aims to build an efficient and adaptable command word recognition model. The specific implementation process is divided into two main stages: first, the general corpus training stage, and then the optimization stage based on the specific command word corpus. In the general corpus training stage, widely applicable language datasets are selected as basic training materials. These datasets usually contain daily conversations, news broadcasts, etc. in multiple language environments, with rich vocabulary and grammatical structure changes. By using these general corpora to train the initial recognition model, it can be ensured that the model has good generalization ability and can handle various complex voice input scenarios. In this stage, the CTC (Connectionist Temporal Classification) algorithm is used for decoding. Because of its advantages such as no need for alignment for sequence tasks, fast calculation speed and low memory usage, it is particularly suitable for embedded devices. As the training progresses, the model will continuously adjust its parameters on the validation set of the general corpus until the loss value or WER (Word Error Rate) no longer decreases significantly, indicating that the model has fully learned the patterns and features in the general corpus.

[0023] After completing the general corpus training, we enter the second stage, which is the optimization based on the command word corpus. The goal of this stage is to enhance the model's ability to recognize command words in a specific field. To this end, in addition to the original general corpus, it is also necessary to introduce a specially designed command word corpus, which covers the commonly used command words in smart home devices, such as "turn on the fan" and "turn off the light". Continue to use the model obtained in the first stage as a starting point, combine the general corpus with the command word corpus, and further fine-tune the model. In this process, pay special attention to the performance on the command word corpus, and continuously adjust the model parameters until the loss value or WER on the validation set of the command word corpus no longer decreases. The key to this step is to ensure that the model can be optimized for the command word recognition task while retaining the original generalization ability. Once this goal is achieved, this part of the weights will be fixed to prevent the impact of these key parameters in the subsequent training process, thereby ensuring the accuracy and stability of the model when processing command words.

[0024] In addition, this step not only improves the model's ability to recognize predefined command words, but also lays a solid foundation for subsequent steps, including the generation and alignment of audio and text embeddings, the application of the self-attention mechanism, and the final similarity comparison. The entire process emphasizes the gradual refinement strategy from general to specific fields, ensuring that the model has both broad adaptability and can provide accurate services in specific application scenarios.

[0025] The main task of the above step S2 is to extract the audio embedding of the input audio from the fixed recognition model, and map these audio embeddings to the same dimensional space with the corresponding text embeddings for alignment. First, after obtaining the fixed recognition model trained and optimized in the S1 stage, the model can effectively process general voice commands and command words in specific fields. In practical applications, when a new audio input is received, a fixed recognition model is used to extract feature representations, namely audio embeddings, from the audio data. Specifically, this process involves the output of the last layer or layers of a deep neural network, and the features before these layers often contain rich semantic information. For example, an audio embedding layer can be derived by extracting the features before the audio linear layer to generate an audio embedding layer that can capture the phoneme-level details in the audio signal.

[0026] Next, you need to build a corresponding text embedding for the input text. This step first involves converting the text into a digital representation, such as splitting the original text into words or character sequences through a tokenizer, and then using a pre-trained word vector model such as Word2Vec, GloVe, or a more complex context-aware model such as lightweight BERT to convert each word into its corresponding embedding vector. In order for audio embeddings and text embeddings to be compared in the same dimensional space, it is necessary to ensure that the two have the same dimensional size. Therefore, methods such as projection matrices can be used to map text embeddings to the same dimensional space as audio embeddings. In this process, some technical means, such as multi-layer perceptrons (MLPs) or simple linear transformations, may be used to ensure that the two types of embeddings can operate under the same framework.

[0027] After completing the above steps, we enter the alignment phase. The goal of alignment is to ensure that the correspondence between audio embeddings and text embeddings reflects their true association as accurately as possible. Here, the Viterbi algorithm can be used to align audio embeddings so that each audio clip can find the text part that best matches it. For example, for a command word like "turn on the fan", through the alignment process, we can determine which parts of the audio sequence correspond to "turn on", "open", "wind", and "fan". This precise alignment not only helps to improve the accuracy of command word recognition, but also enhances the system's ability to understand complex sentence structures. Ultimately, through the aligned audio embeddings and text embeddings, the system can better understand the user's intentions, laying the foundation for further optimizing the matching effect through the self-attention mechanism in the subsequent steps and realizing efficient recognition of custom vocabulary. This process plays a bridging role in the entire command word confirmation process, connecting the audio feature extraction at the front end and the high-level semantic processing at the back end.

[0028] The main task of the above step S3 is to process the audio embeddings aligned by the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding. First, after obtaining the aligned audio embeddings processed by step S2, these embeddings have preliminarily reflected the relationship between the input audio and the corresponding text, but have not fully considered the impact of the context. The self-attention mechanism can effectively solve this problem. It allows the model to dynamically consider the information of other time steps when processing the audio features of each time step, thereby enhancing the ability to understand complex contexts.

[0029] In the specific implementation process, the aligned audio embedding is first passed as input to the self-attention layer. The core of the self-attention layer is to calculate the query (Query, Q), key (Key, K), and value (Value, V), which are obtained by applying different linear transformations to the original audio embedding. In order to further reduce the amount of calculation and the number of parameters, a shared weight approach can be adopted, that is, Q, K, and V are transformed using the same weight matrix. Next, the similarity score between each pair of time steps is calculated, usually through dot products or scaled dot products. The similarity score is then converted to a probability distribution through a softmax function, indicating the importance of each time step relative to other time steps. Based on this probability distribution, the value vector is weighted and summed to generate a new audio embedding representation, which contains relevant information about the current time step and all other time steps, thereby effectively capturing the context.

[0030] This processing method not only enables audio embedding to better reflect the long-distance dependencies in speech signals, but also enhances the system's ability to understand complex sentence structures. For example, when processing command words like "turn on the air conditioner", the self-attention mechanism can help distinguish the two parts of "turn on" and "air conditioner", and accurately identify the user's intention based on contextual information. In addition, by introducing contextual information, the self-attention mechanism can also effectively alleviate the problem of misidentification caused by lack of background knowledge. For example, when a user issues a "turn on the fan" command, the system will not mistakenly identify it as "turn on the light".

[0031] After the self-attention process is completed, the new audio embedding is used to match the text embedding more accurately. In this process, by comparing the feature differences between the audio embedding and the text embedding on each frame, techniques such as contrastive loss (Contrastive Loss) and asymmetric proxy loss (Asymmetric Proxy Loss, AsyP Loss) can be used to ensure that the distance between the positive sample pairs is as close as possible and the distance between the negative sample pairs is as far as possible. This method not only improves the consistency between the audio and text embeddings, but also significantly improves the overall performance of the command word recognition system, especially when dealing with challenging custom vocabularies. In this way, step S3 plays a vital role in the entire command word confirmation process, greatly enhancing the robustness and adaptability of the system.

[0032] In the above step S5, the user-defined command words issued by the user are identified by comparing the similarity between the audio embedding and the text embedding of the custom vocabulary. The core of this step is to accurately determine whether the input audio matches the predefined or user-defined command words, and identify it as the corresponding custom vocabulary when certain conditions are met. In the specific implementation process, the audio embeddings processed by step S3 need to be prepared first. These embeddings have captured rich contextual information through the self-attention mechanism and can better reflect the semantic details in the speech signal.

[0033] Next, for each custom word, calculate its corresponding text embedding in advance. This step usually uses a trained language model or embedding layer (such as BLSTM, BERT, etc.) to convert the custom vocabulary into a vector representation of fixed dimension. In order to ensure that the text embedding and audio embedding can be compared in the same feature space, it may be necessary to perform appropriate normalization or dimension adjustment operations on both. For example, the text embedding can be mapped to the same dimensional space as the audio embedding through linear transformation or other mapping methods.

[0034] After completing the above preparations, the similarity comparison phase begins. For each piece of input audio and its corresponding audio embedding, the system will calculate the similarity with the text embeddings of all custom vocabulary in turn. Commonly used similarity measurement methods include cosine similarity, Euclidean distance, etc. Taking cosine similarity as an example, it measures the similarity between two vectors by calculating the cosine value of the angle between them. The closer the value is to 1, the higher the similarity. In this process, the system will set a preset threshold. Only when the similarity between a pair of audio embeddings and text embeddings exceeds this threshold, the audio clip will be considered to have successfully matched the corresponding custom vocabulary.

[0035] It is worth noting that, considering that custom vocabulary may not have undergone special intensive training, a lower threshold can be set in the initial stage to improve the recall rate, that is, to identify as many custom vocabulary that meet the conditions as possible. As the system is used for a longer time and data is accumulated, this threshold can be gradually adjusted according to actual conditions to achieve a higher precision. In addition, in real-time application scenarios, a streaming comparison method can also be used, that is, a similarity calculation is performed immediately whenever a new phoneme or command word is detected, thereby achieving a quick response.

[0036] Through this detailed and flexible similarity comparison strategy, step S4 not only solves the problem of insufficient support for custom vocabulary in traditional speech recognition systems, but also significantly improves the accuracy of personalized command recognition and user experience. Especially in scenarios such as smart homes and smart terminals, users can easily define their own command words, making the interaction more natural and smooth. Ultimately, this method builds an efficient and robust command word recognition framework through a series of carefully designed steps, from basic model training to advanced feature extraction and matching, providing users with an intelligent and personalized voice interaction solution.

[0037] In one embodiment, after the step of processing the aligned audio embeddings by a self-attention mechanism to capture contextual information in the audio and optimize the matching effect between the audio embeddings and the text embeddings, the method further comprises: S10, calculating the similarity between the audio embedding and the text embedding at the frame level; S11. If there is a certain number of frames whose similarity is lower than a second preset threshold, it is determined to be a misrecognition, and the current input audio is rejected for recognition.

[0038] In this embodiment, after completing the processing of the aligned audio embedding through the self-attention mechanism, capturing the contextual information in the audio and optimizing the matching effect between the audio embedding and the text embedding, the next key step is to calculate the similarity between the audio embedding and the text embedding at the frame level, and decide whether to refuse to recognize the current input audio based on the similarity result. This process first requires converting the audio embedding and text embedding processed by the self-attention mechanism into the same feature space for direct comparison. Typically, these embeddings have been adjusted to the same dimension through mapping or transformation, so that the similarity between them can be directly calculated.

[0039] In the specific implementation, first select a suitable similarity measurement method to calculate the similarity of each pair of audio embedding and text embedding in each frame. Commonly used measurement methods include cosine similarity, Euclidean distance, etc. Taking cosine similarity as an example, it measures the similarity between two vectors by calculating the cosine value of the angle between them. The closer the value is to 1, the higher the similarity. For each frame of audio embedding and corresponding text embedding, the system will calculate the similarity score between them and record it. This step ensures that the subtle differences between audio and text can be accurately captured even in complex contexts.

[0040] The system then analyzes the similarity scores of all frames to determine whether there are any frames whose similarity is lower than the preset second threshold. This threshold is pre-set based on the actual application scenario and the characteristics of the data set, and is intended to filter out audio clips that may be at risk of misidentification. If a certain number of frames (such as several consecutive frames or a certain proportion of frames) are found to have similarity scores lower than the threshold, the system will consider that there is a large uncertainty or possibility of error in the current input audio, and will refuse to recognize the audio input. This strategy effectively improves the robustness of the system and reduces misidentification caused by background noise, unclear pronunciation or other interference factors.

[0041] In addition, in practical applications, in order to further improve accuracy, other technical means such as contrastive loss, asymmetric proxy loss (Asymmetric Proxy Loss, AsyP Loss), etc. can be combined to optimize the frame-level similarity calculation process. For example, by comparing the distance of positive sample pairs (i.e., correctly matched audio embedding and text embedding) with the distance of negative sample pairs (i.e., incorrectly matched audio embedding and text embedding), it can ensure that the distance of positive sample pairs is as small as possible, while the distance of negative sample pairs is as large as possible, thereby enhancing the model's ability to distinguish between correct and incorrect matches.

[0042] In general, by calculating the similarity between audio embedding and text embedding at the frame level and setting a reasonable threshold to determine whether to reject the recognition of the current input audio, this method not only improves the accuracy and reliability of the command word recognition system, but also significantly enhances the system's ability to cope with complex environments and user diversity. This provides a more intelligent, flexible and efficient solution for voice interaction in scenarios such as smart homes and smart terminals. Through a series of carefully designed steps, from basic model training to the final fine verification, the entire framework ensures high accuracy of speech recognition and good consistency of user experience.

[0043] In one embodiment, after the step of processing the aligned audio embedding by a self-attention mechanism to capture contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding, the method further includes: S20, classifying based on the command word list to generate an easily confused recognition list; S21, when the recognized command word is in the easily confused recognition list, traverse each command word in the easily confused recognition list; S22, respectively calculating the similarity between the audio embedding and the text embedding of each command word; S23, selecting the command word with the highest similarity score as the final recognition result, and updating the recognition to the corresponding command word.

[0044] In this embodiment, the key step is to classify based on the command word list, generate an easily confused recognition list, and further refine the command word recognition process on this basis. First, the system generates an easily confused recognition list based on the predefined command word list, especially those easily confused command words. For example, in smart home devices, command words such as "turn on the fan" and "turn off the fan", or "previous song" and "next song" are easily misrecognized due to similar pronunciation or similar semantics. Therefore, these command words are classified into the easily confused recognition list for more detailed processing later.

[0045] When the command word recognized by the system belongs to the easily confused recognition list, it will enter a more sophisticated recognition process. Specifically, for each recognized command word, the system will traverse each command word in the easily confused recognition list and calculate the similarity between the input audio embedding and the text embedding corresponding to each command word. In this process, cosine similarity or other suitable similarity measurement methods are usually used to quantify the degree of match between audio embedding and text embedding. For example, for the two command words "turn on the fan" and "turn off the fan", the system will calculate the similarity score between the current input audio embedding and the text embedding of these two command words.

[0046] Subsequently, the system will select the command word with the highest similarity score as the final recognition result and update the recognition to this command word. This method not only improves the accuracy of recognition, but also effectively solves the problem of misrecognition caused by blurred voice signals or environmental noise. In addition, in this way, the system can better handle complex user-defined command words, especially when there are multiple similar command words, it can still accurately understand the user's intention. This refined processing strategy significantly enhances the robustness and adaptability of the system, making the speech recognition system more intelligent and flexible in application scenarios such as smart homes and smart terminals. The entire process, through a series of carefully designed steps, ensures that from basic model training to the final precise verification, high-accuracy speech recognition services can be provided, thereby improving the user experience.

[0047] In one embodiment, the steps of training the recognition model on the general corpus, optimizing the recognition model based on the command word corpus, and fixing the model weights to obtain a fixed recognition model include: S30, preliminarily train the recognition model on general corpus; S31, optimizing the preliminarily trained model based on command word data including preset command words; S32. After the optimization is completed, the weight parameters of the model are fixed to form a fixed recognition model.

[0048] In this embodiment, in the process of building a command word recognition model, it is first necessary to conduct preliminary training on the recognition model on a general corpus. The goal of this stage is to establish a basic model with broad adaptability, which can handle daily conversations, news broadcasts and other content in multiple language environments, and ensure that the model has good generalization ability. In specific implementation, a large-scale general corpus containing rich vocabulary and grammatical structure changes is selected as a training data set, and a deep learning framework such as TensorFlow or PyTorch is used to implement the model building and training. The CTC (Connectionist Temporal Classification) algorithm is used for decoding, because it does not require alignment for sequence tasks, has fast calculation speed and low memory usage, and is particularly suitable for embedded devices. Through multiple rounds of iterative training, the model parameters are continuously adjusted until the loss value or WER (Word Error Rate) of the model on the validation set on the general corpus no longer decreases significantly, indicating that the model has fully learned the patterns and features in the general corpus.

[0049] After completing the initial training, we enter the optimization phase based on the command word corpus. At this point, in addition to the original general corpus, we also need to introduce a specially designed command word corpus, which covers the commonly used command words in smart home devices, such as "turn on the fan", "turn off the light", etc. Continue to use the model obtained in the first stage as a starting point, combine the general corpus with the command word corpus, and further fine-tune the model. In this process, pay special attention to the performance on the command word corpus, and continuously adjust the model parameters until the loss value or WER no longer decreases on the validation set of the command word corpus. The key to this step is to ensure that the model can be optimized for the command word recognition task while retaining the original generalization ability. For example, for a command word such as "turn on the fan", the model needs to be able to accurately identify the specific content of the command, rather than mistaking it for other similar commands.

[0050] Finally, after the optimization is completed, the weight parameters of the model are fixed to form a fixed recognition model. This process is designed to prevent these key parameters from being affected during subsequent training, thereby ensuring the accuracy and stability of the model when processing command words. The fixed model can be directly applied to actual scenarios to process user voice input in real time and execute corresponding commands. This gradual refinement strategy from general to specific fields not only improves the model's recognition ability for predefined command words, but also lays a solid foundation for subsequent steps, including the generation and alignment of audio and text embeddings, the application of self-attention mechanisms, and the final similarity comparison. The whole process emphasizes that the model must have both broad adaptability and the ability to provide accurate services in specific application scenarios, thereby providing users with a more intelligent and personalized interactive experience.

[0051] In one embodiment, the step of extracting an audio embedding of the input audio based on a fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and aligning them comprises: S40, extracting features from the input audio and generating audio embedding based on a fixed recognition model; S41, converting the input text into an embedding vector to generate text embedding; S42, map the audio embedding to the same dimensional space as the text embedding; S43. Use the Viterbi algorithm to align audio embedding and text embedding on the same timeline.

[0052] In this embodiment, in the process of extracting the audio embedding of the input audio based on the fixed recognition model, and mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space and aligning them, it is first necessary to extract features from the input audio and generate audio embeddings. Specifically, using the trained fixed recognition model, the model has good generalization ability and optimization ability for specific command words, and can extract feature representations containing rich semantic information from the input audio. These features usually come from the last layer or several layers of output of a deep neural network, for example, by extracting features before the audio linear layer to generate audio embeddings. This process ensures that the audio embedding can capture the phoneme-level details in the speech signal, providing a solid foundation for subsequent steps.

[0053] Next, the input text needs to be converted into an embedding vector to generate a text embedding. This step first involves converting the original text into a digital representation, such as splitting the text into words or character sequences through a tokenizer. Then, each word is converted into its corresponding embedding vector using a pre-trained word vector model such as Word2Vec, GloVe, or a more complex context-aware model such as lightweight BERT. In order for audio embeddings and text embeddings to be compared in the same dimensional space, it is necessary to ensure that the two have the same dimensional size. Therefore, methods such as projection matrices can be used to map text embeddings to the same dimensional space as audio embeddings. In this process, some technical means, such as multi-layer perceptrons (MLPs) or simple linear transformations, may be used to ensure that the two types of embeddings can operate under the same framework.

[0054] After completing the above steps, we enter the critical mapping and alignment stage. Map the audio embedding to the same dimensional space as the text embedding to ensure that the two can be compared in the same feature space. This process is usually implemented through linear transformation or other mapping methods, so that the audio embedding and text embedding are not only in the same dimensional space, but also can accurately reflect the semantic relationship between them. Subsequently, the Viterbi algorithm is used to align the audio embedding and text embedding on the same time axis. The Viterbi algorithm is a dynamic programming algorithm that is often used in sequence alignment tasks. It determines the most likely correspondence between audio and text by calculating the optimal path. For example, for a command word such as "turn on the fan", through the alignment process, we can determine which parts of the audio sequence correspond to "turn on", "open", "wind", and "fan". This precise alignment not only helps to improve the accuracy of command word recognition, but also enhances the system's ability to understand complex sentence structures.

[0055] Ultimately, through the aligned audio embedding and text embedding, the system can better understand the user's intention, laying the foundation for further optimizing the matching effect through the self-attention mechanism in the subsequent steps and realizing efficient recognition of custom vocabulary. The entire process plays a bridging role in the entire command word confirmation process, connecting the front-end audio feature extraction and the back-end advanced semantic processing, ensuring that the system has both broad adaptability and can provide accurate services in specific application scenarios. In this way, the robustness and user experience of the speech recognition system are significantly improved.

[0056] In one embodiment, the step of processing the aligned audio embedding by a self-attention mechanism to capture contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding includes: S50, applying a self-attention mechanism to the aligned audio embedding to capture contextual information in the audio sequence; S51, based on the audio embedding and text embedding processed by the self-attention mechanism, perform frame-by-frame similarity calculation in the same dimensional space; S52. Use the contrast loss function to adjust the distance between the audio embedding and the text embedding to optimize the matching effect between the audio embedding and the text embedding.

[0057] In this embodiment, in the process of processing the aligned audio embedding by the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding, it is first necessary to apply the self-attention mechanism to the aligned audio embedding. Specifically, the goal of this stage is to enhance the contextual understanding ability of the audio embedding for each time step in the sequence, so as to better reflect the semantic details in the speech signal. In the implementation process, the aligned audio embedding is first passed as input to the self-attention layer. The core of the self-attention layer is to calculate the query (Query, Q), key (Key, K) and value (Value, V), which are obtained by applying different linear transformations to the original audio embedding. Next, the similarity score between each pair of time steps is calculated, usually by dot product or scaled dot product (Scaled Dot-Product). The similarity score is then converted into a probability distribution through a softmax function, indicating the importance of each time step relative to other time steps. Based on this probability distribution, the value vector is weighted and summed to generate a new audio embedding representation, which contains relevant information about the current time step and all other time steps, thereby achieving effective capture of the context.

[0058] After completing the above steps, the stage of calculating the frame-by-frame similarity of the audio embedding and text embedding processed by the self-attention mechanism in the same dimensional space is entered. In order to ensure that the audio embedding and text embedding can be compared in the same feature space, they need to be mapped to the same dimensional space first. This step can be achieved through linear transformation or other mapping methods, so that the audio embedding and text embedding are not only in the same dimensional space, but also can accurately reflect the semantic relationship between them. Then, for each frame of the audio embedding and the corresponding text embedding, the system will calculate the similarity score between them. Commonly used similarity measurement methods include cosine similarity, Euclidean distance, etc. For example, cosine similarity is used to calculate the cosine value of the angle between each pair of audio embedding and text embedding. The closer the value is to 1, the higher the similarity. This frame-by-frame similarity calculation helps to accurately evaluate the degree of match between audio and text, and provides a basis for further optimization.

[0059] Finally, based on the frame-by-frame similarity calculation, the contrastive loss function is used to adjust the distance between the audio embedding and the text embedding to optimize the matching effect between the audio embedding and the text embedding. The contrastive loss function is a technique widely used in embedding learning, which aims to make the distance between the positive sample pairs (i.e., the correctly matched audio embedding and text embedding) as small as possible, while the distance between the negative sample pairs (i.e., the incorrectly matched audio embedding and text embedding) is as large as possible. In specific implementation, methods such as contrastive loss and asymmetric proxy loss (AsyPLoss) can be used. These loss functions update the model parameters through the back propagation algorithm to make the matching between the audio embedding and the text embedding more accurate. For example, in smart home devices, for command words such as "turn on the fan", the system can more accurately distinguish the correct command words instead of mistaking them for "turn off the fan". This method not only improves the accuracy of the command word recognition system, but also significantly enhances the robustness and adaptability of the system in complex environments, providing users with a more intelligent and personalized interactive experience. The entire process, from basic model training to the final fine verification, ensures the high accuracy of speech recognition and the good consistency of user experience.

[0060] In one embodiment, the step of comparing the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, identifying the command word as the corresponding custom vocabulary, includes: S60, comparing the similarity between the audio embedding and the text embedding frame by frame; S61, comparing the calculated similarity with a first preset threshold; S62: If the similarity exceeds a first preset threshold, the command word is recognized as a corresponding custom word, and a recognition result is output.

[0061] In this embodiment, first, for the case of customization, the user-defined text is calculated in advance as a sequence of text embeddings. This means that before actual use, the system has pre-processed and stored all possible text embeddings of custom words. These text embeddings are converted into vector representations of fixed dimensions by trained language models or embedding layers (such as BLSTM, BERT, etc.).

[0062] Next, the system compares the similarity of the audio embedding and text embedding corresponding to the speech input frame by frame. This step is achieved by calculating the similarity score between each pair of audio embeddings and the corresponding text embeddings. Common similarity measurement methods include cosine similarity and Euclidean distance. For example, cosine similarity is used to calculate the cosine value of the angle between each pair of audio embeddings and text embeddings. The closer the value is to 1, the higher the similarity.

[0063] Then, the calculated similarity is compared with the first preset threshold. This preset threshold is pre-set based on the actual application scenario and the characteristics of the data set, and is used to determine whether the current input audio matches a custom vocabulary. It is worth noting that since the custom vocabulary may not have undergone special intensive training, a lower threshold can be set in the initial stage to increase the recall rate (recall), that is, to identify as many qualified custom vocabulary as possible. Specifically, when the system recognizes a command word at the phoneme level, the threshold at this time can be set very low to ensure that as many potential matches as possible can be captured.

[0064] Finally, if the similarity exceeds the first preset threshold, the command word is identified as the corresponding custom vocabulary and the recognition result is output. Specifically, the system selects the custom vocabulary with the highest similarity score as the final recognition result and uses it as the basis for command execution. This method not only solves the problem of insufficient support for custom vocabulary in traditional speech recognition systems, but also significantly improves the accuracy of personalized command recognition and user experience. Especially in scenarios such as smart homes and smart terminals, users can easily define their own command words, making the interaction more natural and smooth.

[0065] Reference Figure 2 , is a structural block diagram of a command word confirmation device in an embodiment of the present application, the device includes: The training module 100 is used to train the recognition model on the general corpus, then optimize the recognition model based on the command word corpus, and fix the model weights to obtain a fixed recognition model; A mapping module 200 is used to extract an audio embedding of the input audio based on a fixed recognition model, map the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and align them; An optimization module 300 is used to process the aligned audio embedding through a self-attention mechanism to capture contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding; The comparison module 400 is used to compare the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary.

[0066] In one embodiment, the above device further includes a misidentification module, including a misidentification processing unit, configured to: Calculate the similarity between audio embedding and text embedding at the frame level; If there is a certain number of frames where the similarity is lower than a second preset threshold, it is determined to be a misrecognition and the current input audio is rejected.

[0067] In one embodiment, the above device further includes a mixed identification module, including a mixed identification processing unit, configured to: Classify based on the command word list and generate a confusing identification list; When the recognized command word is in the easily confused recognition list, traverse each command word in the easily confused recognition list; Calculate the similarity between the audio embedding and the text embedding of each command word separately; The command word with the highest similarity score is selected as the final recognition result, and the recognition is updated to the corresponding command word.

[0068] In one embodiment, the training module 100 includes a stage training unit for: Conduct preliminary training of the recognition model on general corpus; Optimizing the preliminarily trained model based on command word data including preset command words; After the optimization is completed, the weight parameters of the model are fixed to form a fixed recognition model.

[0069] In one embodiment, the mapping module 200 includes an embedded alignment unit, which is used to: Extract features from input audio and generate audio embedding based on a fixed recognition model; Convert the input text into an embedding vector and generate text embedding; Map audio embeddings to the same dimensional space as text embeddings; Using the Viterbi algorithm, the audio embeddings and text embeddings are aligned on the same timeline.

[0070] In one embodiment, the optimization module 300 includes a similarity calculation unit, which is used to: Apply self-attention mechanism to the aligned audio embeddings to capture contextual information in the audio sequence; Based on the audio embedding and text embedding processed by the self-attention mechanism, frame-by-frame similarity calculation is performed in the same dimensional space; The contrast loss function is used to adjust the distance between audio embedding and text embedding to optimize the matching effect between audio embedding and text embedding.

[0071] In one embodiment, the comparison module 400 includes a result output unit for: Compare the similarity between audio embedding and text embedding frame by frame; Comparing the calculated similarity with a first preset threshold; If the similarity exceeds a first preset threshold, the command word is recognized as a corresponding custom word, and a recognition result is output.

[0072] Reference Figure 3In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store usage data during the command word confirmation method, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the above-mentioned computer program is executed by a processor, a command word confirmation method is implemented, including the following steps: training a recognition model on a general corpus, then optimizing the recognition model based on the command word corpus, and fixing the model weights to obtain a fixed recognition model; extracting the audio embedding of the input audio based on the fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and aligning them; processing the aligned audio embedding through a self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding; comparing the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary.

[0073] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0074] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a command word confirmation method is implemented, including the following steps: training a recognition model on a general corpus, then optimizing the recognition model based on the command word corpus, and fixing the model weights to obtain a fixed recognition model; extracting the audio embedding of the input audio based on the fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and aligning them; processing the aligned audio embedding through a self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding; comparing the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0075] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0077] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A command word confirmation method, characterized in that: The method comprises: The recognition model is trained on the general corpus, then optimized based on the command word corpus, and the model weights are fixed to obtain a fixed recognition model. Extract audio embedding of the input audio based on a fixed recognition model, map the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and align them; The aligned audio embedding is processed through the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding. The audio embedding is compared with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary.

2. The command word confirmation method according to claim 1, characterized in that: After the step of processing the aligned audio embedding by the self-attention mechanism to capture the context information in the audio and optimize the matching effect between the audio embedding and the text embedding, the method further includes: Calculate the similarity between audio embedding and text embedding at the frame level; If there is a certain number of frames where the similarity is lower than a second preset threshold, it is determined to be a misrecognition and the current input audio is rejected.

3. The command word confirmation method according to claim 1, characterized in that: After the step of processing the aligned audio embedding by the self-attention mechanism to capture the contextual information in the audio and optimize the matching effect between the audio embedding and the text embedding, the method further includes: Classify based on the command word list and generate a confusing identification list; When the recognized command word is in the easily confused recognition list, traverse each command word in the easily confused recognition list; Calculate the similarity between the audio embedding and the text embedding of each command word separately; The command word with the highest similarity score is selected as the final recognition result, and the recognition is updated to the corresponding command word.

4. The command word confirmation method according to claim 1, characterized in that: The steps of training the recognition model on the general corpus, optimizing the recognition model based on the command word corpus, and fixing the model weights to obtain a fixed recognition model include: Conduct preliminary training of the recognition model on general corpus; Optimizing the preliminarily trained model based on command word data including preset command words; After the optimization is completed, the weight parameters of the model are fixed to form a fixed recognition model.

5. The command word confirmation method according to claim 1, characterized in that: The step of extracting audio embedding of the input audio based on the fixed recognition model, mapping the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and aligning them comprises: Extract features from input audio and generate audio embedding based on a fixed recognition model; Convert the input text into an embedding vector and generate text embedding; Map audio embeddings to the same dimensional space as text embeddings; Using the Viterbi algorithm, the audio embeddings and text embeddings are aligned on the same timeline.

6. The command word confirmation method according to claim 1, characterized in that: The step of processing the aligned audio embedding through the self-attention mechanism, capturing the context information in the audio, and optimizing the matching effect between the audio embedding and the text embedding includes: Apply self-attention mechanism to the aligned audio embeddings to capture contextual information in the audio sequence; Based on the audio embedding and text embedding processed by the self-attention mechanism, frame-by-frame similarity calculation is performed in the same dimensional space; The contrast loss function is used to adjust the distance between audio embedding and text embedding to optimize the matching effect between audio embedding and text embedding.

7. The command word confirmation method according to claim 1, characterized in that: The step of comparing the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, identifying the command word as the corresponding custom vocabulary, comprises: Compare the similarity between audio embedding and text embedding frame by frame; Comparing the calculated similarity with a first preset threshold; If the similarity exceeds a first preset threshold, the command word is recognized as a corresponding custom word, and a recognition result is output.

8. A command word confirmation device, characterized in that: include: The training module is used to train the recognition model on the general corpus, then optimize the recognition model based on the command word corpus, and fix the model weights to obtain a fixed recognition model; A mapping module is used to extract the audio embedding of the input audio based on a fixed recognition model, map the audio embedding and the text embedding corresponding to the input text into the same dimensional space, and align them; The optimization module is used to process the aligned audio embeddings through the self-attention mechanism, capture the contextual information in the audio, and optimize the matching effect between the audio embedding and the text embedding; A comparison module is used to compare the audio embedding with the text embedding of the input text of the custom vocabulary for similarity, and if the similarity exceeds a first preset threshold, the command word is identified as the corresponding custom vocabulary.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images