Transformer-based audio-visual fusion robot wake-up word recognition method and device

Abstract

The present disclosure relates to a Transformer-based audio-visual fusion robot wake-up word recognition method and device, and relates to the field of audio and video signal processing. The method comprises: extracting audio features and video features in the current environment; based on the self-attention operation and classification operation of the Transformer-based classification network, respectively obtaining the audio wake-up word classification vector and the video wake-up word classification vector of the audio features and the video features; based on the attention weight distribution in the time dimension when performing the self-attention operation on the audio features, the time of the occurrence of the wake-up word is obtained. The present disclosure not only realizes high-precision and robust classification-based wake-up word recognition, but also accurately locates the time of the occurrence of the wake-up word.

Description

Technical Field

[0001] This invention belongs to the field of audio and video signal processing, specifically relating to a method and device for recognizing wake words in an audiovisual fusion robot based on Transformer. Background Technology

[0002] Wake word recognition aims to determine whether a specific wake word is contained in a speech signal using certain technical methods. This technology has gradually become an important part of people's daily lives, with wide applications in smartphones, portable wearable devices, smart homes, and automotive control systems. Human-computer interaction via wake words can significantly reduce device resource consumption; an accurate and robust wake word recognition system is a prerequisite and foundation for human-computer interaction. In recent years, wake word recognition technology has attracted considerable attention from researchers and companies.

[0003] Audio signals are easily affected by noise, and the performance of a speech keyword recognition system will significantly degrade in complex scenarios such as noise and far-field conditions. Human perception of information comes from multiple senses, and when processing audio information, it is often influenced by visual information. Audiovisual fusion technology, by collaboratively utilizing information from both visual and auditory modalities to achieve specific speech tasks, can effectively solve the problem of noise interference. Using audiovisual fusion technology for wake word recognition can improve the accuracy and robustness of recognition, making human-computer interaction more efficient and natural.

[0004] Traditional wake word recognition methods are mainly based on large-scale corpus speech recognition, Hidden Markov Models (HMMs), and Gaussian Mixture Models (GMMs). With the development of deep learning technology, data-driven deep learning methods have become the mainstream for wake word recognition. Deep learning-based methods typically consist of a front-end feature extraction network and a back-end recognition network. The raw speech signal contains a large amount of redundant information and is not suitable as direct input to the model. The purpose of feature extraction is to obtain distinctive features to serve subsequent tasks. In most studies, widely used feature extraction methods can be divided into two categories. One category is handcrafted features derived through mathematical derivation, such as the spectrum and Mel-frequency cepstral coefficients (MFCCs). The other category is features obtained through learning. This type of method treats feature extraction as a deep learning model, typically training different models for different tasks. These models directly use the raw audio signal as input to obtain output feature vectors. The vector extracted by the feature extractor represents the original signal, and the performance of the feature extraction module directly affects the performance of the entire system. After feature extraction, these features are used as input to the back-end recognition model. Traditional methods often use Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) to encode and decode features to obtain the posterior probability of whether an input feature belongs to a wake-up word. Deep learning-based methods simplify wake-up word recognition to a binary classification problem, essentially using a deep neural network to distinguish between positive and negative features. In this classification framework, the Categorical Cross Entropy (CCA) loss function is commonly used to train the wake-up word recognition model. The basic framework of audiovisual fusion technology is similar to the feature extraction-feature classification framework mentioned above, processing audio and video separately before fusing the two modalities. Video feature extraction is similar to audio feature extraction, representing video (images) as a set of vectors using some method. Video contains information in two aspects: temporal changes and the inherent spatial information within each frame. Therefore, video feature extraction is often more complex than audio feature extraction, as the extracted features simultaneously contain spatiotemporal information. Common video feature extraction methods can be divided into geometric shape-based features and motion-based features, extracting features of moving subjects in the video by focusing on changes in object shape and the spatiotemporal similarity between pixels. Of course, like audio feature extraction, video feature extraction based on deep learning models has also gradually become mainstream in recent years. Currently, common modality fusion methods can be divided into two main categories: decision fusion and feature fusion. Decision fusion refers to first having two independent models, one for audio and one for video, output their respective prediction results, and then fusing the two results using a certain method to obtain the final audiovisual fusion prediction result.Feature fusion combines feature vectors extracted from audio and video using methods such as concatenation and averaging into a new vector, which is then used as input to subsequent models. Besides these two basic approaches, some studies have proposed more complex hybrid fusion methods to fully integrate information from both visual and auditory modalities.

[0005] While audiovisual fusion methods can effectively enhance the performance of a speech system, some problems remain. For example, for syllables with similar pronunciations (such as "p" and "b"), the lip movements may be identical, making it impossible to accurately predict audio information using only video information. Furthermore, visual and auditory information exhibit a degree of asynchrony. Therefore, how to fully utilize audio and video information tailored to task characteristics and improve the reliability of fused information is a key issue for audiovisual fusion systems. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a Transformer-based audiovisual fusion robot wake word recognition method and device, which can achieve accurate recognition of robot wake words and precise localization of wake words in unconstrained contexts.

[0007] The technical solution adopted in this invention is as follows:

[0008] A method for recognizing wake words in an audiovisual fusion robot based on Transformer, the method comprising:

[0009] Extract audio and video features from the current environment;

[0010] The self-attention operation and classification operation of the Transformer-based classification network are used to obtain audio wake word classification vectors and video wake word classification vectors based on audio features and video features, respectively.

[0011] The fusion results of the audio wake word classification vector and the video wake word classification vector are classified to obtain the wake word recognition result.

[0012] Furthermore, the extraction of audio features in the current environment includes:

[0013] Train the WavLM model, which consists of a 7-layer convolutional feature encoder and a Transformer encoder with a relative position offset control gate;

[0014] The audio signal from the current environment is fed into the trained WavLM model to obtain audio features.

[0015] Furthermore, the extraction of video features in the current environment includes:

[0016] Training the AV-HuBERT model;

[0017] The video signal from the current environment is fed into the trained AV-HuBERT model so that the video signal passes through the ResNet-Transformer hybrid structure in the AV-HuBERT model to obtain video features.

[0018] Furthermore, the Transformer-based classification network calculates audio wake word classification vectors and video wake word classification vectors based on audio and video features, respectively, including:

[0019] A learnable special classification code is concatenated at the very beginning of both the audio and video features;

[0020] The video features with this special classification code and the video features are input into the Transformer encoder based on the self-attention mechanism;

[0021] In the output of the Transformer encoder, the encoded vector of the first dimension is used as the audio encoding result or the video encoding result;

[0022] The audio encoding result and the video encoding result are respectively fed into the MLP network to obtain the audio wake word classification vector and the video wake word classification vector.

[0023] Further, the classification of the fusion result of the audio wake word classification vector and the video wake word classification vector to obtain the wake word recognition result includes:

[0024] The audio wake word classification vector and the video wake word classification vector are fused by decision to obtain a fusion result; wherein the weights in the decision fusion are obtained through training;

[0025] Using Softmax, the posterior probability of the fusion result is calculated;

[0026] The wake word recognition result is obtained based on the posterior probability.

[0027] Furthermore, the method also includes:

[0028] The attention weight distribution in the time dimension is obtained when the audio features are subjected to self-attention operation;

[0029] When the weight matrix at a certain point in the attention weight distribution is greater than a set threshold, that point in time is taken as the time when the wake word appears.

[0030] A Transformer-based audiovisual fusion robot wake word recognition device, the device comprising:

[0031] The feature extraction module is used to extract audio and video features in the current environment;

[0032] The feature classification module is used for the self-attention operation and classification operation of the Transformer-based classification network, and obtains the audio wake word classification vector and video wake word classification vector of audio features and video features respectively.

[0033] The fusion classification module is used to classify the fusion result of the audio wake word classification vector and the video wake word classification vector to obtain the wake word recognition result.

[0034] Furthermore, the device also includes:

[0035] The wake word localization module is used to obtain the attention weight distribution in the time dimension when the audio features are subjected to self-attention operation, and when the weight matrix of a certain time point in the attention weight distribution is greater than a set threshold, the time point is taken as the time when the wake word appears.

[0036] An electronic device includes a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the Transformer-based audiovisual fusion robot wake word recognition method described above.

[0037] A computer-readable storage medium storing computer program instructions thereon, characterized in that, when the computer program instructions are executed, they implement the Transformer-based audiovisual fusion robot wake word recognition method described above.

[0038] Compared with the prior art, the present invention has at least the following advantages:

[0039] 1) This invention extracts audio and video features through two pre-trained models and then uses a Transformer-based classification network to classify single-modal wake words. It also designs a decision fusion method based on a multilayer perceptron, thereby achieving high-precision and robust classification-based wake word recognition.

[0040] 2) Wake word localization was achieved based on the self-attention weight matrix. Attached Figure Description

[0041] Figure 1 The overall flowchart of the method of this invention. Detailed Implementation

[0042] To describe the method of the present invention more clearly and completely, the present invention will be further explained below with reference to specific embodiments and accompanying drawings.

[0043] This invention employs pre-trained models WavLM and AV-HuBert to extract features from audio and video respectively. A Transformer-based classification network is used to classify the audio and video features of unconstrained sentences into specified wake word categories. An audiovisual fusion network based on Multi-Layer Perception (MLP) is used to obtain the posterior probability that the input audio and video segments contain wake words. The timing of wake word appearance in audio is determined by a self-attention weight matrix.

[0044] Specifically, such as Figure 1 As shown, the audiovisual fusion robot wake word recognition method of the present invention includes the following steps.

[0045] Step 1: Extract audio and video features from the current environment.

[0046] In this invention, WavLM is used as the speech feature extractor, and AV-HuBERT is used as the video feature extractor. WavLM is a pre-trained speech model that is pre-trained on 13,000 hours of speech data by predicting masked speech segments and distinguishing overlapping speech. Its design aims to obtain a universal feature representation for different downstream speech tasks. This model consists of a 7-layer convolutional feature encoder and a Transformer encoder with relative position offset control gates. Similar to WavLM, AV-HuBERT is a self-supervised audiovisual fusion speech recognition framework based on HuBERT. Since there is currently no universal pre-trained lip feature extractor, the video portion of AV-HuBERT is used as the video feature extractor. This model encodes image sequences into video features using a ResNet-Transformer hybrid architecture.

[0047] The input to the feature extractor is the raw audio and video signal, with the following dimensions:

[0048]

[0049] Where T represents the duration of the signal, f a and f v Let h and w represent the frequencies of audio and video, respectively, and h×w represent the video resolution. After passing through the feature extractor, the resulting feature dimensions are:

[0050]

[0051] Where T a and T v These represent the time dimension of the feature, which changes with T. D a and D vThis refers to the feature dimensions of audio and video features. In this method, D... a =1024,D v =728.

[0052] Step 2: Perform self-attention and classification operations on the Transformer-based classification network to obtain audio wake word classification vectors and video wake word classification vectors for audio and video features, respectively.

[0053] The wake-word classification network in this invention is implemented using a Transformer-based encoder and an MLP-based classifier. Audio and video features are input into a 4-layer, 8-head Transformer encoder. Before inputting into the model, a learnable special classification code (CLS) is concatenated to the front end of each feature. The purpose of adding this code is to allow it to learn the global representation of the entire feature during training, encompassing variable-length sequence information within a fixed-length code. The operation of the Transformer encoder can be represented by the following formula:

[0054]

[0055]

[0056] In the encoder's output, only the dimension corresponding to CLS is extracted separately and used as the input to the subsequent MLP classifier. Therefore, after the encoder, the audio and video features are encoded into two fixed-length feature encoding vectors Z. a and Z v These two feature vectors are then fed into an MLP network consisting of two fully connected layers with ReLU activation to obtain an audio wake-word classification vector. With video wake word classification vector

[0057]

[0058]

[0059] Specifically, step 2 can be divided into four steps:

[0060] a) Add a learnable classification code before the audio and video features extracted in step 1;

[0061] b) Input the vector with this encoding into a Transformer encoder based on a self-attention mechanism;

[0062] c) In the output of the Transformer encoder, select the encoded vector of the first dimension as the input of the MLP-based classifier;

[0063] d) Obtain audio classification predictions and video classification predictions.

[0064] Step 3: Classify the fusion result of the audio wake word classification vector and the video wake word classification vector to obtain the wake word recognition result.

[0065] This invention utilizes a decision fusion-based audiovisual fusion network to obtain wake word recognition results. This network is implemented using a multilayer perceptron, taking single-modal wake word classification predictions of audio and video as input, and outputting the decision fusion result, i.e., the posterior probability that the current audio / video segment contains a certain wake word.

[0066] Specifically, this invention employs vector concatenation and a fully connected layer as the fusion layer. Let the number of categories be N, then... and Concatenating the vectors results in a vector of length 2N. As input to the fusion layer, the parameters of the fully connected layer are (2N, N), which remaps the fused vector back to the number of classes.

[0067]

[0068] Will The probabilistic form of the audiovisual fusion wake-word classification result can be obtained using a Softmax function.

[0069]

[0070] In addition, the present invention may also include step 4: the distribution of attention weights in the time dimension when performing self-attention operation based on audio features, to obtain the time when the wake word appears.

[0071] This step is implemented based on the self-attention weight matrix of the Transformer network. By analyzing the numerical distribution of the weight matrix in the last attention layer of the Transformer, the timing of the wake word's appearance in the input audio can be inferred.

[0072] Specifically, based on the structure of the Transformer encoder, during the self-attention operation, the input vector will first W q W k W v These three projection parameter matrices are multiplied. Assume... The input to the self-attention layer yields three matrices Q, K, and V:

[0073]

[0074] In the classic Transformer encoder architecture, Q,K∈RT×d Therefore, we can define the self-attention weight matrix:

[0075] M = Q × K T ∈R T×T .

[0076] M represents the distribution of attention weights for the current input. The part with the largest attention weight is the part that the model focuses on most. By analyzing the distribution of attention scores over time, the location of the wake word in a sentence can be inferred. A wake word is considered to exist when the value in the weight matrix is ​​greater than a threshold τ.

[0077] In summary, this invention achieves high-precision and robust classification-based wake word recognition. It extracts audio and video features using two pre-trained models; implements single-modal wake word classification using a Transformer-based classification network; designs a multilayer perceptron-based decision fusion method to fuse the prediction results from audio and video; and achieves wake word localization based on a self-attention weight matrix.

[0078] Based on the same inventive concept, this invention also discloses a Transformer-based audiovisual fusion robot wake word recognition device, characterized in that the device includes:

[0079] The feature extraction module is used to extract audio and video features in the current environment;

[0080] The feature classification module is used for the self-attention operation and classification operation of the Transformer-based classification network, and obtains the audio wake word classification vector and video wake word classification vector of audio features and video features respectively.

[0081] The fusion classification module is used to classify the fusion result of the audio wake word classification vector and the video wake word classification vector to obtain the wake word recognition result.

[0082] In one example, the device further includes a wake word localization module, which is used to obtain the attention weight distribution in the time dimension when the audio features perform self-attention operation, and when the weight matrix at a certain time point in the attention weight distribution is greater than a set threshold, the time point is taken as the time when the wake word appears.

[0083] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0084] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A method for recognizing wake words in a robot based on Transformer-based audiovisual fusion, characterized in that, The method includes: Extract audio and video features from the current environment; The Transformer-based classification network performs self-attention and classification operations to obtain audio and video wake-up word classification vectors for audio and video features, respectively. Specifically, the Transformer-based classification network calculates the audio and video wake-up word classification vectors for audio and video features, respectively, including: A learnable special classification code is concatenated at the very beginning of both the audio and video features; The video features with this special classification code and the video features are input into the Transformer encoder based on the self-attention mechanism; In the output of the Transformer encoder, the encoded vector of the first dimension is used as the audio encoding result or the video encoding result; The audio encoding result and the video encoding result are respectively fed into the MLP network to obtain the audio wake word classification vector and the video wake word classification vector; The fusion results of the audio wake word classification vector and the video wake word classification vector are classified to obtain the wake word recognition result.

2. The method as described in claim 1, characterized in that, The extraction of audio features in the current environment includes: Train the WavLM model, which consists of a 7-layer convolutional feature encoder and a Transformer encoder with a relative position offset control gate; The audio signal from the current environment is fed into the trained WavLM model to obtain audio features.

3. The method as described in claim 1, characterized in that, The extraction of video features in the current environment includes: Training the AV-HuBERT model; The video signal from the current environment is fed into the trained AV-HuBERT model so that the video signal passes through the ResNet-Transformer hybrid structure in the AV-HuBERT model to obtain video features.

4. The method as described in claim 1, characterized in that, The classification of the fusion results of the audio wake-up word classification vector and the video wake-up word classification vector to obtain the wake-up word recognition result includes: The audio wake word classification vector and the video wake word classification vector are fused by decision to obtain a fusion result; wherein the weights in the decision fusion are obtained through training; Using Softmax, the posterior probability of the fusion result is calculated; The wake word recognition result is obtained based on the posterior probability.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The attention weight distribution in the time dimension is obtained when the audio features are subjected to self-attention operation; When the weight matrix at a certain point in the attention weight distribution is greater than a set threshold, that point in time is taken as the time when the wake word appears.

6. A Transformer-based audiovisual fusion robot wake word recognition device, characterized in that, The device includes: The feature extraction module is used to extract audio and video features in the current environment; The feature classification module is used for self-attention and classification operations of the Transformer-based classification network to obtain audio wake-up word classification vectors and video wake-up word classification vectors for audio and video features, respectively. The Transformer-based classification network calculates the audio wake-up word classification vectors and video wake-up word classification vectors for audio and video features, respectively, including: A learnable special classification code is concatenated at the very beginning of both the audio and video features; The video features with this special classification code and the video features are input into the Transformer encoder based on the self-attention mechanism; In the output of the Transformer encoder, the encoded vector of the first dimension is used as the audio encoding result or the video encoding result; The audio encoding result and the video encoding result are respectively fed into the MLP network to obtain the audio wake word classification vector and the video wake word classification vector; The fusion classification module is used to classify the fusion result of the audio wake word classification vector and the video wake word classification vector to obtain the wake word recognition result.

7. The apparatus as claimed in claim 6, characterized in that, The device further includes: The wake word localization module is used to obtain the attention weight distribution in the time dimension when the audio features are subjected to self-attention operation, and when the weight matrix of a certain time point in the attention weight distribution is greater than a set threshold, the time point is taken as the time when the wake word appears.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the Transformer-based audiovisual fusion robot wake word recognition method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed, they implement the Transformer-based audiovisual fusion robot wake word recognition method as described in any one of claims 1 to 5.

Images