A method and device for wake-up word recognition in audiovisual fusion for multi-person scenarios

Abstract

This invention discloses a method and apparatus for wake word recognition in multi-person scenarios using audiovisual fusion. The method includes: acquiring original audio and lip movement videos of each person in the multi-person scenario; and feeding the original audio and lip movement videos into a wake word recognition network to obtain wake word recognition results. This invention introduces an attention mechanism to calculate a temporal score matrix and detects potential speakers by comparing the audiovisual scores of each frame. It also uses knowledge distillation to transfer knowledge from a large model to a lightweight model on the device, thereby reducing the computational complexity of the model.

Description

Technical Field

[0001] This invention belongs to the field of audio and video signal processing, specifically relating to a method and device for audiovisual fusion wake-up word recognition in multi-person scenarios. Background Technology

[0002] Wake word recognition detects the presence of a specified wake word in a utterance by classifying it into a predefined set of words. In recent years, with the rapid development of artificial intelligence, wake word recognition technology has been widely applied in various fields, such as mobile phones, voice assistants, intelligent robots, and smart home devices. For example, Microsoft's Cortana and Amazon's Alexa rely on specific wake words for activation to facilitate further human-computer interaction. Furthermore, to enable faster machine response times, wake word recognition is typically performed on portable devices. Compared to Automatic Speech Recognition (ASR), wake word recognition does not require recognizing the entire input utterance, thus significantly reducing the computational cost of unnecessary operations. Early research in wake word recognition often utilized pure audio information to identify wake words. In noise-free, pure audio environments, wake word recognition performance far surpasses the capabilities of the human auditory system. Although pure audio methods are commonly used for wake word recognition in daily life, methods relying solely on audio patterns face insurmountable drawbacks. For example, in various complex acoustic scenarios, especially in noisy environments, wake word recognition performance significantly degrades. Moreover, factors such as the number of speakers, their gender, age, and speaking style also affect wake word recognition. With the development of multimodal technologies, audiovisual fusion methods are considered the most promising solution for robust wake word recognition. The human speech perception system is bimodal, relying on both audio and visual information. Therefore, in everyday face-to-face communication, observable articulatory organs such as the lips are also important sources of information. Visual information is particularly effective in real-world scenarios with severe acoustic distortions (such as strong background noise and mixing) because it is unaffected by these distortions. Therefore, speech processing systems can use visual and audio information jointly to improve the performance of wake word recognition in various complex scenarios.

[0003] Audio-Visual Wake Word Spotting (AV-WWS) utilizes lip movement and audio information to identify the presence of a specified wake word in a video stream. Traditional AV-WWS methods use manually labeled lip movement features and audio spectral features as input to a Hidden Markov Model (HMM) to achieve the AV-WWS effect. However, because traditional AV-WWS methods rely on manually extracted features, their general applicability is limited. Furthermore, the accuracy of traditional methods is far from meeting the requirements of practical applications when used for model deployment. Modifying these manually created models for different situations is challenging, as these models are created for specific objectives. To address these issues, deep learning-based AV-WWS wake word spotting has attracted widespread attention. Deep learning-based AV-WWS wake word spotting methods are data-driven and therefore adapt well to the audio and video signals present in the training data. Although significant progress has been made in AV-WWS wake word spotting technology, most existing work focuses on single-person scenarios. However, in daily life, it is extremely common for multiple people to be present in human-computer interaction environments simultaneously. When multiple people are present in a scene, existing audiovisual fusion wake-word recognition models struggle to accurately identify wake-words using the Region-Of-Interest (ROI) sequence around the mouth. Therefore, the visual redundancy that is difficult to process effectively causes a sharp decline in the performance of audiovisual fusion wake-word recognition models designed for single-person scenarios. This presents a new challenge for audiovisual fusion-based wake-word recognition tasks. Furthermore, audiovisual fusion wake-word recognition models typically need to be deployed on portable devices. Therefore, model size and inference time are crucial. However, current audiovisual fusion wake-word recognition methods often train large models, resulting in slow prediction speeds on portable devices. This also poses a challenge to the audiovisual fusion wake-word recognition task. Summary of the Invention

[0004] To address the challenges of wake-up word recognition and lightweight model design in multi-user scenarios involving audiovisual fusion, this invention proposes a wake-up word recognition method and apparatus for multi-user scenarios involving audiovisual fusion, which reduces computational complexity while achieving excellent performance.

[0005] The invention includes:

[0006] A wake-word recognition method for multi-person scenarios based on audiovisual fusion, characterized in that the method includes:

[0007] Collect raw audio and video of each person's lip movements in multi-person scenarios;

[0008] The original audio and the lip movement video are fed together into a wake word recognition network to obtain a wake word recognition result; wherein, the wake word recognition network includes:

[0009] A video feature extraction network is used to extract video features from each frame of a lip movement video;

[0010] An audio feature extraction network is used to extract audio features from the original audio.

[0011] The fusion and recognition network is used to model the sum of all video features and the audio features in the time dimension through a multi-head attention mechanism, fuse the modeled video feature sequence and audio feature sequence, and obtain the wake word recognition result based on the fusion result.

[0012] Furthermore, the extraction of video features from each frame of the lip movement video includes:

[0013] The spatiotemporal information of the lip movement video was extracted using a 3D convolutional network.

[0014] The spatiotemporal information is passed through a residual network feature encoder with several layers to obtain the video features of each frame.

[0015] Furthermore, the step of modeling the temporal dimension of the summation result of all video features using a multi-head attention mechanism includes:

[0016] The video features of each frame in all lip movement videos are summed to obtain the summation result;

[0017] The summation structure is fed into a Transformer encoder using a multi-head attention mechanism to obtain a video feature sequence.

[0018] Further, the extraction of the audio feature sequence from the original audio includes:

[0019] The original audio is fed into the TC-ResNet model based on a CNN network to obtain the audio feature sequence.

[0020] Furthermore, the fused video feature sequence and audio feature sequence are used to obtain the wake-up word recognition result based on the fusion result, including:

[0021] Use a K-max Pooling layer to extract the n frames of data with the largest feature values ​​in the video feature sequence;

[0022] Use another K-max Pooling layer to extract the n frames of data with the largest feature values ​​in the audio feature sequence;

[0023] The n frames of data with the largest feature value in the video feature sequence and the n frames of data with the largest feature value in the audio feature sequence are input into a fully connected layer to obtain the predicted probability of each wake word.

[0024] Based on the predicted probability, the wake word recognition result is obtained.

[0025] Furthermore, the wake word recognition network is trained through the following steps:

[0026] Construct a training dataset; the training dataset includes audio samples from multi-person or single-person scenarios and video samples of each person's lip movements, as well as the real labels of the corresponding wake words;

[0027] The one-hot vector of the real label is used as the hard label of the wake word, and the soft label of the wake word is obtained by performing softmax normalization on the hard label.

[0028] Construct a teacher wake word recognition network; the teacher wake word recognition network includes:

[0029] An MFCC feature extraction network is used to extract MFCC features from the audio samples;

[0030] A video feature extraction network is used to extract sample video features from each frame of a video sample of an active speaker.

[0031] An active speaker detection network is used to obtain active speakers based on the MFCC features and the sample video features;

[0032] An audio feature extraction network is used to extract sample audio features from the audio samples.

[0033] A fusion and recognition network is used to model the sum of all sample video features and the sample audio features in the time dimension through a multi-head attention mechanism, fuse the modeled sample video feature sequence and sample audio feature sequence, and obtain the wake word recognition result based on the fusion result;

[0034] Based on the wake word recognition results of the sample data obtained by the teacher wake word recognition network, the training results of the wake word recognition network are supervised, and the cross-entropy loss function and Kullback-leibler loss function are used to minimize the loss of hard labels and soft labels.

[0035] Further, the step of obtaining the active speaker based on the MFCC features and the video features of the video sample includes:

[0036] The MFCC features and the sample video features are passed through a linear layer to obtain new audio features Q and new video features K;

[0037] The new audio feature Q is expanded in dimension using Python's broadcast mechanism to obtain the new audio feature Q. ′ ;

[0038] For the new audio feature Q ′ The time score matrix S is obtained by performing element-wise dot product with the new video feature Q.

[0039] The score matrix S is summed along the audio time dimension to obtain the score matrix S. ′ ;

[0040] Based on the number of people in the scene, the scoring matrix S ′ Normalize and then find the index of the maximum value of the average normalization result across the visual dimension;

[0041] The active speaker is obtained based on the index of each video sample in the same scene.

[0042] A wake-word recognition method for multi-person scenarios based on audiovisual fusion, the device comprising:

[0043] The data acquisition module is used to acquire raw audio and video of each person's lip movements in multi-person scenarios.

[0044] A data recognition module is used to send the original audio and the lip movement video together into a wake-word recognition network to obtain a wake-word recognition result; wherein, the wake-word recognition network includes:

[0045] A video feature extraction network is used to extract video features from each frame of a lip movement video;

[0046] An audio feature extraction network is used to extract audio features from the original audio.

[0047] The fusion and recognition network is used to model the sum of all video features and the audio features in the time dimension through a multi-head attention mechanism, fuse the modeled video feature sequence and audio feature sequence, and obtain the wake word recognition result based on the fusion result.

[0048] An electronic device includes a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the wake word recognition method for multi-person scenarios based on audiovisual fusion as described above.

[0049] A computer-readable storage medium storing computer program instructions thereon, characterized in that, when the computer program instructions are executed, they implement the wake word recognition method for multi-person scenarios based on audiovisual fusion as described above.

[0050] Compared with existing technologies, this invention proposes an active speaker detection module based on an attention mechanism for audiovisual fusion and a model compression method based on knowledge distillation. By introducing an attention mechanism to calculate the temporal score matrix and comparing the audiovisual scores of each frame, potential speakers are detected. Knowledge distillation transfers knowledge from a large model (teacher model) to a lightweight model (student model) on the device, thereby reducing the computational complexity of the model. Compared with traditional methods based on manual feature extraction, the proposed audiovisual model achieves superior performance. Attached Figure Description

[0051] Figure 1 The overall flowchart of the method of this invention. Detailed Implementation

[0052] To describe the method of the present invention more clearly and completely, the present invention will be further explained below with reference to specific embodiments and accompanying drawings.

[0053] The audiovisual fusion wake word recognition method of the present invention reduces computational complexity by transferring knowledge from a large model (teacher wake word recognition network) to a lightweight model (wake word recognition network) on the device through knowledge distillation.

[0054] like Figure 1 As shown, the present invention includes the following steps 1-2.

[0055] Step 1: Collect the original audio and video of each person's lip movements in a multi-person scene.

[0056] This invention uses electronic devices to collect raw audio and video of each person's lip movements when multiple people are simultaneously present in a human-computer interaction environment.

[0057] Step 2: Send the original audio and the lip movement video together into the wake word recognition network to obtain the wake word recognition result.

[0058] 1) Structure of the wake word recognition network.

[0059] The wake word recognition network of the present invention includes: a video feature temporal modeling network, an audio feature temporal modeling network, and a fusion and recognition network.

[0060] The video feature extraction network utilizes a 3D convolutional network to extract the spatiotemporal information of lip movements, and then uses an 18-layer residual network feature encoder to extract the internal information of each frame. The 3D convolutional layers each have 16 feature channels.

[0061] The audio feature extraction network uses a lightweight CNN-based model, TC-ResNet, whose input is the raw audio after global mean and variance normalization, with a sampling rate of 16kHz, and the output feature dimension is 128-dimensional.

[0062] The fusion and recognition network first performs temporal modeling on the audio and video features extracted from the front end, utilizing four encoder blocks. Each block contains a multi-head self-attention module and two multi-head cross-encoders. Then, the temporally modeled audio and visual features are concatenated using an MLP. It is important to note that this method requires applying LayerNorm to each modality before multimodal fusion, and then concatenating them along the feature dimension. This effectively avoids a single modality covering the entire feature set with a large variance.

[0063] Then, the fusion and recognition network uses audio and video information, after temporal modeling, to make fusion decisions. The classification layer consists of two layers: a K-max Pooling layer and a fully connected layer. The K-max Pooling layer selects the 50 frames with the largest feature values ​​after feature extraction as the feature basis for the final wake-word recognition. The fully connected layer compresses the audio and video networks into multiple output units representing the number of wake-words, and uses their outputs as the prediction probabilities for the audio feature extraction network and the visual feature extraction network, respectively. By equally weighting the posterior probabilities from the audio and video networks, audio and video fusion is achieved.

[0064] Furthermore, during classification, since this method uses a decision fusion approach, and the input feature sizes and parameter counts of the feature extraction network differ between the two visual and auditory modes, this can lead to different convergence speeds for different modes. One mode is prone to overfitting. This method adds parameters to limit overfitting to the auditory mode, which converges relatively quickly, in order to slow down the convergence speed.

[0065] 2) Training of the wake word recognition network.

[0066] This invention is based on a teacher wake-up word recognition network with a similar structure to the wake-up word recognition network, and trains the wake-up word recognition network using a knowledge distillation method. In addition to the video feature extraction network, audio feature extraction network, and fusion and recognition network mentioned above, the teacher wake-up word recognition network also includes an MFCC feature extraction network and an active speaker detection network. The fusion and recognition network of the teacher wake-up word recognition network has the same structure as the fusion and recognition network described above.

[0067] In the video feature extraction network of the teacher wake word recognition network, a 3D convolutional network was also used, but the number of feature channels in the 3D convolutional layer was changed from 16 to 64.

[0068] In the audio feature extraction network of the teacher wake word recognition network, Wav2vec, which is pre-trained on the large speech dataset Librispeech, is used. This model is able to extract audio features very well.

[0069] The input to the MFCC feature extraction network is the MFCC corresponding to the original audio, and the output is the MFCC feature.

[0070] The active speaker detection network first opens up a dimension for the number of speakers using visual features obtained from the video feature extraction network. Visual features are stacked on this newly opened dimension to obtain visual features for multi-person scenes. Then, the obtained visual features V and audio features A extracted by MFCC are passed through a linear layer to transform them into new features Q and K. Next, Python's broadcast mechanism is used to expand the dimensionality of the audio features, allowing for the dot product of features K and Q to obtain a temporal score matrix S. When multiple people appear on the screen simultaneously, this score matrix is ​​used to determine and select the active speaker. Finally, the score matrix S is summed along the audio temporal dimension to obtain a score matrix S' calculated from each frame of the lip movement video and the overall audio features. This score matrix is ​​then normalized, and the face with the highest score is selected as the active speaker. Since the module's input includes both visual and audio information, attention scores from audio and video are used to determine which person is speaking. The attention score is calculated, normalization is performed, and the index corresponding to the active speaker is derived using the following formula:

[0071]

[0072]

[0073]

[0074]

[0075] In the formulas, all symbols and subscripts are represented in lowercase. The meanings of the subscripts are explained below: t a t represents the audio feature dimension. v q represents the visual feature dimension, and t represents the visual feature t. v The dimension obtained after passing through the linear layer, k, represents the audio feature t. a The dimension obtained after passing through the linear layer, n, represents the total number of faces (including lip counts) in the video. Now, let's explain the symbols in the formula. The attention score represents the feature score. and characteristics Calculated, due to features and characteristics The dimensions may be inconsistent, so it is multiplied by a transformation matrix W. qk To ensure the calculation proceeds normally, It is the attention score matrix The result is obtained by superimposing audio features, and it will be used for subsequent selection of active speakers. yes The result after normalization in the dimension of face count (lip count dimension), N max It is an index that is the maximum value of the normalized features after averaging in the visual dimension. The active speaker can be selected by using this index. This invention believes that the person with the highest score is considered a potential active speaker.

[0076] In the specific training process, the one-hot vectors corresponding to the true labels in the training set are used as hard labels. A distillation temperature T is introduced, fully utilizing the characteristics of the Softmax function to soften the hard labels. This reduces the gradient differences between training instances, thus providing more information to the student model. Knowledge distillation features are calculated using the following formula.

[0077]

[0078] Where z i It is the i-th hard tag, q i This is the i-th soft label, obtained by softmax normalization of the hard labels, but the softmax here incorporates the distillation temperature T. KD The formula shows that the higher the temperature, the smoother the distribution of soft labels corresponding to hard labels, thus making the target distribution more uniform. Finally, during training, both the cross-entropy loss function and the Kullback-Leibler loss function are used to minimize the loss of hard and soft labels.

[0079] In summary, the teacher wake word recognition network of this invention employs a fully supervised pre-trained visual model and an unsupervised pre-trained acoustic model to extract visual and auditory features respectively. It also designs an Audio-Visual VoiceActivity Detection (AV-VAD) module and utilizes an attention mechanism to calculate the temporal score matrix. By comparing the audiovisual scores of each frame, it detects potential speakers. Finally, it trains the wake word recognition model based on the Knowledge Distillation (KD) method to meet the requirement of low computational complexity.

[0080] Based on the same concept, this invention also discloses a wake-word recognition method for multi-person scenarios based on audiovisual fusion, the device comprising:

[0081] The data acquisition module is used to acquire raw audio and video of each person's lip movements in multi-person scenarios.

[0082] A data recognition module is used to send the original audio and the lip movement video together into a wake-word recognition network to obtain a wake-word recognition result; wherein, the wake-word recognition network includes:

[0083] A video feature extraction network is used to extract video features from each frame of a lip movement video;

[0084] An audio feature extraction network is used to extract audio features from the original audio.

[0085] The fusion and recognition network is used to model the sum of all video features and the audio features in the time dimension through a multi-head attention mechanism, fuse the modeled video feature sequence and audio feature sequence, and obtain the wake word recognition result based on the fusion result.

[0086] The exemplary device is an embodiment of the device corresponding to the exemplary method described above. The specific operation of each module can be understood with reference to the description of the method embodiment, and will not be repeated here.

[0087] Based on the same concept, the present invention also discloses an electronic device. This electronic device may be a computer device, a laptop computer, a server, or other types of electronic equipment.

[0088] An electronic device may include at least one processor and a memory. The processor can execute instructions stored in the memory. The processor is communicatively connected to the memory via a data bus. In addition to the memory, the processor may also be communicatively connected to input devices, output devices, and communication devices via the data bus.

[0089] The processor can be any conventional processor. Processors can include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), system-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0090] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0091] In this embodiment of the present disclosure, an executable instruction is stored in the memory. The processor can read the executable instruction from the memory and execute the instruction to implement all or part of the steps of the vehicle maneuverability evaluation method in the exemplary embodiments described above.

[0092] Based on the same concept, the present invention also discloses a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0093] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0094] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0095] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for wake-word recognition in a multi-person scene based on audio-visual fusion, characterized in that, The method comprises: collecting original audio and lip movement video of each person in a multi-person scene; feeding the original audio and the lip movement video into a wake-up word recognition network to obtain a wake-up word recognition result; wherein the wake-up word recognition network comprises: a video feature extraction network configured to extract video features of each frame in the lip movement video; an audio feature extraction network configured to extract audio features of the original audio; a fusion and recognition network configured to model time dimensions of an addition result of all video features and the audio features through a multi-head attention mechanism, fuse the modeled video feature sequence and the audio feature sequence, and obtain the wake-up word recognition result based on a fusion result; wherein the fusion of the modeled video feature sequence and the audio feature sequence and the obtaining of the wake-up word recognition result based on the fusion result comprises: using a K-max Pool layer to extract a maximum feature value in the video feature sequence frame data; using another K-max Pool layer to extract the maximum feature value in the audio feature sequence frame data; the frame data and the feature value in the audio feature sequence with the largest the frame data and the feature value in the audio feature sequence with the largest frame data into a full connection layer to obtain a prediction probability of each wake-up word; obtaining the wake-up word recognition result based on the prediction probability.

2. The method of claim 1, wherein, The extraction of the video features of each frame in the lip movement video comprises: extracting spatio-temporal information of the lip movement video using a 3D convolution network; encoding the spatio-temporal information through a residual network feature encoder to obtain the video features of each frame.

3. The method of claim 1, wherein, The modeling of the time dimensions of the addition result of all video features through the multi-head attention mechanism comprises: summing the video features of each frame in all lip movement videos to obtain the addition result; feeding the addition result into a Transfomer encoder using a multi-head attention mechanism to obtain a video feature sequence.

4. The method of claim 1, wherein, The extraction of the audio feature sequence of the original audio comprises: feeding the original audio into a TC-ResNet model based on a CNN network to obtain an audio feature sequence.

5. The method of claim 1, wherein, The wake-up word recognition network is trained through the following steps: constructing a training data set; the training data set comprises audio samples and video samples of lip movement of each person in a multi-person or single-person scene, and real labels of corresponding wake-up words; taking a one-hot vector of the real label as a hard label of the wake-up word, and obtaining a soft label of the wake-up word by performing softmax normalization processing on the hard label; constructing a teacher wake-up word recognition network; the teacher wake-up word recognition network comprises: an MFCC feature extraction network configured to extract MFCC features of the audio samples; a video feature extraction network configured to extract sample video features of each frame in the video sample of the active speaker; an active speaker detection network configured to obtain the active speaker according to the MFCC features and the sample video features; an audio feature extraction network configured to extract sample audio features of the audio samples; a fusion and recognition network configured to model time dimensions of an addition result of all sample video features and the sample audio features through a multi-head attention mechanism, fuse the modeled sample video feature sequence and the sample audio feature sequence, and obtain a wake-up word recognition result based on a fusion result. The training result of the wake-up word recognition network is supervised based on a wake-up word recognition result of sample data obtained by the teacher wake-up word recognition network, and a hard label and a soft label are loss-minimized by using a cross-entropy loss function and a Kullback-leibler loss function.

6. The method of claim 5, wherein, The active speaker is obtained according to the MFCC feature and a video feature of the video sample, and the method comprises the following steps: passing the MFCC features and the sample video features through a linear layer to obtain new audio features and new video features ; using a broadcast mechanism of Python to the new audio features dimensional expansion, obtaining new audio features ; Regarding the new audio features With the new video features Perform element-wise dot products to obtain the time score matrix. ; Summing the score matrix S over the audio time dimension results in a score matrix S ; Based on the number of people in the scene, the scoring matrix is... Normalize and then find the index of the maximum value of the average normalization result across the visual dimension; The active speaker is obtained based on the index of each video sample in the same scene.

7. A wake-word recognition device for multi-person scenarios based on audiovisual fusion, characterized in that, The device comprises: a data acquisition module configured to acquire original audio and lip movement video of each person in a multi-person scene; a data recognition module configured to send the original audio and the lip movement video to a wake-up word recognition network together to obtain a wake-up word recognition result, wherein the wake-up word recognition network comprises: a video feature extraction network configured to extract a video feature of each frame in the lip movement video; an audio feature extraction network configured to extract an audio feature of the original audio; a fusion and recognition network configured to model the addition result of all video features and the audio feature in a time dimension by using a multi-head attention mechanism, to fuse a video feature sequence and an audio feature sequence after modeling, and to obtain the wake-up word recognition result based on a fusion result. The active speaker is obtained based on the index of each video sample in the same scene. using a K-max Pool layer to extract a maximum feature value in the video feature sequence frame data; using another K-max Pool layer to extract the maximum feature value in the audio feature sequence frame data; the frame data and the feature value in the audio feature sequence with the largest the frame data and the feature value in the audio feature sequence with the largest frame data into a full connection layer to obtain a prediction probability of each wake-up word; The wake-up word recognition result is obtained based on the prediction probability. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the wake-up word recognition method in a multi-person scene based on audio-visual fusion according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed, implement the wake-up word recognition method in a multi-person scene based on audio-visual fusion according to any one of claims 1 to 6.

Images