Multimodal keyword spotting method and device based on audio and video

Abstract

A multimodal keyword spotting method and device based on audio and a video. The method comprises: acquiring an image sequence and an audio sequence; performing lip detection processing on the image sequence, extracting a detected lip part image, and using a lip feature extraction neural network to process an extracted lip image sequence to obtain an image feature; extracting an audio feature of the audio sequence, and using an audio feature processing neural network to process the extracted audio feature to obtain a high-dimensional audio feature; carrying out feature fusion on the image feature and the high-dimensional audio feature; using a multi-modal feature processing neural network to process an audio and video feature obtained by fusion, to obtain a multi-modal high-dimensional feature; fusing the image feature, the high-dimensional audio feature and the multi-modal high-dimensional feature into a mixed high-dimensional feature; and on the basis of the mixed high-dimensional feature, using a keyword spotting classifier to determine whether spotting needs to be performed.

Description

A method and device for keyword awakening in audio and video multimodality Technical Field

[0001] The present invention belongs to the technical field of audio and video multimodal signal processing, and in particular relates to a keyword wake-up method and device for audio and video multimodal. Background Art

[0002] In application scenarios involving image and audio signal processing, there's often a need to wake devices with keywords. In smart homes and the Internet of Things, keyword wakeup is often an essential feature. However, in daily use, keyword wakeup often suffers from unsatisfactory accuracy and performance. The solution in the "Voice Wakeup Method, Device, and Storage Medium for Terminals" (Publication No.: CN 202211498799.2) utilizes multiple voice terminals and a joint processing model adapted for multiple voice terminals to improve voice wakeup accuracy. This approach only considers the differences between multiple voice terminals, ignoring the impact of ambient noise on voice wakeup accuracy and failing to incorporate image information to improve wakeup accuracy. The solution in the "Multimodal Voice Wakeup Method, Device, and Computer-Readable Storage Medium" (Publication No.: CN 202210098130.8) utilizes the user's facial features to improve voice wakeup accuracy. A drawback of this solution is that it only utilizes facial features such as lip shape, eye image features, and voice information independently, failing to further explore and combine these features, nor effectively integrating these diverse features and data.

[0003] In summary, there is currently a lack of a method that fully utilizes audio and video information to enhance keyword awakening. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present application provides a keyword awakening method and device for audio and video multimodality, which improves the accuracy of keyword awakening during human-computer interaction by utilizing a neural network to process the fused audio and video multimodal feature information.

[0005] According to a first aspect of an embodiment of the present application, a keyword wake-up method for audio and video multimodality is provided, comprising:

[0006] Obtain image sequences and audio sequences;

[0007] Performing lip detection on the image sequence and extracting the detected lip portion image, processing the extracted lip image sequence using a lip feature extraction neural network to obtain image features;

[0008] Extracting audio features from the audio sequence, and processing the extracted audio features using an audio feature processing neural network to obtain audio high-dimensional features;

[0009] Performing feature fusion on the image features and the audio high-dimensional features;

[0010] The fused audio and video features are processed using a multi-modal feature processing neural network to obtain multi-modal high-dimensional features;

[0011] fusing the image features, audio high-dimensional features, and multimodal high-dimensional features into a hybrid high-dimensional feature;

[0012] Based on the mixed high-dimensional features, a keyword awakening classifier is used to determine whether awakening is required.

[0013] Furthermore, the image sequence and the audio sequence are respectively acquired at a predetermined frame rate and audio sampling rate using an audio and video interactive terminal device. If the predetermined frame rate and audio sampling rate are inconsistent with the actual ones, they are adjusted to the predetermined frame rate and audio sampling rate through resampling.

[0014] Furthermore, before the extracted lip image sequence is processed by the lip feature extraction neural network, the lip image sequence is preprocessed, and the preprocessing operations include scaling and converting to grayscale.

[0015] Furthermore, audio feature extraction is performed on the audio sequence, including:

[0016] Performing a short-time Fourier transform on the audio sequence, taking amplitude values ​​of the transformed frequency-domain complex numbers, and obtaining a frequency-domain amplitude feature sequence;

[0017] Extracting MFCC features from each frame of audio data;

[0018] The frequency domain amplitude feature sequence is concatenated with the extracted MFCC feature sequence to obtain audio features.

[0019] Furthermore, the multimodal feature processing neural network is a neural network based on an attention mechanism.

[0020] Furthermore, feature fusion is achieved by splicing data vector dimensions, adding and subtracting data vectors, or using a fully connected neural network.

[0021] Furthermore, the keyword awakening classifier is a neural network or machine learning classifier, and the output result is 0 or 1, 0 means that the user is not detected to have said the keyword and the user is not awakened; 1 means that the user is detected to have said the keyword and the user is awakened.

[0022] According to a second aspect of an embodiment of the present application, there is provided an audio and video multimodal keyword wake-up device, comprising:

[0023] An acquisition module, used to acquire image sequences and audio sequences;

[0024] An image feature extraction module is used to perform lip detection processing on the image sequence and extract the detected lip part image, and process the extracted lip image sequence using a lip feature extraction neural network to obtain image features;

[0025] An audio feature extraction module is used to extract audio features from the audio sequence and process the extracted audio features using an audio feature processing neural network to obtain audio high-dimensional features;

[0026] A first feature fusion module, configured to perform feature fusion on the image features and the audio high-dimensional features;

[0027] The multi-mode feature processing module is used to process the fused audio and video features using a multi-mode feature processing neural network to obtain multi-mode high-dimensional features;

[0028] A second feature fusion module is used to fuse the image features, audio high-dimensional features and multi-modal high-dimensional features into a hybrid high-dimensional feature;

[0029] The wake-up judgment module is used to judge whether wake-up is needed based on the mixed high-dimensional features and using a keyword wake-up classifier.

[0030] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0031] one or more processors;

[0032] a memory for storing one or more programs;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0034] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0035] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0036] It can be seen from the above embodiments that the present application integrates lip image features and audio high-dimensional features, and uses a multi-modal feature processing module to mine audio and video high-dimensional features that are conducive to audio wake-up from the integrated high-dimensional features; designs a multi-modal feature processing neural network for processing audio and video multi-modal features; uses lip image features, audio high-dimensional features and audio and video high-dimensional features to construct a high-dimensional mixed feature, uses a classifier to process the high-dimensional mixed feature, and judges whether to wake up, thereby improving the accuracy of wake-up; forms audio and video multi-modal features by fusing image features and audio features, and uses a neural network to process the audio and video multi-modal features, fully mining the high-dimensional mixed features and information in audio and video that are conducive to keyword wake-up, and fully utilizing the complementarity of audio and video in keyword wake-up, providing a reliable solution for keyword wake-up in a noisy environment.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] FIG1 is a flowchart showing a keyword wake-up method for audio and video multimodality according to an exemplary embodiment.

[0040] FIG2 is a schematic diagram of a model structure involved in a keyword wake-up method for audio and video multimodality according to an exemplary embodiment.

[0041] Fig. 3 is a block diagram showing an audio and video multimodal keyword wake-up device according to an exemplary embodiment.

[0042] Fig. 4 is a schematic diagram showing an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0044] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] The present application provides a keyword awakening method for audio and video multimodality. Please refer to FIG1 , which is a flow chart of an embodiment of the keyword awakening method for audio and video multimodality of the present application. As shown in FIG1 , the method includes the following steps:

[0047] Step S11: Acquire an image sequence and an audio sequence;

[0048] On an audio and video interactive terminal device, image sequence data and audio sequence data are acquired in real time through a camera and a microphone. In this embodiment, the image frame rate is 25 frames per second; at the same time, an audio sequence is acquired from the microphone. The audio sampling rate is 16kHz, and each 1280 sampling points is divided into one audio frame. There is an overlap of 640 points between frames, and 25 frames of audio data per second. The acquired audio and video data is recorded as ,in Indicates the timestamp, Indicates the corresponding timestamp image data, Indicates the corresponding timestamp If the frame rate or audio sampling rate of the terminal device is different from the above settings (i.e., the image frame rate is 25 frames per second and the audio sampling rate is 16kHz), it will be adjusted to the specified image frame rate and audio data sampling rate through resampling.

[0049] Step S12: performing lip detection processing on the image sequence and extracting the detected lip portion image, processing the extracted lip image sequence using a lip feature extraction neural network to obtain image features;

[0050] Specifically, the image sequence is input into the lip image detection module to detect the image of the lip part, and the image of the lip part is extracted and recorded as the lip image sequence In this application, the Yolo series model can be used as the lip image detection module. The model needs to be trained before detection. The training data set can be collected and annotated by yourself or use the open source data set LRW-1000 and other lip shape detection data sets. It should be noted that the images in the image sequence do not necessarily contain the lip part, so this application sets that at least 25 consecutive frames of images in the image sequence contain the lip part to form a lip image sequence, and then perform subsequent operations. Use a neural network to extract features from the lip image sequence, and the input of the network is , the output is a set of features , the input dimension of the network is ,if The image size does not meet the set , then it is adjusted to , the output dimension of the network is Therefore, the output feature set The dimension is . The model structure of the lip feature extraction neural network is shown in the "lip feature extraction" part of Figure 2, including several two-dimensional convolutional layers, pooling layers and fully connected layers. In one embodiment, it is specifically a two-dimensional convolutional layer, a maximum pooling layer, a two-dimensional convolutional layer, a maximum pooling layer, a two-dimensional convolutional layer, a maximum pooling layer, and three fully connected layers connected in sequence, wherein the output of the last pooling layer is expanded into a one-dimensional vector through a reshape operation and then input into the three fully connected layers. A 3 to 5-layer ResNet structure or an Inception structure can also be used as a lip feature extraction neural network. In this application, the lip part image is first detected from the image sequence, and then the feature extraction of the lip part image is performed to avoid subsequent processing when there are no lips in the image; and other images on the face except the lips are excluded from interfering with the feature extraction. Since the purpose of this method is to use audio and video information for keyword wake-up, it is obvious that the lips and audio signals contain the useful information required for this goal, so the reliability and effectiveness of achieving this goal can be improved by extracting the lip part image.

[0051] Before using a neural network to extract features from the lip image sequence, the lip image sequence is preprocessed. The preprocessing method includes but is not limited to: scaling and converting to grayscale.

[0052] Step S13: extracting audio features from the audio sequence, and processing the extracted audio features using an audio feature processing neural network to obtain audio high-dimensional features;

[0053] Specifically, the audio sequence data is subjected to short-time Fourier transform, with a frame shift of 640 points, a frame length of 1280 points, and an FFT point number of 1280. In this way, 25 frequency domain features can be obtained per second. The amplitude value of the converted frequency domain complex number is taken, and 25 frequency domain amplitude feature sequences are obtained. Each sequence length is 1280, which is recorded as ,in ,…, The dimension is A real number vector. For each frame of the audio sequence data, 39-dimensional MFCC (Mel Frequency Cepstral Coefficents) features are extracted, that is, MFCC static features plus first-order and second-order differential dynamic features, recorded as .Will and Splicing into audio features , The dimension is F1 represents the frequency domain of the audio, which is low-dimensional information; F2 represents MFCC, which can be understood as further feature extraction of the frequency domain signal, that is, higher-dimensional features. Splicing F1 and F2 together can ensure that the subsequent network not only receives low-dimensional frequency domain information, but also higher-dimensional features. This feature splicing ensures that the network receives both low-dimensional and high-dimensional features of the data. Audio feature extraction methods such as Bark spectrum and Mel spectrum can also be used.

[0054] Use neural network to extract audio features. The input of the network is , the output is the audio high-dimensional feature , the input dimension of the network is , the output dimension of the network is The model structure of the neural network is shown in the "Audio High-Dimensional Feature Extraction" section in Figure 2, which includes several two-dimensional convolutional layers, pooling layers, and fully connected layers. In one embodiment, it specifically includes a two-dimensional convolutional layer, a maximum pooling layer, a two-dimensional convolutional layer, a maximum pooling layer, and three fully connected layers connected in sequence. Other network structures can also be adopted, such as a 3- to 5-layer ResNet structure based on a 1-dimensional convolutional layer. The above operations reduce the data dimension, remove redundant information, and extract features that play a key role in the wake-up word.

[0055] Step S14: performing feature fusion on the image features and the audio high-dimensional features;

[0056] Specifically, the fusion method includes but is not limited to: data dimension splicing, data addition and subtraction operations, using a fully connected neural network, etc. In one embodiment, the image features are and audio high-dimensional features Splice into an audio and video feature , The dimension is .

[0057] Step S15: The fused audio and video features Use multi-modal feature processing neural network to process and obtain multi-modal high-dimensional features;

[0058] In one embodiment, as shown in the "Multi-mode Feature Processing" section in FIG2 , first Input to three convolutional layers, output after convolutional layers The data dimension is , then Input to the self-attention mechanism neural network, the output of the self-attention mechanism neural network is recorded as , The dimension is ,Will Input to the normalization layer, record the output of the normalization layer Expand along the column direction , its dimension is .Will Input to a fully connected layer, the output dimension is , recorded as multimodal high-dimensional features In a specific implementation, the multimodal feature processing neural network can be different attention-based neural networks, including but not limited to common neural network structures such as self-attention mechanism and mutual attention mechanism neural network, and can be one or a combination of several of them.

[0059] Step S16: fusing the image features, audio high-dimensional features, and multi-modal high-dimensional features into a hybrid high-dimensional feature;

[0060] Specifically, the fusion methods include but are not limited to data vector dimension splicing, data vector addition and subtraction, and fusion using a fully connected neural network. (Dimension ), audio high-dimensional features (Dimension ) and high-dimensional audio and video feature information (Dimension ) are spliced ​​into a mixed high-dimensional feature , whose dimensions are In the specific implementation, multi-modal high-dimensional features may lose some useful information of audio and image, so the audio high-dimensional features and image features are also spliced ​​in to avoid information loss.

[0061] Step S17: Based on the mixed high-dimensional features ,Use the keyword wakeup classifier to determine whether wakeup is needed;

[0062] Specifically, the classifier can be a different neural network or machine learning classifier; the classification form can be a binary classifier of whether to wake up or not, or it can output a wake-up probability value. In one embodiment, the classifier model uses logistic regression, or it can be a machine learning classifier such as SVM (support vector machine). The input of the classifier is a mixed high-dimensional feature The output result is 0 or 1. 0 means that the user is not detected to have said the keyword and the user is not awakened; 1 means that the user is detected to have said the keyword and the user is awakened.

[0063] In this embodiment, image and audio features are obtained by detecting lip images and audio sequences. Based on the audio features, neural networks are used to further obtain high-dimensional audio features. The high-dimensional audio features and image features are fused, and multimodal feature processing is performed using multiple neural network structures to obtain high-dimensional information useful for audio and video wake-up. Finally, a classifier is used to process the image features, high-dimensional audio features, and mixed high-dimensional features. This fully utilizes audio and image information, deeply explores features useful for multimodal keyword wake-up in audio and video multimodal processing, and effectively improves the accuracy of audio and video keyword wake-up.

[0064] Corresponding to the embodiment of the aforementioned audio and video multimodal keyword awakening method, the present application also provides an embodiment of an audio and video multimodal keyword awakening device.

[0065] FIG3 is a block diagram of a multi-modal audio and video keyword wake-up device according to an exemplary embodiment. Referring to FIG3 , the device may include:

[0066] An acquisition module 21 is used to acquire an image sequence and an audio sequence;

[0067] An image feature extraction module 22 is configured to perform lip detection on the image sequence and extract the detected lip portion image, and process the extracted lip image sequence using a lip feature extraction neural network to obtain image features;

[0068] An audio feature extraction module 23 is configured to extract audio features from the audio sequence and process the extracted audio features using an audio feature processing neural network to obtain audio high-dimensional features;

[0069] A first feature fusion module 24 is configured to perform feature fusion on the image features and the audio high-dimensional features;

[0070] The multi-mode feature processing module 25 is used to process the fused audio and video features using a multi-mode feature processing neural network to obtain multi-mode high-dimensional features;

[0071] A second feature fusion module 26 is used to fuse the image features, audio high-dimensional features and multi-modal high-dimensional features into a mixed high-dimensional feature;

[0072] The wake-up judgment module 27 is configured to judge whether wake-up is required based on the mixed high-dimensional features and using a keyword wake-up classifier.

[0073] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0074] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0075] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned audio and video multimodal keyword wake-up method. As shown in FIG4 , a hardware structure diagram of any device with data processing capability where a deep learning data set access system is provided in an embodiment of the present invention is provided. In addition to the processor, memory and network interface shown in FIG4 , any device with data processing capability where the apparatus in the embodiment is located may also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.

[0076] Correspondingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned audio and video multimodal keyword wake-up method. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0077] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0078] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A keyword awakening method for audio and video multimodality, characterized in that: include: Get image sequence and audio sequence; Performing lip detection processing on the image sequence and extracting the detected lip part image, processing the extracted lip image sequence using a lip feature extraction neural network to obtain image features; Extracting audio features from the audio sequence, and processing the extracted audio features using an audio feature processing neural network to obtain audio high-dimensional features; Performing feature fusion on the image features and the audio high-dimensional features; The fused audio and video features are processed using a multi-mode feature processing neural network to obtain multi-mode high-dimensional features; fusing the image features, audio high-dimensional features and multi-modal high-dimensional features into a mixed high-dimensional feature; Based on the mixed high-dimensional features, a keyword awakening classifier is used to determine whether awakening is needed.

2. The method according to claim 1, characterized in that The image sequence and the audio sequence are respectively acquired at a predetermined frame rate and audio sampling rate using an audio and video interactive terminal device. If the predetermined frame rate and audio sampling rate are inconsistent with the actual ones, they are adjusted to the predetermined frame rate and audio sampling rate through resampling.

3. The method according to claim 1, characterized in that Before the extracted lip image sequence is processed by using the lip feature extraction neural network, the lip image sequence is preprocessed, and the preprocessing operations include scaling and grayscale conversion.

4. The method according to claim 1, characterized in that: Extracting audio features from the audio sequence includes: Performing short-time Fourier transform on the audio sequence, taking amplitude values ​​of the transformed frequency domain complex numbers, and obtaining a frequency domain amplitude feature sequence; Extracting MFCC features from each frame of data in the audio sequence; The frequency domain amplitude feature sequence is concatenated with the extracted MFCC feature sequence to obtain audio features.

5. The method according to claim 1, characterized in that The multimodal feature processing neural network is a neural network based on an attention mechanism.

6. The method according to claim 1, characterized in that Feature fusion is achieved by concatenating data vector dimensions, adding and subtracting data vectors, or using a fully connected neural network.

7. The method according to claim 1, characterized in that The keyword wake-up classifier is a neural network or a machine learning classifier, and the output result is 0 or 1, 0 means that it is not detected that the user said the keyword and the user is not woken up; 1 means that it is detected that the user said the keyword and the user is woken up.

8. An audio and video multi-modal keyword awakening device, characterized in that: include: An acquisition module, used for acquiring image sequences and audio sequences; An image feature extraction module is used to perform lip detection processing on the image sequence and extract the detected lip part image, and process the extracted lip image sequence using a lip feature extraction neural network to obtain image features; An audio feature extraction module is used to extract audio features from the audio sequence, and process the extracted audio features using an audio feature processing neural network to obtain audio high-dimensional features; A first feature fusion module, used for fusing the image features and the audio high-dimensional features; A multi-mode feature processing module is used to process the fused audio and video features using a multi-mode feature processing neural network to obtain multi-mode high-dimensional features; A second feature fusion module is used to fuse the image feature, the audio high-dimensional feature and the multi-mode high-dimensional feature into a mixed high-dimensional feature; The wake-up judgment module is used to judge whether wake-up is needed based on the mixed high-dimensional features and using a keyword wake-up classifier.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images