Audio-video multimodal keyword wake-up method and apparatus

Abstract

The present invention provides an audio-video multimodal keyword wake-up method and apparatus. [Solution] The method includes: acquiring an image sequence and an audio sequence; performing lip detection processing on the image sequence to extract detected lip images; processing the extracted lip image sequence via a lip feature extraction neural network to obtain image features; extracting audio features from the audio sequence; processing the extracted audio features via an audio feature processing neural network to obtain audio high-dimensional features; performing feature fusion of the image features and the audio high-dimensional features; processing the obtained fused audio-video features via a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; fusing the image features, audio high-dimensional features, and multi-modal high-dimensional features into one mixed high-dimensional feature; and determining whether wake-up is required based on the mixed high-dimensional feature using a keyword wake-up classifier.

Description

[Technical Field]

[0001] The present invention relates to the technical field of audio-video multimodal signal processing, and in particular to an audio-video multimodal keyword wake-up method and device. [Background technology]

[0002] Applications involving image and audio signal processing always require a keyword wake-up device, and keyword wake-up is an essential function in smart homes and the Internet of Things. However, in everyday use, the accuracy and effectiveness of keyword wake-up are not always as desired. In "Method, Device, and Storage Medium for Voice Wake-Up of Terminals" (Publication No. CN202211498799.2), the solution improves voice wake-up accuracy by using multiple voice terminals and a collaborative processing model that is compatible with the multiple voice terminals. However, it only takes into account the differences between the multiple voice terminals, does not consider the impact of environmental noise on voice wake-up accuracy, and does not combine image information to improve wake-up accuracy. In "Method, Device, and Computer-Readable Storage Medium for Multimodal Voice Wake-Up" (Publication No. CN202210098130.8), the solution improves voice wake-up accuracy by using the user's facial features. The drawback of this solution is that it only uses facial lip shape features, eye image features and audio information independently, without further mining and collaborative use of the above features, and without truly fusing various different features and data information.

[0003] For these reasons, there is currently no method that fully utilizes audio-video information to improve keyword wake-up accuracy. Summary of the Invention [Problem to be solved by the invention]

[0004] To address the problems of the prior art, this application provides an audio-video multimodal keyword wake-up method and apparatus, which improves the accuracy of keyword wake-up during man-machine interaction by processing and fusing multimodal feature information about audio and video via neural networks. [Means for solving the problem]

[0005] According to a first aspect of an embodiment of the present application, there is provided an audio-video multimodal keyword wake-up method, comprising: acquiring an image sequence and an audio sequence; performing a lip detection process on the image sequence, extracting detected lip images, and processing the extracted lip image sequence through a lip feature extraction neural network to obtain image features; extracting audio features from the audio sequence and processing the extracted audio features through an audio feature processing neural network to obtain audio high-dimensional features; performing feature fusion between the image features and the audio high-dimensional features; processing the fused audio-video features through a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; Fusing the image features, audio high-dimensional features and multi-modal high-dimensional features into one mixed high-dimensional feature; and determining whether it is necessary to wake up based on the mixed high-dimensional features using a keyword wake-up classifier.

[0006] Furthermore, the image sequence and the audio sequence are acquired by an audio-video interactive terminal device at a predetermined frame rate and audio sampling rate, respectively, and if the predetermined frame rate and audio sampling rate do not match the actual ones, they are adjusted to the predetermined frame rate and audio sampling rate by resampling.

[0007] Furthermore, before the extracted lip image sequence is processed through the lip feature extraction neural network, the lip image sequence is pre-processed, and the pre-processing operations include zooming and converting to a grayscale map.

[0008] Further, extracting audio features from the audio sequence includes: performing a short-time Fourier transform of the audio sequence and taking amplitude values ​​for the transformed frequency domain complex numbers to obtain a frequency domain amplitude feature sequence; extracting MFCC features for each frame of data in the audio sequence; and concatenating the frequency domain amplitude feature sequence and the extracted MFCC feature sequence to obtain audio features.

[0009] Furthermore, the multi-modal feature processing neural network is a neural network based on an attention mechanism.

[0010] Furthermore, feature fusion is performed by splicing data vector dimensions, adding and subtracting data vectors, or using fully connected neural networks.

[0011] Furthermore, the keyword wake-up classifier is a neural network or machine learning classifier, and the output result is 0 or 1, where 0 indicates that the user has not said the keyword and the user will not be woken up, and 1 indicates that the user has said the keyword and the user will be woken up.

[0012] According to a second aspect of an embodiment of the present application, there is provided an audio-video multimodal keyword wake-up device, comprising: an acquisition module for acquiring image sequences and audio sequences; an image feature extraction module for performing a lip detection process on the image sequence, extracting detected lip images, and processing the extracted lip image sequence via a lip feature extraction neural network to obtain image features; an audio feature extraction module for extracting audio features from the audio sequence and processing the extracted audio features via an audio feature processing neural network to obtain audio high-dimensional features; a first feature fusion module for performing feature fusion between the image features and the audio high-dimensional features; a multi-modal feature processing module for processing the fused audio-video features via a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; a second feature fusion module for fusing the image features, the audio high-dimensional features, and the multi-modal high-dimensional features into one mixed high-dimensional feature; and a wake-up determination module for determining whether it is necessary to wake up using a keyword wake-up classifier based on the mixed high-dimensional features.

[0013] According to a third aspect of the present invention, there is provided an electronic device, one or more processors; a memory for storing one or more programs; The one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of the first aspect.

[0014] According to a fourth aspect of an embodiment of the present application, there is provided a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, perform the steps of the method according to the first aspect. [Effects of the Invention]

[0015] The technical solutions according to the embodiments of the present application may include the following beneficial effects.

[0016] As can be seen from the above embodiments, the present application fuses lip shape image features and audio high-dimensional features, and uses a multi-modal feature processing module to mine audio-video high-dimensional features contributing to audio wake-up from the fused high-dimensional features; designs a multi-modal feature processing neural network for processing audio-video multimodal features; constructs a high-dimensional mixed feature using the lip shape image features, audio high-dimensional features, and audio-video high-dimensional features; processes the high-dimensional mixed feature using a classifier to determine whether to wake up and improve the wake-up accuracy; forms audio-video multi-modal features by fusing image features and audio features, and processes the audio-video multi-modal features through a neural network; fully mines high-dimensional mixed features and information contributing to audio-video keyword wake-up; fully utilizes the complementarity between audio and video keyword wake-up; and provides a reliable solution to keyword wake-up in noisy environments.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. [Brief explanation of the drawings]

[0018] The drawings herein are incorporated by reference into and made a part of this specification, illustrate embodiments consistent with the present application, and together with the specification, serve to explain the principles of the present application. [Figure 1] FIG. 1 is a flowchart of an audio-video multimodal keyword wake-up method according to an exemplary embodiment. [Figure 2] FIG. 2 is a structural schematic diagram of a model involved in an audio-video multimodal keyword wake-up method according to an exemplary embodiment. [Figure 3]FIG. 3 is a block diagram of an audio-video multimodal keyword wake-up device according to an exemplary embodiment. [Figure 4] FIG. 4 is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019]

[0013] Exemplary embodiments will now be described in detail, examples of which are illustrated in the drawings. When the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise noted. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.

[0020] The terms used herein are merely for the purpose of describing particular embodiments and are not intended to be limiting. As used in this application and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the term "and / or," as used herein, refers to any and all combinations including one or more of the associated listed items.

[0021] It should be understood that although various pieces of information may be described herein using terms such as "first," "second," "third," etc., these terms should not be limited to these terms. These terms are merely used to distinguish between the same types of information. For example, first information may be referred to as second information, and similarly, second information may be referred to as first information, without departing from the scope of the present application. Depending on the context, the term "○○ba" used herein may be interpreted as "when ○○," "in ○○'s case," or "in response to a decision."

[0022] The present application provides an audio-video multimodal keyword wake-up method. Refer to FIG. 1, which is a flowchart of one embodiment of the audio-video multimodal keyword wake-up method of the present application. As shown in FIG. 1, the method includes the following steps:

[0023] In step S11, an image sequence and an audio sequence are obtained; In one audio-video interactive terminal device, image sequence data and audio sequence data are acquired in real time by a camera and a microphone. In this embodiment, the image frame rate is 25 frames per second, and the audio sequence is acquired from the microphone. The audio sampling rate is 16 kHz, and every 1280 sampling points constitute one frame of audio. There are 640 overlapping points between frames. There are 25 frames of audio data per second. The collected audio-video data is represented as {t0:(frame0, audio0), t1:(frame1, audio1), ...}, where t n indicates the timestamp, and frame n is the timestamp t n indicates the image data corresponding to audio n is the timestamp t n If the frame rate or audio sampling rate of the terminal device is different from the above settings (i.e., the image frame rate is 25 frames per second and the audio sampling rate is 16 kHz), the image frame rate and audio data sampling rate are adjusted to the specified value by resampling.

[0024] In step S12, a lip detection process is performed on the image sequence to extract detected lip images, and the extracted lip image sequence is processed through a lip feature extraction neural network to obtain image features; Specifically, the image sequence is input to a lip image detection module to detect lip images, and the lip images are extracted to generate a lip image sequence Lips=[Lip1,...,Lip 25], and in this application, the Yolo series model can be used as the lip image detection module. The model needs to be trained before detection. The training dataset can be collected and annotated by the user, or a lip shape detection dataset such as the open source dataset LRW-1000 can be used. Since lips are not necessarily included in the images in the image sequence, in this application, the lip image sequence cannot be formed unless lips are included in at least 25 consecutive frames in the image sequence, and further operations are performed. The lip image sequence is extracted using a neural network, and the input of the network is Lips = [Lip1, ..., Lip 25 ] and the output is a set of features F lips =[F lip0 ,…,F lip25 ], and the input dimension of the network is 200x200. If the image size in Lips does not match the set 200x200, it is resampled to 200x200, and the output dimension of the network is 50x1. Therefore, the output feature set F lips =[F lip0 ,…,F lip25The dimensions of [subtraction] are 50x25. The model structure of the lip feature extraction neural network is shown in the "Lip Feature Extraction" section of Figure 2 and includes several 2D convolutional layers, pooling layers, and fully connected layers. In one embodiment, it is a sequentially connected 2D convolutional layer, max pooling layer, 2D convolutional layer, max pooling layer, 2D convolutional layer, max pooling layer, and 3-layer fully connected layer. The output of the last pooling layer is expanded into a 1D vector by a reshape operation and then input to the 3-layer fully connected layer. A 3- to 5-layer ResNet structure may also be used, and the Inception structure is used as the lip feature extraction neural network. In this application, lip images are first detected in an image sequence, and then feature extraction of the lip images is performed. This avoids further processing even when lips are not present in the image, and prevents other images other than the lips from interfering with the feature extraction. Since the purpose of this method is to perform keyword wake-up using audio-video information, lip and audio signals contain useful information necessary for this purpose. Extracting lip images improves the reliability and effectiveness of achieving this purpose.

[0025] Before the lip image sequence is subjected to feature extraction by the neural network, the lip image sequence is pre-processed, and the pre-processing methods include, but are not limited to, zooming and converting to a grayscale map.

[0026] In step S13, the audio sequence is subjected to audio feature extraction, and the extracted audio features are processed through an audio feature processing neural network to obtain audio high-dimensional features; Specifically, the audio sequence data is subjected to a short-time Fourier transform, with a frame shift of 640 points, a frame length of 1280 points, and an FFT point count of 1280. In this way, 25 frequency domain features can be obtained per second. After the transformation, the amplitude value is taken for the frequency domain complex number to obtain 25 frequency domain amplitude feature sequences, each with a length of 1280, where F1 = [F ft0 ,…,F ft25 ], except that Fft0 , …, F ft25 is a 1280 × 1-dimensional real vector. For each frame of the audio sequence data, 39-dimensional MFCC (Mel Frequency Cepstral Coefficient) features are extracted, that is, the MFCC static features plus first-order and second-order differential dynamic features are expressed as F2 = [F mfcc0 ,…,F mfcc25 ]. F1 and F2 are used as audio features. TIFF0007752760000001.tif1850 and splice it together. audio The dimensions of F1 are 1319x25. F1 represents the frequency domain of the audio and belongs to low-dimensional information, while F2 represents MFCC, which can be understood as a further feature extraction of the frequency domain signal, i.e., a higher-dimensional feature. By concatenating F1 and F2, the subsequent network can ensure that it not only receives low-dimensional frequency domain information, but also higher-dimensional features. This feature synthesis ensures that the network simultaneously receives both low-dimensional and high-dimensional features of the data. Furthermore, audio feature extraction methods such as Bark spectrum and Mel spectrum can be used.

[0027] The neural network extracts audio features, and the input of the network is TIFF0007752760000002.tif1849, and the output is the audio high-dimensional feature F audio_higher The network has an input dimension of 1319x25 and an output dimension of 50x25. The neural network model structure is shown in the "Extracting Audio High-Dimensional Features" section of Figure 2, and includes several 2D convolutional layers, pooling layers, and fully connected layers. In one embodiment, the neural network model is a sequentially connected 2D convolutional layer, max pooling layer, 2D convolutional layer, max pooling layer, and 3-layer fully connected layer. Other network structures, such as a 3-5 layer ResNet structure based on 1D convolutional layers, may also be used. The above operations reduce the data dimension, remove redundant information, and extract features that play an important role in the wake-up word.

[0028] In step S14, feature fusion is performed on the image features and the audio high-dimensional features; Specifically, the fusion methods include, but are not limited to, data dimensional splicing, data addition and subtraction, and the use of fully connected neural networks. lips and the audio high-dimensional features F audio_higher and one audio-video feature TIFF0007752760000003.tif1881 and F V-A The dimension of is 100×25.

[0029] In step S15, the fused audio-video features F V-A is processed via a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; In one embodiment, as shown in the "Multimodal Feature Processing" section of FIG. V-A is input to three convolutional layers, and the output after passing through the convolutional layers is V cnn The data dimension of is 20×5, then V cnn is input to the self-attention mechanism neural network, and the output of the self-attention mechanism neural network is V selfatten V selfatten The dimension of is 5x5, and V selfatten is input to the normalization layer, and the output of the normalization layer is V norm expands along the column direction to V norm-col and its dimension is 25×1. V norm-col is input to one fully connected layer, and the output dimension is 50 × 1. The multi-mode high-dimensional feature F V-A_higher In specific implementation, the multi-modal feature processing neural network may be a neural network based on different attention mechanisms, including but not limited to commonly used neural network structures such as self-attention mechanism and cross-attention mechanism neural network, and may be one or a combination thereof.

[0030] In step S16, the image features, the audio high-dimensional features and the multi-modal high-dimensional features are fused into one mixed high-dimensional feature; Specifically, the fusion methods include, but are not limited to, data vector dimension splicing, data vector addition and subtraction, and fusion using a fully connected neural network. lips (dimension 50×25), audio high-dimensional feature F audio_higher (dimensions 50 × 25) and high-dimensional audio-video feature information F V-A_higher (dimension 50×1) into one mixed high-dimensional feature F V-A_mixed The dimensions are 2550 × 1. In concrete implementation, there is a risk that some useful information of audio and image may be lost in the multi-modal high-dimensional features, so the audio high-dimensional features and image features are also combined to avoid the information loss.

[0031] In step S17, the mixed high-dimensional feature F V-A_mixed Based on this, a keyword wake-up classifier is used to determine whether it is necessary to wake up; Specifically, the classifier may be a different neural network or machine learning classifier, and the classification format may be a binary classifier of whether to wake up or not, or may output a single wake-up probability value. In one embodiment, the classifier model may selectively use logistic regression, and may also be a machine learning classifier such as an SVM (support vector machine). The input of the classifier is a mixture of high-dimensional features F V-A_mixed and the output result is 0 or 1, where 0 indicates that the user has not said the keyword and will not wake up the user, and 1 indicates that the user has said the keyword and will wake up the user.

[0032] In this embodiment, image features and audio features are obtained by detecting lip shape images and audio sequences, and high-dimensional audio features are further obtained based on the audio features using a neural network. The high-dimensional audio features and image features are combined and multi-modal feature processing is performed using multiple neural network structures to obtain high-dimensional information useful for audio-video wake-up. Finally, image features, high-dimensional audio features, and mixed high-dimensional features are processed using a classifier. By fully utilizing audio and image information, features useful for multimodal keyword wake-up in audio-video multimodal processing are deeply mined, effectively improving the accuracy of audio-video keyword wake-up.

[0033] Corresponding to the above embodiment of the audio-video multimodal keyword wake-up method, the present application further provides an embodiment of an audio-video multimodal keyword wake-up device.

[0034] 3 is a block diagram of an audio-video multimodal keyword wake-up device according to an exemplary embodiment. an acquisition module 21 for acquiring image and audio sequences; an image feature extraction module 22 for performing a lip detection process on the image sequence, extracting detected lip images, and processing the extracted lip image sequence via a lip feature extraction neural network to obtain image features; an audio feature extraction module 23 for extracting audio features from the audio sequence and processing the extracted audio features via an audio feature processing neural network to obtain audio high-dimensional features; a first feature fusion module 24 for performing feature fusion between the image features and the audio high-dimensional features; a multi-modal feature processing module 25 for processing the fused audio-video features via a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; a second feature fusion module 26 for fusing the image features, audio high-dimensional features and multi-modal high-dimensional features into one mixed high-dimensional feature; The device may further include a wake-up determination module 27 for determining whether it is necessary to wake up using a keyword wake-up classifier based on the mixed high-dimensional features.

[0035] Regarding the apparatus of the above embodiment, the specific manner in which each module performs the operation has already been described in detail in the embodiment of the method, so a detailed description thereof will be omitted here.

[0036] The device embodiments basically correspond to the method embodiments, so please refer to the relevant parts of the description of the method embodiments. The device embodiments described above are merely schematic, and the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., they may be located in one place or may be arranged in multiple network units. Depending on actual needs, some or all of the modules may be selected to achieve the purpose of the solution of the present application. Those skilled in the art can understand and implement them without any creative effort.

[0037]

[0013] Accordingly, the present application further provides an electronic device, comprising one or more processors and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute the audio-video multimodal keyword wake-up method. Figure 4 is a hardware structural diagram of any device with data processing capabilities in which a deep learning dataset access system according to an embodiment of the present invention is located. In addition to the processor, internal memory, and network interface shown in Figure 4, any device with data processing capabilities in which the device of the embodiment is located may generally include other hardware depending on the actual function of the device with data processing capabilities, and detailed descriptions thereof will be omitted here.

[0038] Correspondingly, the present application further provides a computer-readable storage medium on which computer instructions are stored, which, when executed by a processor, perform the above-mentioned audio-video multimodal keyword wake-up method. The computer-readable storage medium may be an internal storage unit, such as a hard disk or internal memory, of any device having data processing capabilities described in any one of the above embodiments. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, or a flash memory card, disposed in the device. Furthermore, the computer-readable storage medium may include an external storage device in addition to the internal storage unit of any device having data processing capabilities. The computer-readable storage medium is intended to store the computer program and other programs and data required by the device having data processing capabilities, and may also be intended to temporarily store data that has already been output or is to be output.

[0039] Other embodiments of the present application will be readily apparent to those skilled in the art after consideration of the specification and practice of the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include techniques well known or customary in the art, but which are not disclosed herein.

[0040] It is to be understood that the present application is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof.

Claims

1. An audio-video multimodal keyword wake-up method, the keyword wake-up method comprising: waking up a device using an audio-video multimodal keyword; The keyword wake-up method includes: acquiring an image sequence and an audio sequence; performing a lip detection process on the image sequence, extracting detected lip images, and processing the extracted lip image sequence through a lip feature extraction neural network to obtain image features; extracting audio features from the audio sequence and processing the extracted audio features through an audio feature processing neural network to obtain audio high-dimensional features; performing feature fusion between the image features and the audio high-dimensional features; processing the fused audio-video features through a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; Fusing the image features, audio high-dimensional features and multi-modal high-dimensional features into one mixed high-dimensional feature; and determining whether it is necessary to wake up based on the mixed high-dimensional features by a keyword wake-up classifier. An audio-video multimodal keyword wake-up method comprising:

2. The audio-video interactive terminal device acquires the image sequence and the audio sequence at a predetermined frame rate and audio sampling rate, respectively, and if the predetermined frame rate and audio sampling rate do not match the preset values, adjusts the frame rate and audio sampling rate to the predetermined values ​​by resampling. The audio-video multimodal keyword wake-up method according to claim 1.

3. and performing pre-processing on the extracted lip image sequence before processing the lip image sequence through the lip feature extraction neural network, the pre-processing operations including zooming and converting to a grayscale map. The audio-video multimodal keyword wake-up method according to claim 1.

4. extracting audio features from the audio sequence includes: performing a short-time Fourier transform of the audio sequence and taking amplitude values ​​for the transformed frequency domain complex numbers to obtain a frequency domain amplitude feature sequence; extracting MFCC features for each frame of data in the audio sequence; and concatenating the frequency domain amplitude feature sequence and the extracted MFCC feature sequence to obtain audio features. The audio-video multimodal keyword wake-up method according to claim 1.

5. The multi-modal feature processing neural network is a neural network based on an attention mechanism. The audio-video multimodal keyword wake-up method according to claim 1.

6. Perform the feature fusion by stitching together data vector dimensions, adding and subtracting data vectors, or using fully connected neural networks. The audio-video multimodal keyword wake-up method according to claim 1.

7. The keyword wake-up classifier is a neural network or machine learning classifier, and outputs a result of 0 or 1, where 0 indicates that the user has not uttered a keyword and the device should not be woken up, and 1 indicates that the user has uttered a keyword and the device should be woken up. The audio-video multimodal keyword wake-up method according to claim 1.

8. An audio-video multimodal keyword wake-up device, the keyword wake-up device being a device for waking up a device using an audio-video multimodal keyword; The keyword wake-up device includes: an acquisition module, an image feature extraction module, an audio feature extraction module, a first feature fusion module, a multi-modal feature processing module, a second feature fusion module, and a wake-up determination module; the acquisition module is used to acquire an image sequence and an audio sequence; the image feature extraction module is used for performing a lip detection process on the image sequence, extracting detected lip images, and processing the extracted lip image sequence through a lip feature extraction neural network to obtain image features; the audio feature extraction module is used for extracting audio features from the audio sequence, and processing the extracted audio features through an audio feature processing neural network to obtain high-dimensional audio features; The first feature fusion module is used for performing feature fusion between the image features and the audio high-dimensional features; the multi-modal feature processing module is used for processing the fused and acquired audio-video features through a multi-modal feature processing neural network to obtain multi-modal high-dimensional features; the second feature fusion module is used for fusing the image features, the audio high-dimensional features and the multi-modal high-dimensional features into one mixed high-dimensional feature; The wake-up judgment module is used to judge whether it is necessary to wake up based on the mixed high-dimensional features by using a keyword wake-up classifier. An audio-video multimodal keyword wake-up device characterized by:

9. An electronic device, one or more processors; a memory for storing one or more programs; The one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 7. An electronic device characterized by:

10. A computer-readable storage medium having computer instructions stored thereon, The computer instructions, when executed by a processor, perform the steps of the method of any one of claims 1 to 7. A computer-readable storage medium comprising:

Images