Multi-mode speech enhancement system based on audio and video

Abstract

The invention discloses a multi-mode speech enhancement system based on audio and video, and the system comprises a data collection module which collects target audio, background noise and interference sound through a microphone; a camera is used for synchronously collecting environment visual information, including facial expressions and mouth shapes of speakers, and noise and interference sounds are normalized to ensure that the noise and the interference sounds are matched with the dynamic range of a target audio; the data analysis module is used for preprocessing the collected audio, including noise suppression and speech enhancement, and improving the speech definition by applying a deep learning model; through the noise reduction and voice enhancement technology, the interference of background noise can be effectively reduced, and the voice is clearer. Thus, audiences can more easily understand speech contents, especially in noisy environments, such as public transport means or busy streets; and the voice recognition accuracy is improved, and the quality of the voice signal can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, computer vision and speech enhancement, and in particular relates to a multimodal speech enhancement system based on audio and video. Background Art

[0002] Speech separation technology, also known as sound source separation technology or beamforming technology, is mainly based on acoustic signal processing, and its purpose is to extract specific speech signals from complex audio signals. In a digital learning environment, online learning platforms provide language learners with unprecedented flexibility and convenience. However, with the increase in the number of people participating in online classes and learning activities, how to effectively separate the speech of a single learner from a noisy audio environment to ensure accurate assessment of language proficiency and improve teaching effectiveness has become an urgent challenge. For example, in an online learning environment, participants may come from all over the world and face different languages ​​and noise backgrounds, which increases the difficulty of speech separation; at the same time, the position change and movement of the speaker may cause the propagation path of the sound to change, which increases the difficulty of implementing the separation technology; and the speech separation system needs to be able to maintain good performance in different environments and have a certain adaptability to unknown speakers and noise sources. The application of speech separation technology has demonstrated its significant practicality in many fields. In an online language learning environment, this technology can significantly improve the accuracy of scoring and ensure the fairness of the evaluation results by clearly separating the speech of each learner. In addition, it can also significantly improve the quality of teaching interaction, allowing teachers to hear students' questions and answers more clearly, thereby improving classroom communication. In noisy public places, such as airports and transportation, speech separation technology can improve the clarity and effectiveness of remote meetings and provide participants with a smoother communication experience. At the same time, speech separation technology has also had a positive impact on the design and application of smart hearing aids, providing higher quality hearing support for the hearing-impaired by filtering background noise.

[0003] At present, the main focus of speech enhancement technology is to improve the quality and intelligibility of speech. Speech quality is usually the result of subjective perception and is greatly influenced by the listener's personal expectations, so there are often large differences in evaluation. Although subjective evaluation is crucial to the practical application of the system, it is time-consuming and costly, so it is often assisted by objective indicators. Mainstream objective indicators include Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), and Signal-to-Noise Ratio (SDR) or Scale-Invariant Signal-to-Distortion Ratio (SI-SDR). The higher the values ​​of these indicators, the better the speech separation effect.

[0004] Traditional methods of speech enhancement technology usually improve speech quality by removing background noise and retaining speech signals. Existing patents related to speech enhancement systems and methods include "A speech enhancement method, device and storage medium" with application number CN202011501035.5 and "Speech enhancement method, model training method, device and related equipment" with application number CN202410801433.0. Both methods predict the enhanced speech results based on the audio energy spectrum. However, these methods have certain difficulties in dealing with the separation of voices of different speakers.

[0005] In order to improve this problem, researchers in recent years have begun to use various external clues for audio separation. Application No. CN202410268765.7 "A joint speech enhancement method and its model building method" separates the speech of a specific identity by processing audio features and visual features. This method uses the joint processing of audio and visual information to improve the accuracy of speech separation. Although this patent performs well in sound source separation, it usually relies on high-quality visual information and precise synchronization streams, which may be limited in practical applications. On the basis of audio processing, our method further enhances the adaptability to complex noise environments and dynamic sound source configurations, pays more attention to various interferences and limitations faced in actual application scenarios, and provides stronger robustness. Application No. CN202110441281.4 "A method and device for audio and video speech enhancement processing" combines dynamic clues of visual features, such as the lip movements of the target speaker, to separate the voice of the target speaker from multiple speech signals. Its focus is on improving the separation effect of speech signals through visual information. This patent mainly relies on visual features (such as lip movements) and static / dynamic clues for speech separation, and may have certain limitations in its adaptability to the environment and processing range. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal speech enhancement system based on audio and video, which analyzes the facial movements and sound information of the target speaker when speaking through computer vision technology, separates the target speaker's speech from other interfering sounds and environmental noise, develops a simple and easy-to-use speech enhancement system, improves the clarity and intelligibility of the target speaker's speech, and provides users with more efficient tools to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a multimodal speech enhancement system based on audio and video, the enhancement system comprises

[0008] The data acquisition module collects the target audio, background noise and interference sound through the microphone; uses the camera to synchronously collect environmental visual information, including the speaker's facial expressions and lip shape, and normalizes the noise and interference sound to ensure that it matches the dynamic range of the target audio;

[0009] The data analysis module pre-processes the collected audio, including noise suppression and speech enhancement, and applies deep learning models to improve speech clarity; analyzes the visual information captured by the camera and assists in audio processing to improve the accuracy of speech recognition and enhancement; extracts audio features, performs data fusion processing, comprehensively utilizes audio and visual information to optimize speech enhancement effects, and evaluates and optimizes processing results;

[0010] Interactive interface for browsing and comparing the collected original audio and video and the audio and video after voice enhancement; supports exporting the processed audio and video data into standard format files;

[0011] The data acquisition module includes a data acquisition part and a data preprocessing part;

[0012] The data analysis module includes a speech enhancement model, see Figure 1 The main function is to separate the target speech signal corresponding to other input modes (video, facial appearance information) from a given noisy speech signal, and filter out the other parts of the signal (other speakers or background noise) to capture the temporal dynamic characteristics of the speaker's facial movements and sound sequences.

[0013] As a preferred technical solution of the present invention, the data preprocessing part processes the video part as follows:

[0014] S1, segment the long video, remove the first and last frames, and avoid the influence of interference factors;

[0015] S2, using 3D face model technology to rotate, translate and scale the image frames and markers to ensure that the facial markers in different image frames are aligned to similar positions;

[0016] S3, normalize the image frame and convert the pixel value from the range of [0,1] to the range of [-1,1];

[0017] S4, through the center cropping operation, the head is located in the center of the image, and the image size is adjusted to 224×224 pixels, and then a square occluder is added to the lips in some video frames, and the ratio of clear frames to occluded frames is maintained at 3:1;

[0018] As a preferred technical solution of the present invention, the data preprocessing part processes the audio part as follows:

[0019] The sampled target audio is mixed with the ambient noise audio signal in AudioSet and the interference audio signal containing a speaker in a ratio of 20% noise, 30% interference audio, and 50% target audio. At the same time, it is normalized according to the maximum amplitude of the mixed audio, and the mixed audio signal is saved as a wav file as a mixed audio containing noise.

[0020] As a preferred technical solution in the present invention, the speech enhancement model is divided into a spatiotemporal feature extraction module and a feature fusion module, wherein

[0021] The spatiotemporal feature module uses convolution blocks to extract feature information of audio, video and facial appearance as heterogeneous modal information, while the feature fusion module uses the Transformer model to fuse the heterogeneous modal information and uses multimodal information to effectively separate the target speaker's voice from other noises.

[0022] As a preferred technical solution in the present invention, after the speech enhancement module processes the audio, the encoder part of the U-Net composed of a one-dimensional convolutional neural network (1-Dimensional Convolutional Neural Network, 1D CNN), the encoder and decoder networks each have L layers, and the layers of the decoder are numbered in reverse order, the encoder network takes the original waveform as input and outputs a potential representation; each layer of the network includes a linear rectified linear unit activation function layer (Rectified Linear Unit activation layer, ReLU), a convolutional layer and a gated linear unit activation layer (Gated Linear Unit activation layer, GLU).

[0023] As a preferred technical solution in the present invention, after processing the video frame, the speech enhancement module uses a feature extraction network composed of a three-dimensional convolutional layer and ResNet-18 to extract the spatiotemporal features of the speaker's facial movement, and then randomly extracts a single face image from the face trajectory as input by using the ResNet-18 network, extracts the face embedding of the corresponding dimension to encode the speaker's facial attributes, and then replicates it along the time dimension to obtain facial attribute features;

[0024] The invention connects the three extracted audio and video features in series along the time dimension through the Transformer bottleneck, fuses the three inputs together, and uses the temporal Transformer encoder to extract temporal features. The temporal Transformer uses the self-attention mechanism to capture the long-distance dependencies in the temporal data, and improves the representation ability of the model through the multi-head attention mechanism. After the fused audio and video features are processed by the spatial feature extraction module, spatial features can be obtained. Then, these spatial features are combined with the position encoding and input into the temporal Transformer encoder. In the encoder, each spatial feature and position encoding is converted into a query, key and value vector, and multi-head attention calculation is performed. The calculated weighted value is used to generate intermediate features, and after layer normalization and residual connection, the output of a single Transformer encoder block is obtained. The outputs of multiple encoders are averaged to obtain the final space-time representation. Through this process, the model can effectively capture the temporal dynamic features of the speaker's facial movements and sound sequences, and the output is the corresponding enhanced audio representation Y. Afterwards, the enhanced audio representation Y is decoded into a speech-enhanced waveform corresponding to the video input by a decoder of a U-Net consisting of multiple layers of one-dimensional transposed convolutions including an audio encoder.

[0025] As a preferred technical solution in the present invention, the sampling rate of the microphone in the data acquisition module is 44.1kHz, and the camera has a resolution of 1080p and 30fps, and the file is saved in .npy format for subsequent analysis.

[0026] As a preferred technical solution in the present invention, the types of ambient noise audio signals include music, laughter, crying, engine sound, and wind sound.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. Improve speech clarity:

[0029] The invention uses noise reduction and speech enhancement technology to effectively reduce the interference of background noise and make speech clearer, so that listeners can understand the speech more easily, especially in noisy environments such as public transportation or busy streets.

[0030] 2. Improve speech recognition accuracy:

[0031] The invention can improve the quality of speech signals, thereby improving the accuracy of speech recognition systems. Clarifying speech can reduce recognition errors and improve the performance of voice assistants, automatic transcription and voice control systems.

[0032] 3. Enhanced communication experience:

[0033] In voice calls and video conferences, the invention can reduce echoes, noise and other interference, providing a more natural and smooth communication experience. This improvement makes remote collaboration and communication more efficient, which is especially important for remote work and international meetings.

[0034] 4. Assisting people with hearing impairment:

[0035] For people with hearing loss, the invention can help them hear speech information more clearly by improving the contrast between speech signals and background noise. This technology is particularly important in hearing aids and hearing rehabilitation equipment, which can significantly improve their daily quality of life. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A speech enhancement model diagram based on audio and video of the present invention;

[0037] Figure 2 It is a system diagram of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 and Figure 2 The present invention provides a technical solution: a multimodal speech enhancement system based on audio and video, the enhancement system comprises

[0040] The data acquisition module collects the target audio, background noise and interference sound through the microphone; uses the camera to synchronously collect environmental visual information, including the speaker's facial expressions and lip shape, and normalizes the noise and interference sound to ensure that it matches the dynamic range of the target audio;

[0041] The data analysis module pre-processes the collected audio, including noise suppression and speech enhancement, and applies deep learning models to improve speech clarity; analyzes the visual information captured by the camera and assists in audio processing to improve the accuracy of speech recognition and enhancement; extracts audio features, performs data fusion processing, comprehensively utilizes audio and visual information to optimize speech enhancement effects, and evaluates and optimizes processing results;

[0042] Interactive interface for browsing and comparing the collected original audio and video and the audio and video after voice enhancement; supports exporting the processed audio and video data into standard format files;

[0043] The data acquisition module includes a data acquisition part and a data preprocessing part;

[0044] The data analysis module includes a speech enhancement model, see Figure 1 The main function is to separate the target speech signal corresponding to other input modes (video, facial appearance information) from a given noisy speech signal, and filter out the other parts of the signal (other speakers or background noise) to capture the temporal dynamic characteristics of the speaker's facial movements and sound sequences.

[0045] In this embodiment, the data preprocessing part processes the video part as follows:

[0046] S1, segment the long video, remove the first and last frames, and avoid the influence of interference factors;

[0047] S2, using 3D face model technology to rotate, translate and scale the image frames and markers to ensure that the facial markers in different image frames are aligned to similar positions;

[0048] S3, normalize the image frame and convert the pixel value from the range of [0,1] to the range of [-1,1];

[0049] S4, through the center cropping operation, the head is located in the center of the image, and the image size is adjusted to 224×224 pixels, and then a square occluder is added to the lips in some video frames, and the ratio of clear frames to occluded frames is maintained at 3:1;

[0050] In this embodiment, the data preprocessing part processes the audio part as follows:

[0051] The sampled target audio is mixed with the ambient noise audio signal in AudioSet and the interference audio signal containing a speaker in a ratio of 20% noise, 30% interference audio, and 50% target audio. At the same time, it is normalized according to the maximum amplitude of the mixed audio, and the mixed audio signal is saved as a wav file as a mixed audio containing noise. The types of ambient noise audio signals include music, laughter, crying, engine sound, and wind sound.

[0052] In this embodiment, the speech enhancement model is divided into a spatiotemporal feature extraction module and a feature fusion module, wherein

[0053] The spatiotemporal feature module uses convolution blocks to extract feature information of audio, video and facial appearance as heterogeneous modal information, while the feature fusion module uses the Transformer model to fuse the heterogeneous modal information and uses multimodal information to effectively separate the target speaker's voice from other noises.

[0054] In this embodiment, after the speech enhancement module processes the audio, the encoder part of the U-Net composed of a one-dimensional convolutional neural network (1D CNN) has L layers in the encoder and decoder networks, and the layers of the decoder are numbered in reverse order. The encoder network takes the original waveform as input and outputs a potential representation; each layer of the network includes a rectified linear unit activation layer (Rectified Linear Unit activation layer, ReLU), a convolution layer and a gated linear unit activation layer (Gated Linear Unit activation layer, GLU); after the speech enhancement module processes the video frame, a feature extraction network composed of a three-dimensional convolution layer and ResNet-18 is used to extract the spatiotemporal features of the speaker's facial movement, and then a single face image is randomly extracted from the face trajectory as input by using the ResNet-18 network, and the facial attributes of the speaker are encoded by extracting the face embedding of the corresponding dimension, and then the facial attribute features are obtained by replicating along the time dimension;

[0055] The invention connects the three extracted audio and video features in series along the time dimension through the Transformer bottleneck, fuses the three inputs together, and uses the temporal Transformer encoder to extract temporal features. The temporal Transformer uses the self-attention mechanism to capture the long-distance dependencies in the temporal data, and improves the representation ability of the model through the multi-head attention mechanism. After the fused audio and video features are processed by the spatial feature extraction module, spatial features can be obtained. Then, these spatial features are combined with the position encoding and input into the temporal Transformer encoder. In the encoder, each spatial feature and position encoding is converted into a query, key and value vector, and multi-head attention calculation is performed. The calculated weighted value is used to generate intermediate features, and after layer normalization and residual connection, the output of a single Transformer encoder block is obtained. The outputs of multiple encoders are averaged to obtain the final space-time representation. Through this process, the model can effectively capture the temporal dynamic features of the speaker's facial movements and sound sequences, and the output is the corresponding enhanced audio representation Y. Afterwards, the enhanced audio representation Y is decoded into a speech-enhanced waveform corresponding to the video input by a decoder of a U-Net consisting of multiple layers of one-dimensional transposed convolutions including an audio encoder.

[0056] In this embodiment, the sampling rate of the microphone in the data acquisition module is 44.1kHz, while the camera has a resolution of 1080p and 30fps, and the file is saved in .npy format for subsequent analysis.

[0057] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal speech enhancement system based on audio and video, characterized in that: The enhancement system includes The data acquisition module collects the target audio as well as background noise and interference sound through a microphone; Use the camera to synchronously collect environmental visual information, including the speaker's facial expressions and lip shapes, and normalize noise and interference sounds to ensure that they match the dynamic range of the target audio. The data analysis module pre-processes the collected audio, including noise suppression and speech enhancement, and applies deep learning models to improve speech clarity; Analyze visual information captured by the camera and assist audio processing to improve the accuracy of speech recognition and enhancement; Extract audio features, perform data fusion processing, comprehensively utilize audio and visual information to optimize speech enhancement effects, and evaluate and optimize processing results; An interactive interface for browsing and comparing the collected original audio and video and the audio and video after voice enhancement; The data acquisition module includes a data acquisition part and a data preprocessing part; The data analysis module includes a speech enhancement model for capturing the temporal dynamic features of the speaker's facial movements and sound sequences.

2. The multimodal speech enhancement system based on audio and video according to claim 1, characterized in that: The processing process of the video part by the data preprocessing part is as follows: S1, segment the long video, remove the first and last frames, and avoid the influence of interference factors; S2, using 3D face model technology to rotate, translate and scale the image frames and markers to ensure that the facial markers in different image frames are aligned to similar positions; S3, normalize the image frame and convert the pixel value from the range of [0,1] to the range of [-1,1]; S4. Through the center cropping operation, the head is located in the center of the image, and the image size is adjusted to 224×224 pixels. Then, a square occluder is added to the lips in some video frames, and the ratio of clear frames to occluded frames is maintained at 3:

1.

3. The multimodal speech enhancement system based on audio and video according to claim 1, characterized in that: The processing process of the audio part by the data preprocessing part is as follows: The sampled target audio is mixed with the ambient noise audio signal in AudioSet and the interference audio signal containing a speaker in a ratio of 20% noise, 30% interference audio, and 50% target audio. At the same time, it is normalized according to the maximum amplitude of the mixed audio, and the mixed audio signal is saved as a wav file as a mixed audio containing noise.

4. The multimodal speech enhancement system based on audio and video according to claim 1, characterized in that: The speech enhancement model is divided into a spatiotemporal feature extraction module and a feature fusion module, wherein The spatiotemporal feature module uses convolution blocks to extract feature information of audio, video and facial appearance as heterogeneous modal information, while the feature fusion module uses the Transformer model to fuse the heterogeneous modal information and uses multimodal information to effectively separate the target speaker's voice from other noises.

5. The multimodal speech enhancement system based on audio and video according to claim 4, characterized in that: After the speech enhancement module processes the audio, the encoder part of the U-Net composed of a one-dimensional convolutional neural network, the encoder and decoder networks each have L layers, and the layers of the decoder are numbered in reverse order. The encoder network takes the original waveform as input and outputs a potential representation; each layer of the network contains a linear rectification activation function layer, a convolution layer, and a gated linear unit activation layer.

6. The multimodal speech enhancement system based on audio and video according to claim 4, characterized in that: After processing the video frames, the speech enhancement module uses a feature extraction network composed of a three-dimensional convolutional layer and ResNet-18 to extract the spatiotemporal features of the speaker's facial motion, and then uses the ResNet-18 network to randomly extract a single face image from the face trajectory as input, extracts the face embedding of the corresponding dimension to encode the speaker's facial attributes, and then replicates it along the time dimension to obtain facial attribute features.

7. The multimodal speech enhancement system based on audio and video according to claim 1, characterized in that: The microphone sampling rate in the data acquisition module is 44.1kHz, while the camera has a 1080p resolution and 30fps. The files are saved in .npy format for subsequent analysis.

8. The multimodal speech enhancement system based on audio and video according to claim 3, characterized in that: The types of the ambient noise audio signals include music, laughter, crying, engine sound, and wind sound.

Images