An audio-video-based multi-modal speech enhancement system

Abstract

The application discloses a kind of multi-modal speech enhancement systems based on audio and video, and the enhancement system includes data acquisition module, and target audio and background noise and interference sound are acquired by microphone;Utilize camera to synchronously collect environmental visual information, including the facial expression and mouth shape of speaker, normalize the noise and interference sound, ensure that the dynamic range matches with target audio;Data analysis module, the audio collected is preprocessed, including noise suppression and speech enhancement, and the intelligibility of speech is improved by applying deep learning model;Improve speech intelligibility: the application can effectively reduce the interference of background noise by noise reduction and speech enhancement technology, so that speech is clearer. In this way, the audience can more easily understand the content of the speech, especially in noisy environments, such as public transportation or busy streets;Improve speech recognition accuracy: the application can improve the quality of speech signals.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning, computer vision and speech enhancement technology, and specifically relates to a multimodal speech enhancement system based on audio and video. Background Technology

[0002] Speech separation technology, also known as sound source separation technology or beamforming technology, is primarily based on acoustic signal processing. Its purpose is to extract specific speech signals from complex audio signals. In digital learning environments, online learning platforms offer language learners unprecedented flexibility and convenience. However, with the increasing number of participants in online classes and learning activities, effectively separating the speech of individual learners from noisy audio environments to ensure accurate assessment of language proficiency and improved teaching effectiveness has become a pressing challenge. For example, in online learning environments, participants may come from all over the world, facing different language and noise backgrounds, increasing the difficulty of speech separation. Simultaneously, changes in speaker location and movement can alter the sound propagation path, further complicating the implementation of separation technology. Furthermore, speech separation systems need to maintain good performance in different environments and be adaptable to unknown speakers and noise sources. Speech separation technology has demonstrated significant practicality in multiple fields. In online language learning environments, this technology can significantly improve the accuracy of scoring by clearly separating each learner's speech, ensuring the fairness of assessment results. Furthermore, it significantly improves the quality of teaching interaction, enabling teachers to hear students' questions and answers more clearly, thereby enhancing classroom communication. In noisy public places, such as airports and public transportation, speech separation technology can improve the clarity and effectiveness of remote meetings, providing participants with a smoother communication experience. At the same time, speech separation technology has also positively impacted the design and application of smart hearing aids, providing higher-quality hearing support for people with hearing impairments by filtering background noise.

[0003] Currently, the main focus of speech enhancement technology is improving speech quality and intelligibility. Speech quality is usually a subjective perception, greatly influenced by the listener's personal expectations, thus often resulting in significant differences in evaluation. Although subjective evaluation is crucial for the practical application of the system, it is time-consuming and costly, often requiring the assistance of objective metrics. Mainstream objective metrics include Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), and Signal-to-Noise Ratio (SDR) or Scale-Invariant Signal-to-Distortion Ratio (SI-SDR). Higher values ​​for these metrics generally indicate better speech separation performance.

[0004] Traditional methods of speech enhancement typically improve speech quality by removing background noise while preserving the speech signal. Existing published patents related to speech enhancement systems and methods include CN202011501035.5, "A Speech Enhancement Method, Apparatus, and Storage Medium," and CN202410801433.0, "Speech Enhancement Method, Model Training Method, Apparatus, and Related Equipment." Both methods predict the enhanced speech result based on audio energy spectra. However, these methods face certain difficulties in handling the separation of voices from different speakers.

[0005] To address this issue, researchers in recent years have begun utilizing various external cues for audio separation. Application CN202410268765.7, entitled "A Joint Speech Enhancement Method and Its Model Building Method," separates the speech of a specific identity by processing audio and visual features. This method utilizes the joint processing of audio and visual information to improve the accuracy of speech separation. While this patent performs well in sound source separation, it typically relies on high-quality visual information and precise synchronization streams, which may be limited in practical applications. Our method, based on audio processing, further enhances its adaptability to complex noisy environments and dynamic sound source configurations, paying more attention to various interferences and limitations faced in real-world application scenarios, providing stronger robustness. Application CN202110441281.4, entitled "An Audio-Visual Speech Enhancement Processing Method and Apparatus," combines dynamic cues of visual features, such as the target speaker's lip movements, to separate the target speaker's voice from multiple speech signals. Its focus is on improving the speech signal separation effect through visual information. This patent primarily relies on visual features (such as lip movements) and static / dynamic cues for speech separation, which may have certain limitations in terms of environmental adaptability and processing range. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal speech enhancement system based on audio and video. By analyzing the facial movements and vocal information of the target speaker during speech using computer vision technology, the system separates the target speaker's speech from other interfering sounds and environmental noise. This results in a simple and easy-to-use speech enhancement system that improves the clarity and intelligibility of the target speaker's speech, providing users with a more efficient tool to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multimodal speech enhancement system based on audio and video, the enhancement system comprising...

[0008] The data acquisition module collects target audio, background noise, and interference sounds through a microphone; it also uses a camera to simultaneously collect environmental visual information, including the speaker's facial expressions and lip movements, and normalizes noise and interference sounds to ensure that the dynamic range matches the target audio.

[0009] The data analysis module preprocesses the acquired audio, including noise suppression and speech enhancement, and applies deep learning models to improve speech clarity; it analyzes visual information captured by the camera to assist audio processing and improve the accuracy of speech recognition and enhancement; it extracts audio features, performs data fusion processing, comprehensively utilizes audio and visual information to optimize the speech enhancement effect, and evaluates and optimizes the processing results.

[0010] An interactive interface for browsing and comparing the initial audio and video data collected with the audio and video data after speech enhancement; supports exporting processed audio and video data to standard format files;

[0011] The data acquisition module includes a data acquisition section and a data preprocessing section;

[0012] The data analysis module includes a speech enhancement model, as detailed below. Figure 1 Its main function is to separate the target speech signal from a given noisy speech signal and extract the target speech signal that corresponds to other input modes (video, facial appearance information), and filter out other parts of the signal (other speakers or background noise) to capture the temporal dynamic features of the speaker's facial movements and sound sequence.

[0013] As a preferred technical solution of the present invention, the data preprocessing part processes the video part as follows:

[0014] S1. Segment the long video and remove the first and last frames to avoid the influence of interference factors;

[0015] S2. Use 3D face model technology to rotate, translate and scale the image frames and markers to ensure that the facial markers in different image frames are aligned to similar positions;

[0016] S3. Standardize the image frames, converting pixel values ​​from the range [0,1] to the range [-1,1].

[0017] S4. By performing a center cropping operation, the head is placed in the center of the image, and the image size is adjusted to 224×224 pixels. Then, square occlusions are added to the lips in some video frames, and the ratio of clear frames to occluded frames is maintained at 3:1.

[0018] As a preferred technical solution of the present invention, the data preprocessing section processes the audio portion as follows:

[0019] The sampled target audio is mixed with the ambient noise audio signal and the interference audio signal containing a speaker in the AudioSet, with noise accounting for 20%, interference audio accounting for 30%, and target audio accounting for 50%. At the same time, it is normalized according to the maximum amplitude of the mixed audio. The mixed audio signal is saved as a wav file as mixed audio containing noise.

[0020] As a preferred technical solution of the present invention, the speech enhancement model is divided into a spatiotemporal feature extraction module and a feature fusion module, wherein...

[0021] The spatiotemporal feature module uses convolutional blocks to extract feature information from audio, video, and facial appearance as heterogeneous modal information, while the feature fusion module uses the Transformer model to fuse the heterogeneous modal information, effectively separating the target speaker's voice from other noise using multimodal information.

[0022] As a preferred technical solution of the present invention, the speech enhancement module processes the audio and then uses the encoder part of the U-Net composed of a 1-Dimensional Convolutional Neural Network (1D CNN). The encoder and decoder networks each have L layers, and the layers of the decoder are numbered in reverse order. The encoder network takes the original waveform as input and outputs a latent representation. Each layer of the network contains a rectified linear unit activation layer (ReLU), a convolutional layer, and a gated linear unit activation layer (GLU).

[0023] As a preferred technical solution of the present invention, the speech enhancement module processes the video frames and uses a feature extraction network composed of a three-dimensional convolutional layer and ResNet-18 to extract the spatiotemporal features of the speaker's facial motion. Then, by using the ResNet-18 network, a single face image is randomly extracted from the face trajectory as input. The speaker's facial attributes are encoded by extracting the face embedding of the corresponding dimension. Then, the facial attribute features are obtained by copying along the time dimension.

[0024] This invention concatenates three extracted audio and video features along the time dimension using a Transformer bottleneck, fusing the three inputs together, and employs a temporal Transformer encoder to extract temporal features. The temporal Transformer utilizes a self-attention mechanism to capture long-range dependencies in temporal data, while simultaneously improving the model's representational power through a multi-head attention mechanism. After processing by the spatial feature extraction module, the fused audio and video features yield spatial features. These spatial features are then combined with positional encodings and input into the temporal Transformer encoder. In the encoder, each spatial feature and positional encoding is converted into a query, key, and value vector, and multi-head attention is calculated. The calculated weighted values ​​are used to generate intermediate features, which, after layer normalization and residual connections, yield the output of a single Transformer encoder block. The outputs of multiple encoders are averaged to obtain the final spatial-temporal representation. Through this process, the model can effectively capture the temporal dynamic features of the speaker's facial movements and sound sequences, outputting a corresponding enhanced audio representation Y. Subsequently, the enhanced audio representation Y is decoded by a decoder of a U-Net consisting of multiple layers of one-dimensional transposed convolutions containing an audio encoder into a speech enhancement waveform corresponding to the video input.

[0025] As a preferred technical solution in this invention, the microphone sampling rate in the data acquisition module is 44.1kHz, while the camera has a resolution of 1080p and a frame rate of 30fps, and the file is saved in .npy format for subsequent analysis.

[0026] As a preferred technical solution of the present invention, the types of environmental noise audio signals include music, laughter, crying, engine sounds, and wind sounds.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] 1. Improve speech clarity:

[0029] This invention effectively reduces background noise interference and makes speech clearer through noise reduction and speech enhancement technologies. This makes it easier for listeners to understand what is being said, especially in noisy environments such as public transportation or busy streets.

[0030] 2. Improve speech recognition accuracy:

[0031] This invention can improve the quality of speech signals, thereby enhancing the accuracy of speech recognition systems. Clarifying speech reduces recognition errors and improves the performance of voice assistants, automatic transcription, and voice control systems.

[0032] 3. Enhance communication experience:

[0033] In voice calls and video conferencing, this invention reduces echo, background noise, and other interference, providing a more natural and fluid communication experience. This improvement makes remote collaboration and communication more efficient, especially important for remote work and international conferences.

[0034] 4. Individuals with hearing impairments:

[0035] For people with hearing impairments, this invention helps them hear speech information more clearly by increasing the contrast between the speech signal and background noise. This technology is particularly important in hearing aids and hearing rehabilitation devices, as it can significantly improve their quality of life. Attached Figure Description

[0036] Figure 1 This is a diagram of the audio / video-based speech enhancement model of the present invention;

[0037] Figure 2 This is a system diagram of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Please see Figure 1 and Figure 2 This invention provides a technical solution: a multimodal speech enhancement system based on audio and video, the enhancement system comprising...

[0040] The data acquisition module collects target audio, background noise, and interference sounds through a microphone; it also uses a camera to simultaneously collect environmental visual information, including the speaker's facial expressions and lip movements, and normalizes noise and interference sounds to ensure that the dynamic range matches the target audio.

[0041] The data analysis module preprocesses the acquired audio, including noise suppression and speech enhancement, and applies deep learning models to improve speech clarity; it analyzes visual information captured by the camera to assist audio processing and improve the accuracy of speech recognition and enhancement; it extracts audio features, performs data fusion processing, comprehensively utilizes audio and visual information to optimize the speech enhancement effect, and evaluates and optimizes the processing results.

[0042] An interactive interface for browsing and comparing the initial audio and video data collected with the audio and video data after speech enhancement; supports exporting processed audio and video data to standard format files;

[0043] The data acquisition module includes a data acquisition section and a data preprocessing section;

[0044] The data analysis module includes a speech enhancement model, see details below. Figure 1 Its main function is to separate the target speech signal from a given noisy speech signal and extract the target speech signal that corresponds to other input modes (video, facial appearance information), and filter out other parts of the signal (other speakers or background noise) to capture the temporal dynamic features of the speaker's facial movements and sound sequence.

[0045] In this embodiment, the data preprocessing part processes the video portion as follows:

[0046] S1. Segment the long video and remove the first and last frames to avoid the influence of interference factors;

[0047] S2. Use 3D face model technology to rotate, translate and scale the image frames and markers to ensure that the facial markers in different image frames are aligned to similar positions;

[0048] S3. Standardize the image frames, converting pixel values ​​from the range [0,1] to the range [-1,1].

[0049] S4. By performing a center cropping operation, the head is placed in the center of the image, and the image size is adjusted to 224×224 pixels. Then, square occlusions are added to the lips in some video frames, and the ratio of clear frames to occluded frames is maintained at 3:1.

[0050] In this embodiment, the data preprocessing section processes the audio portion as follows:

[0051] The sampled target audio is mixed with the ambient noise audio signal and the interference audio signal containing a speaker from the AudioSet, with noise accounting for 20%, interference audio accounting for 30%, and target audio accounting for 50%. At the same time, the mixed audio is normalized according to the maximum amplitude of the mixed audio. The mixed audio signal is saved as a wav file as a mixed audio containing noise. The types of ambient noise audio signals include music, laughter, crying, engine sound, and wind sound.

[0052] In this embodiment, the speech enhancement model is divided into a spatiotemporal feature extraction module and a feature fusion module, wherein...

[0053] The spatiotemporal feature module uses convolutional blocks to extract feature information from audio, video, and facial appearance as heterogeneous modal information, while the feature fusion module uses the Transformer model to fuse the heterogeneous modal information, effectively separating the target speaker's voice from other noise using multimodal information.

[0054] In this embodiment, after the audio is processed by the speech enhancement module, the encoder part of the U-Net, composed of a 1-Dimensional Convolutional Neural Network (1D CNN), has L layers in both the encoder and decoder networks, and the layers of the decoder are numbered in reverse order. The encoder network takes the original waveform as input and outputs a latent representation. Each layer of the network contains a Rectified Linear Unit Activation (ReLU) layer, a convolutional layer, and a Gated Linear Unit Activation (GLU) layer. After processing the video frames, the speech enhancement module uses a feature extraction network composed of a 3D convolutional layer and ResNet-18 to extract the spatiotemporal features of the speaker's facial movements. Then, by using the ResNet-18 network, a single face image is randomly extracted from the face trajectory as input. The speaker's facial attributes are encoded by extracting the face embedding of the corresponding dimension, and then copied along the time dimension to obtain the facial attribute features.

[0055] This invention concatenates three extracted audio and video features along the time dimension using a Transformer bottleneck, fusing the three inputs together, and employs a temporal Transformer encoder to extract temporal features. The temporal Transformer utilizes a self-attention mechanism to capture long-range dependencies in temporal data, while simultaneously improving the model's representational power through a multi-head attention mechanism. After processing by the spatial feature extraction module, the fused audio and video features yield spatial features. These spatial features are then combined with positional encodings and input into the temporal Transformer encoder. In the encoder, each spatial feature and positional encoding is converted into a query, key, and value vector, and multi-head attention is calculated. The calculated weighted values ​​are used to generate intermediate features, which, after layer normalization and residual connections, yield the output of a single Transformer encoder block. The outputs of multiple encoders are averaged to obtain the final spatial-temporal representation. Through this process, the model can effectively capture the temporal dynamic features of the speaker's facial movements and sound sequences, outputting a corresponding enhanced audio representation Y. Subsequently, the enhanced audio representation Y is decoded by a decoder of a U-Net consisting of multiple layers of one-dimensional transposed convolutions containing an audio encoder into a speech enhancement waveform corresponding to the video input.

[0056] In this embodiment, the microphone sampling rate in the data acquisition module is 44.1kHz, while the camera has a resolution of 1080p and a frame rate of 30fps. The file is saved in .npy format for subsequent analysis.

[0057] Although embodiments of the invention have been shown and described (see the detailed description above), it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal speech enhancement system based on audio and video, characterized in that: The enhancement system includes The data acquisition module collects target audio, background noise, and interference sounds via a microphone; The system uses a camera to simultaneously capture environmental visual information, including the speaker's facial expressions and lip movements, and normalizes noise and interference sounds to ensure that the dynamic range matches the target audio. The data analysis module preprocesses the acquired audio, including noise suppression and speech enhancement, and applies deep learning models to improve speech clarity. Analyze the visual information captured by the camera to assist audio processing, thereby improving the accuracy of speech recognition and enhancement; Extract audio features, perform data fusion processing, comprehensively utilize audio and visual information to optimize speech enhancement effects, and evaluate and optimize the processing results; An interactive interface for browsing and comparing the initial audio and video recordings with those enhanced by speech; The data acquisition module includes a data acquisition section and a data preprocessing section; The data analysis module includes a speech enhancement model, used to capture the temporal dynamic features of the speaker's facial movements and sound sequences; The data preprocessing section processes the video portion as follows: S1. Segment the long video and remove the first and last frames to avoid the influence of interference factors; S2. Use 3D face model technology to rotate, translate and scale the image frames and markers to ensure that the facial markers in different image frames are aligned to similar positions; S3. Standardize the image frames, converting pixel values ​​from the [0,1] range to the [-1,1] range; S4. By performing a center cropping operation, the head is placed in the center of the image, and the image size is adjusted to 224×224 pixels. Then, square occlusions are added to the lips in some video frames, and the ratio of clear frames to occluded frames is maintained at 3:

1. The data preprocessing section handles the audio portion as follows: The sampled target audio is mixed with the ambient noise audio signal and the interference audio signal containing a speaker in the AudioSet, with noise accounting for 20%, interference audio accounting for 30%, and target audio accounting for 50%. At the same time, it is normalized according to the maximum amplitude of the mixed audio. The mixed audio signal is saved as a wav file as the mixed audio containing noise. The speech enhancement model is divided into a spatiotemporal feature extraction module and a feature fusion module, wherein... The spatiotemporal feature module uses convolutional blocks to extract feature information from audio, video, and facial appearance as heterogeneous modal information, while the feature fusion module uses the Transformer model to fuse heterogeneous modal information, effectively separating the target speaker's voice from other noise using multimodal information; The speech enhancement model processes the audio and consists of a U-Net encoder part composed of a one-dimensional convolutional neural network. The encoder and decoder networks each have L layers, and the decoder layers are numbered in reverse order. The encoder network takes the original waveform as input and outputs a latent representation. Each layer of the network contains a linear rectified activation function layer, a convolutional layer, and a gated linear unit activation layer. The speech enhancement model processes video frames and then uses a feature extraction network consisting of a three-dimensional convolutional layer and ResNet-18 to extract the spatiotemporal features of the speaker's facial movements. Then, by using the ResNet-18 network, a single face image is randomly extracted from the face trajectory as input. The corresponding dimension of face embedding is extracted to encode the speaker's facial attributes. After that, it is copied along the time dimension to obtain the facial attribute features. The microphone in the data acquisition module has a sampling rate of 44.1 kHz, while the camera has a resolution of 1080p and a frame rate of 30 fps. The file is saved in .npy format for subsequent analysis. The types of ambient noise audio signals include music, laughter, crying, engine sounds, and wind sounds.

Images