Scene-aware audio-visual speech enhancement method, device, medium and program product

Abstract

The present invention provides a scene-aware audio-visual speech enhancement method, device, medium and program product, wherein the method comprises the following steps: extracting the visual scene embedding and scene-aware audio embedding corresponding to the current scene by a pre-trained contrastive audio-visual mask autoencoder; using a one-dimensional convolutional layer to encode the noisy speech signal after short-time Fourier transform to obtain an acoustic spectrum embedding; fusing the visual scene embedding, scene-aware audio embedding and acoustic spectrum embedding to obtain a fused feature representation; processing the fused feature representation by a ConMamba module to obtain a phase-sensitive mask; performing an inverse short-time Fourier transform based on the phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal. The present invention uses visual cues as a component of the enhancement strategy, and processes them through the ConMamba architecture for a larger visual context range to improve the effect of speech enhancement.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and in particular to a scene-aware audiovisual speech enhancement method, device, medium and program product. Background Art

[0002] Speech enhancement (SE) aims to extract clean, original speech signals from noisy speech recordings, suppressing or reducing the noise interference, thereby improving speech reliability and perceived quality. Speech enhancement plays an important role in various applications, such as hearing aids, speech recognition, and speaker tracking.

[0003] With the development of machine learning, related machine learning technologies have been applied to speech enhancement, especially speech enhancement schemes based on deep neural networks, which have shown powerful capabilities in extracting features and separating signals from complex acoustic environments.

[0004] Humans use multimodal cues to explore, capture, and perceive the displayed world. In recent decades, research has focused on the semantic, temporal, and spatial coherence between audio and vision. Using video information as a supplement can uncover details lost in the acoustic scene, enabling audio-visual speech enhancement (AVSE) to provide more robust and precise enhancement capabilities.

[0005] One existing approach uses time-synchronized facial or lip movements to improve the clarity and perception of speech enhancement. While this approach is superior to typical speech enhancement methods, in practice, it often fails to accurately capture changes in facial or lip movements. Inaccurate facial or lip detection can lead to degraded speech enhancement performance.

[0006] Another existing solution is a contextual information-based speech enhancement solution, which enhances audio data through contextual information, which includes multiple sources such as speaker identity, emotional state, and environmental noise. This solution processes these inputs separately through multiple independent encoders, each designed to process a specific data type. This modular design allows the system to focus on analyzing different aspects of the audio signal and the surrounding context. Contextual data mainly comes from metadata or additional audio cues, and is then used to enhance the clarity of speech signals in noisy environments. However, this solution still needs to improve its speech enhancement effect because it uses specialized encoders to analyze and encode various contextual factors. Summary of the Invention

[0007] The first object of the present invention is to provide a scene-aware audiovisual speech enhancement method that improves speech enhancement effect.

[0008] A second object of the present invention is to provide a computer device for implementing the above-mentioned scene-aware audiovisual speech enhancement method.

[0009] A third object of the present invention is to provide a computer-readable storage medium for implementing the above-mentioned scene-aware audiovisual speech enhancement method.

[0010] A fourth object of the present invention is to provide a computer program product for implementing the above-mentioned scene-aware audiovisual speech enhancement method.

[0011] In order to achieve the above-mentioned first objective, the present invention provides a scene-aware audiovisual speech enhancement method, which includes the following steps: extracting a visual scene embedding and a scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audiovisual mask autoencoder; using a one-dimensional convolutional layer to encode the noisy speech signal after short-time Fourier transform to obtain an acoustic spectrum embedding; fusing the visual scene embedding, the scene-aware audio embedding and the acoustic spectrum embedding to obtain a fused feature representation; processing the fused feature representation through a ConMamba module to obtain a phase-sensitive mask; and performing an inverse short-time Fourier transform based on the phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal.

[0012] As can be seen from the above scheme, the present invention provides a multimodal method that integrates visual scene information with audio data. The visual input comes from a video stream and can capture noise sources in the environmental context. This broader range of visual context is handled by the ConMamba architecture, which combines the Conformer module and the Mamba module. The Conformer module focuses on global interactions in the data, while the Mamba module utilizes selective state space models (SSMs) to handle temporal dependencies. This combined architecture enables the system to capture both long-range and local features. This makes the integration of visual and audio data particularly effective for speech enhancement in complex noisy environments. Compared to facial or lip movements, visual cues of environmental information based on noisy scenes or background objects emitting noise are easier to capture in the present invention. Using visual environmental cues provides a valuable supplement to speech enhancement and is more practical. Compared to methods that use specialized encoders to analyze and encode contextual factors and then use this information to refine the audio signal, the present invention incorporates visual cues as an integral part of the enhancement strategy, further improving the effectiveness of speech enhancement.

[0013] A further approach is to extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder, including using an upsampling layer to match the video feature length and audio feature length of the current scene.

[0014] It can be seen that in the process of extracting visual scene embedding and scene-aware audio embedding, in order to ensure the time synchronization between video and audio, an upsampling layer is set to match the video feature length and the audio feature length to ensure the effect of subsequent feature fusion.

[0015] A further solution is to extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder, including: extracting the visual scene embedding corresponding to the current scene through the video encoder in the pre-trained contrastive audio-visual mask autoencoder; extracting the scene-aware audio embedding corresponding to the current scene through the audio encoder in the pre-trained contrastive audio-visual mask autoencoder.

[0016] It can be seen that since the pre-trained contrastive audio-visual mask autoencoder adopts a contrastive learning strategy, it captures the temporal and semantic consistency between audio and video, and can extract visual scene embedding and scene-aware audio embedding respectively.

[0017] A further approach is to concatenate visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding to obtain a fused feature representation.

[0018] It can be seen that by fusing visual scene embedding, scene-aware audio embedding and acoustic spectrum embedding, better speech enhancement quality effect is achieved.

[0019] A further solution is that the ConMamba module is obtained through model training, which includes: setting a dataset of noise and corresponding visual scenes and a clean speech dataset; dynamically mixing clean speech segments of the clean speech dataset with noise segments of the noise and corresponding visual scene dataset, and generating noisy mixed segments in real time with a randomly sampled signal-to-noise ratio between -10 and 20 dB; and training the ConMamba module based on the noisy mixed segments.

[0020] A further solution is that during model training, the ConMamba module also includes performing a gradient update using 10 noisy mixed fragments.

[0021] This shows that the model training effect can be improved.

[0022] In order to achieve the above-mentioned second purpose, the present invention provides a computer device, including a processor and a memory, wherein: the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned scene-aware audio-visual speech enhancement method is implemented.

[0023] In order to achieve the third objective mentioned above, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, the above-mentioned scene-aware audiovisual speech enhancement method is implemented.

[0024] In order to achieve the fourth objective mentioned above, the present invention provides a computer program product, comprising computer instructions, wherein: when the computer instructions are executed by a processor, the above-mentioned scene-aware audio-visual speech enhancement method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of a specific implementation of the scene-aware audiovisual speech enhancement method of the present invention.

[0026] Figure 2 It is a flow chart of an embodiment of the scene-aware audiovisual speech enhancement method of the present invention.

[0027] Figure 3 This is a network structure diagram of an embodiment of the scene-aware audiovisual speech enhancement method of the present invention.

[0028] Figure 4 This is an example diagram of the focus of the video encoder on the input image in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention.

[0029] Figure 5 2. It is a schematic diagram of the scores of the unprocessed case in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention, the ConMamba module of this embodiment, and other models on the PESQ index and the STOI index.

[0030] Figure 6 3. This is a schematic diagram of the scores of the ConMamba module of this embodiment on the PESQ indicator in the embodiment of the scene-aware audio-visual speech enhancement method of the present invention, under two conditions: reducing the input of the visual scene embedding and the scene-aware audio embedding and not reducing the visual scene embedding and the scene-aware audio embedding.

[0031] Figure 7 3. This is a schematic diagram of the scores of the ConMamba module of this embodiment on the STOI indicator in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention, with and without reducing the input of the visual scene embedding and scene-aware audio embedding.

[0032] Figure 8 3. It is a schematic diagram of the scores on the PESQ index and the STOI index after the ablation experiment in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention.

[0033] Figure 9 1 is a logarithmic spectrum diagram of background noise, clean speech, a first processing situation, and a second processing situation in an embodiment of the scene-aware audiovisual speech enhancement method of the present invention.

[0034] Figure 101 is a schematic diagram of the scores of different fusion strategies on the PESQ index and STOI index in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention.

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0036] The scene-aware audiovisual speech enhancement method of the present invention improves the performance of speech enhancement by integrating the visual information of the current scene. Figure 1 The visual cues in the current scene are encoded by a video encoder (Visual Encoder), and the noisy speech, a mixture of background noise and clean speech in the current scene, is encoded by an audio encoder (Audio Encoder). The encoded features are then fused (AV Fusion) and passed through a ConMamba-based speech enhancement backbone network (SE Backbone) to restore clean speech from the noisy speech as closely as possible. The present invention also provides a computer device, computer-readable storage medium, and computer program product for implementing the above-mentioned scene-aware audiovisual speech enhancement method.

[0037] Embodiment of scene-aware audiovisual speech enhancement method:

[0038] See also Figure 2 This embodiment is implemented based on a computer program and specifically includes the following steps:

[0039] S1: Extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder.

[0040] S2: Use a one-dimensional convolutional layer to encode the noisy speech signal after short-time Fourier transform to obtain acoustic spectrum embedding.

[0041] S3: Fusion of visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding to obtain fused feature representation.

[0042] S4: The fused feature representation is processed by the ConMamba module to obtain a phase-sensitive mask.

[0043] S5: Perform inverse short-time Fourier transform based on the phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal.

[0044] In the current scenario, the noisy speech signal It can be expressed as the clean speech signal corresponding to the clean speech Noise signal corresponding to background noise The superposition of , that is: In this embodiment, through the above steps, the noisy speech signal of the current scene is known. In the case of .

[0045] See also Figure 3 , the pre-trained contrastive audiovisual masked autoencoder (CAV-MAE) includes a video encoder and an audio encoder. The pre-trained contrastive audiovisual masked autoencoder uses contrastive learning and masked data modeling to jointly learn coordinated audiovisual representations. This helps to generate more robust audio features and video features, which is beneficial for multimodal tasks of understanding cross-modal interactions. In addition, in the process of extracting visual scene embedding and scene-aware audio embedding, in order to ensure the time synchronization between video and audio, the pre-trained contrastive audiovisual masked autoencoder of this embodiment is also provided with an upsampling layer to match the length of video features extracted from visual cues with the length of audio features to ensure the effect of subsequent feature fusion.

[0046] In the above step S1, the video features corresponding to the current scene are extracted by the video encoder in the pre-trained contrastive audio-visual mask autoencoder and expressed as a visual scene embedding, i.e. ,in, is the visual scene embedding, is a video encoder, It is the video input.

[0047] The audio features corresponding to the current scene are extracted by the audio encoder in the pre-trained contrastive audio-visual mask autoencoder and expressed as scene-aware audio embedding, i.e. ,in, is the visual scene embedding, is the audio encoder, is the Fbank acoustic feature of the noisy speech signal.

[0048] In the above step S2, the noisy speech signal After performing the short-time Fourier transform (STFT) operation, a one-dimensional convolution layer is used for encoding to obtain the acoustic spectrum embedding, which is expressed as ,in, is the acoustic spectrum embedding, is a one-dimensional convolution, is the noisy speech signal after short-time Fourier transform Through one-dimensional convolutional layer encoding, the acoustic spectrum is embedded Embedding with visual scenes and scene-aware audio embedding have the same dimensions.

[0049] So far, we have obtained the visual scene embedding , scene-aware audio embedding , Acoustic Spectrum Embedding ,in, 、 、 denote the time steps of visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding, respectively. Represents the feature dimension.

[0050] In the above step S3, the visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding are concatenated to obtain a fused feature representation, which is expressed as ,in, is the fusion feature representation, Represents an embedded concatenation operation.

[0051] In the above step S4, the fused feature representation is input into the ConMamba module to obtain the phase sensitive mask (PSM), which is expressed as in, is the phase-sensitive mask, is the fusion feature representation, is a trainable parameter is the output layer (the output layer is a linear layer).

[0052] Continue to see Figure 3 ,The ConMamba module includes the Conformer module and the BiMamba module. The Conformer module is used to capture ,long-distance dependencies, and the BiMamba module is used to capture ,granularity features.

[0053] The ConMamba module includes multiple ConMamba layers and an output layer. A ConMamba layer consists of the following parts: (1) Feedforward Network (FFN) layer pre-processing: . (2) Bidirectional Mamba layer: . (3) Convolutional layer: . (4) Layer normalization: .in, is the input of the i-th ConMamba layer, is the output of the input of the i-th ConMamba module, is the feed-forward network layer (Feed-Forward Network), For the BiMamba module, is the convolutional layer, is the layer normalization function.

[0054] The BiMamba module consists of two Mamba layers, one for processing the input sequence along different timelines (forward and backward). By capturing the complex patterns and dependencies of the input sequence, the BiMamba module effectively enhances speech signal processing capabilities.

[0055] The specific processing process of the bidirectional Mamba layer includes: (1) Forward Mamba layer processing: processing the input sequence The forward part of ,in represents the input sequence features, represents the number of time steps, Represents the feature dimension. (2) Backward Mamba layer processing: processing the input sequence The backward part, by flipping the input sequence Implementation, expressed as ,in, Indicates that the input sequence The input sequence is sorted in reverse order. Represents the output of the backward Mamba layer. (3) Fusion of forward and backward results: The forward and backward outputs are fused by adding them element by element to generate a combined bidirectional output , ,in, represents the output of the forward Mamba layer, Represents the output of the backward Mamba layer (4) Normalize the output layer; combine the bidirectional output and the input sequence Added and normalized by layer Get the final output , ,in, Representation layer normalization function, Represents the output sequence of the BiMamba layer. This process ensures that the bidirectional Mamba layer can capture the global dependencies in the input sequence, thereby improving the performance of the model.

[0056] The internal decomposition of the Mamba layer includes: (1) Input decomposition: input features is decomposed into two parallel processing paths, the first path performs a linear transformation and SiLU activation : in, represents the output of the first path; the second path performs linear transformations in sequence , depthwise convolution 、SiLU activation , SSM layer and layer normalization , ,in, Represents the output of the second path. (2) Feature merging: multiplying elements one by one The outputs of the two paths are combined and passed through a linear layer to get the final output: ,in Represents an element-by-element multiplication operation, Represents the final output of the Mamba layer.

[0057] In the above step S5, the phase sensitive mask Get the enhanced speech signal: ,in, is the inverse short-time Fourier transform operation, Represents an element-by-element multiplication operation, is the phase-sensitive mask, is the noisy speech signal after short-time Fourier transform operation The enhanced speech signal is the noisy speech signal Extract as clean speech signals as possible .

[0058] The ConMamba module in this example is obtained through model training. Regarding the dataset setup, the train-clean-100 subset of the LibriSpeech corpus was used for clean speech data, which includes 28,539 sentences from 251 speakers. For noise and corresponding visual scenes, three commonly used datasets were used: Music (MUSIC), Audiovisual Speech (AVSpeech), and AudioSet. The MUSIC dataset includes specialized, high-quality music recordings, the AVSpeech dataset includes a variety of interfering speakers, languages, and facial gestures, simulating complex multi-speaker scenarios, and the AudioSet dataset includes samples of diverse environmental sounds. These three datasets cover a wide range of audio environments, providing a comprehensive evaluation.

[0059] The above-mentioned MUSIC, AVSpeech, and AudioSet datasets are all existing datasets. For the MUSIC dataset, please refer to "The sound of motions" published in the proceedings of the IEEE International Conference on Computer Vision. For the AVSpeech dataset, please refer to "Seeing through noise: Visually driven speaker separation and enhancement" published in the proceedings of the IEEE International Conference on Audio, Speech, and Signal Processing. For the AudioSet dataset, please refer to "Audio Set: An ontologyand human-labeled dataset for audio events" published in the proceedings of the IEEE International Conference on Audio, Speech, and Signal Processing.

[0060] For the above-mentioned noise and corresponding visual scene datasets, noise recordings longer than 30 seconds were segmented into segments no longer than 30 seconds, resulting in 2500 segments corresponding to the MUSIC dataset, 3130 segments corresponding to the AVSpeech dataset, and 8560 segments corresponding to the AudioSet dataset.

[0061] We randomly sampled 1,000 clean speech segments from the train-clean-100 subset of the LibriSpeech corpus, 400 noise segments from the segmented MUSIC dataset, 500 noise segments from the segmented AVSpeech dataset, and 1,000 noise segments from the segmented AVSpeech dataset. All segments were resampled at 16 kHz.

[0062] Each clean speech segment was mixed with random noise segments (in 1dB increments) at random signal-to-noise ratios (SNRs) ranging from -10 to 20dB to create the validation set. Specifically, 200 clean speech segments were randomly selected (without replacement) from the train-clean-100 subset of the LibriSpeech corpus. A total of 200 noise segments were randomly selected from the MUSIC, AVSpeech, and AudioSet datasets after the aforementioned segmentation. Each clean speech segment was mixed with a noise segment selected from these randomly selected noise segments, with the noise segments selected varying between clean speech segments. The validation set was then degraded at five different SNR levels (i.e., -5dB, 0dB, 5dB, 10dB, and 15dB). This generated 1,000 noisy audio and video segments for evaluation.

[0063] For the short-time Fourier transform (STFT) spectrum, a Hanning window of 32 milliseconds (512 samples) and a hop length of 16 milliseconds (256 samples) are used to calculate the spectrum, thus obtaining a 257-point amplitude spectrum as the network input. In the time dimension, Indicates the maximum time range within a batch. and They are fixed at 49 and 128 respectively. Feature dimensions Set to 256. For the Mamba layer, the default hyperparameters are used, where the state dimension is is 16, where the convolution dimension is is 16, where the expansion factor is 2.

[0064] The model generates noisy mixed segments in real time by dynamically mixing clean speech segments with noise segments and randomly sampling the signal-to-noise ratio (SNR) between -10 and 20dB (in 1dB increments). The model uses a batch of 10 noisy mixed segments for gradient update training. The initial learning rate is 1e -3 The Adam optimizer was used with the default hyperparameters of β1 = 0.9 and β2 = 0.999. A warm-up training strategy of 40,000 warm-up steps was used to adjust the learning rate. Gradient clipping was used to clamp gradients to the range of -1 to 1. The root mean square error (MSE) was used as the objective function.

[0065] Thus, the ConMamba module is obtained by training the above dataset and training conditions. Based on the ConMamba module, the above specific scene-aware audiovisual speech enhancement method is implemented. Figure 4 , which image regions contribute to speech enhancement through gradient-weighted class activation heatmaps (Grad-CAM), Figure 4 The image contains a row of original video frames, and each frame has a corresponding Grad-CAM heat map below it. These heat maps emphasize the key areas of the noise source, namely Figure 4 The heat map shows the ability of the method of this embodiment to focus on contextual elements, such as the moving duck, pedestrians, and guitar.

[0066] In this embodiment, three baseline models are provided for comparison with the ConMamba module: the Conformer-based MP-SENet model, the AViTAR model, and the ExtBi-Mamba model. The Conformer-based MP-SENet model adopts a codec architecture for SE, in which the encoder and decoder are connected by a Conformer module; the AViTAR model transforms sound from one space to another by changing the scene-driven acoustic features of the sound; and the ExtBi-Mamba model applies bidirectional Mamba to monophonic speech enhancement for the first time. The above-mentioned Conformer-based MP-SENet model, AViTAR model, and ExtBi-Mamba model are trained under similar training conditions as the ConMamba module, including using the same dataset and training conditions.

[0067] See also Figure 5 For the MUSIC dataset, AVSpeech dataset, or AudioSet dataset, in terms of the PESQ (Perceptual evaluation of speech quality) and STOI (Short-Time Objective Intelligibility) indicators, compared with the unprocessed (i.e., Umprocessed in the figure), the above-mentioned MP-SENet model, AViTAR model, ExtBi-Mamba model, and the ConMamba module (VC-S) applied in this embodiment are significantly better. 2E) all achieved speech enhancement. The speech enhancement method using the ConMamba module in this embodiment demonstrated superior performance in both the PESQ and STOI metrics on three datasets compared to the speech enhancement methods using the Conformer-based MP-SENet model, the AViTAR model, or the ExtBiMamba model.

[0068] See also Figure 6 and Figure 7 , Figure 6 and Figure 7The figure shows the scores of the PESQ index and the STOI index when the method of applying the ConMamba module of this embodiment is verified in different categories of the AudioSet dataset. Among them, category "1" represents the "Fire" category in the AudioSet dataset, category "2" represents the "Child speech" category in the AudioSet dataset, category "3" represents the "Male speech" category in the AudioSet dataset, category "4" represents the "Musical instrument" category in the AudioSet dataset, category "5" represents the "Music" category in the AudioSet dataset, category "6" represents the "Water" category in the AudioSet dataset, and category "7" represents the "Female speech" category in the AudioSet dataset. The following examples represent the "female speech" category in the AudioSet dataset: category "8" represents the "Thunderstorm" category in the AudioSet dataset; category "9" represents the "Vehicle" category in the AudioSet dataset; category "10" represents the "Speech" category in the AudioSet dataset; category "11" represents the "Thunder" category in the AudioSet dataset; category "12" represents the "Windnoise" category in the AudioSet dataset; category "13" represents the "Plucked instrument" category in the AudioSet dataset; category "14" represents the "Bird" category in the AudioSet dataset; and category "15" represents the "Rowboat" category in the AudioSet dataset. When visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding are input, superior performance is achieved in both the PESQ and STOI metrics compared to inputting only acoustic spectrum embedding. For example, on category "1" of the AudioSet dataset, when visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding are input, the score of the PESQ indicator is improved from 2.50 to 2.62, and the score of the STOI indicator is improved from 91.61% to 92.57% compared to only inputting acoustic spectrum embedding.

[0069] See also Figure 8 , Figure 8 Figure 2 shows the score diagram of the PESQ index and the STOI index in the ablation experiment. Figure 8In the method of speech enhancement using the ConMamba module of this embodiment, when visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding are present as inputs, the method shows better results in both PESQ and STOI indicators compared to when the visual scene embedding and scene-aware audio embedding inputs are reduced, and compared to when the scene-aware audio embedding inputs are reduced. As for the method of speech enhancement using the Conformer module, although the method shows better results in both PESQ and STOI indicators compared to when the visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding inputs are present as inputs, the overall effect is still inferior to the method of this embodiment using the ConMamba module.

[0070] See also Figure 9 , Figure 9 The following diagrams show the log-magnitude spectrograms corresponding to background noise, clean speech, the first processing scenario, and the second processing scenario, respectively. The first processing scenario corresponds to the ConMamba module of this embodiment when visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding are present as input. The second processing scenario corresponds to the ConMamba module of this embodiment when only acoustic spectrum embedding is present as input. As can be seen from the white circled areas in the first and second processing scenarios, the first processing scenario has better noise reduction capabilities.

[0071] See also Figure 10 , Figure 10 The corresponding PESQ and STOI scores for different feature fusion strategies are shown. It can be confirmed that visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding achieve superior results in both PESQ and STOI compared to other feature fusion methods.

[0072] In summary, this invention provides a multimodal approach that integrates visual scene information with audio data. The visual input comes from a video stream, capturing noise sources within the context of the environment, such as a dog barking or a passing vehicle. This broader visual context is processed using the ConMamba architecture. The Conformer module focuses on global interactions in the data, while the Mamba module utilizes a selective state-space model (SSM) to address temporal dependencies. This combined architecture enables the system to capture both long-range and local features, making the integration of visual and audio data particularly effective for speech enhancement in complex noisy environments. This invention incorporates visual cues as an integral part of the enhancement strategy, processing a wider range of visual context using the ConMamba architecture to improve speech enhancement.

[0073] Computer device embodiment:

[0074] The computer device of this embodiment includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the above-mentioned scene-aware audiovisual speech enhancement method embodiment is implemented.

[0075] A computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that a computer device may include more or fewer components, or a combination of certain components, or different components. For example, a computer device may also include input and output devices, network access devices, a bus, and the like.

[0076] For example, a processor can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microcontroller or any conventional processor. The processor is the control center of a computer device, connecting the various parts of the entire computer device using various interfaces and lines.

[0077] The memory can be used to store computer programs and / or modules. The controller implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. For example, the memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound receiving function, a sound-to-text conversion function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, text data, etc.). In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0078] Computer readable storage medium embodiment:

[0079] If the modules integrated into the computer device of the above-mentioned embodiment are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the process of the embodiment of the scene-aware audiovisual speech enhancement method can also be completed by a computer program instructing the relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a controller, it can implement the steps of the above-mentioned embodiment of the scene-aware audiovisual speech enhancement method. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The storage medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0080] Computer program product embodiment:

[0081] The computer program product of this embodiment includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the above-described embodiment of the scene-aware audiovisual speech enhancement method.

Claims

1. A scene-aware audio-visual speech enhancement method, characterized in that: The following steps are involved: Extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder; The noisy speech signal after short-time Fourier transform is encoded using a one-dimensional convolutional layer to obtain an acoustic spectrum embedding; fusing the visual scene embedding, the scene-aware audio embedding, and the acoustic spectrum embedding to obtain a fused feature representation; The fused feature representation is processed by the ConMamba module to obtain a phase-sensitive mask ;in, , is the phase-sensitive mask, is the fusion feature representation, is a trainable parameter, is the output layer; An inverse short-time Fourier transform is performed based on the phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal. ;in, , is to enhance the speech signal, is the inverse short-time Fourier transform operation, Represents an element-by-element multiplication operation, is the phase-sensitive mask, is a noisy speech signal after short-time Fourier transform operation.

2. A scene-aware audio-visual speech enhancement method as claimed in claim 1, characterized in that: Extracting the visual scene embedding and the scene-aware audio embedding corresponding to the current scene through the pre-trained contrastive audio-visual mask autoencoder includes: using an upsampling layer to match the video feature length and the audio feature length of the current scene.

3. A scene-aware audio-visual speech enhancement method as claimed in claim 2, characterized in that: Extracting the visual scene embedding and the scene-aware audio embedding corresponding to the current scene through the pre-trained contrastive audio-visual mask autoencoder, comprising: extracting a visual scene embedding corresponding to the current scene through a video encoder in the pre-trained contrastive audio-visual mask autoencoder; A scene-aware audio embedding corresponding to the current scene is extracted by an audio encoder in the pre-trained contrastive audio-visual mask autoencoder.

4. A scene-aware audio-visual speech enhancement method as claimed in claim 1, characterized in that: The visual scene embedding, the scene-aware audio embedding and the acoustic spectrum embedding are concatenated to obtain the fused feature representation.

5. A scene-aware audio-visual speech enhancement method according to any one of claims 1 to 4, characterized in that: The ConMamba module is obtained through model training, and the model training includes: Set up a dataset of noise and corresponding visual scenes and a clean speech dataset; Dynamically mixing clean speech segments of the clean speech data set with noise segments of the noise and corresponding visual scene data sets, and generating noisy mixed segments in real time with a signal-to-noise ratio randomly sampled between -10 and 20 dB; The ConMamba module is trained based on the noisy mixed segment.

6. A scene-aware audio-visual speech enhancement method as claimed in claim 5, characterized in that: When the model is trained, it also includes: the ConMamba module uses 10 noisy mixed fragments to perform a gradient update.

7. A computer device comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the scene-aware audiovisual speech enhancement method described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the scene-aware audiovisual speech enhancement method described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the scene-aware audiovisual speech enhancement method described in any one of claims 1 to 6 is implemented.

Images