Scene-aware audiovisual speech enhancement method and device, medium and program product
By comparing the fusion feature processing of the audio-visual mask automatic encoder and the ConMamba module, integrating visual and audio data, the problem of poor voice enhancement effect in complex noise environments is solved, and more efficient voice signals are achieved recover.
Patent Information
- Application Number
- CN202411407184.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-10-10
AI Technical Summary
Existing voice enhancement technologies do not work well in complex noise environments, especially performance degradation caused by inaccurate facial or lip motion detection, and the context information processing effect based on specialized encoders still has room for improvement.
The pre-trained contrast audio mask automatic encoder extracts visual scene embedding and scene-aware audio embedding, and fusion feature representations are carried out through the ConMamba module, combining Conformer and Mamba module processing, captures remote and local features, and integrates visual and audio data to enhance voice quality.
The effect of speech enhancement is significantly improved in complex noise environments. Compared with traditional methods, it can capture environmental noise sources more effectively, provide more practical visual clues, and improve the clarity and reliability of speech signals.
Smart Images

Figure CN118918913A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech enhancement, and in particular to a scene-aware audiovisual speech enhancement method, device, medium and program product. Background Art
[0002] Speech Enhancement (SE) aims to extract clean original speech signals from noise-corrupted speech recordings to suppress or reduce noise interference, thereby improving speech reliability and perceived quality. Speech enhancement plays an important role in various applications, such as hearing aids, speech recognition, and speaker tracking.
[0003] With the development of machine learning, related machine learning technologies have been applied in speech enhancement, especially speech enhancement schemes based on deep neural networks, which have shown powerful capabilities in extracting features and separating signals from complex acoustic environments.
[0004] Humans use multimodal cues to explore, capture, and perceive the displayed world. In recent decades, people have studied the semantic, temporal, and spatial consistency between audio and vision. Using video information as a supplement can discover details lost in the acoustic scene, and the resulting Audio-Visual Speech Enhancement (AVSE) provides more robust and more precise enhancement capabilities.
[0005] An existing solution is to use time-synchronized facial or lip movements to improve the clarity and perception of speech enhancement. Although this solution is better than general speech enhancement solutions, in actual applications, many scenarios cannot accurately capture changes in facial or lip movements, and inaccurate facial or lip detection may lead to a decrease in the performance of speech enhancement.
[0006] Another existing solution is a speech enhancement solution based on contextual information, which enhances audio data through contextual information, which includes multiple sources such as speaker identity, emotional state, and environmental noise. This solution processes these inputs separately through multiple independent encoders, each of which is designed to process a specific data type. This modular design allows the system to focus on analyzing different aspects of the audio signal and the surrounding context. Contextual data mainly comes from metadata or additional audio clues, which are then used to enhance the clarity of speech signals in noisy environments. However, this solution still needs to be improved in terms of speech enhancement effects because it analyzes and encodes various contextual factors through specialized encoders. Summary of the invention
[0007] The first object of the present invention is to provide a scene-aware audiovisual speech enhancement method for improving speech enhancement effect.
[0008] The second object of the present invention is to provide a computer device for implementing the above-mentioned scene-aware audiovisual speech enhancement method.
[0009] A third object of the present invention is to provide a computer-readable storage medium for implementing the above-mentioned scene-aware audiovisual speech enhancement method.
[0010] A fourth object of the present invention is to provide a computer program product for implementing the above-mentioned scene-aware audiovisual speech enhancement method.
[0011] In order to achieve the above-mentioned first purpose, the present invention provides a scene-aware audiovisual speech enhancement method, which includes the following steps: extracting a visual scene embedding and a scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audiovisual mask autoencoder; using a one-dimensional convolutional layer to encode the noisy speech signal after short-time Fourier transform to obtain an acoustic spectrum embedding; fusing the visual scene embedding, the scene-aware audio embedding and the acoustic spectrum embedding to obtain a fused feature representation; processing the fused feature representation through a ConMamba module to obtain an extended phase-sensitive mask; performing an inverse short-time Fourier transform based on the extended phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal.
[0012] As can be seen from the above scheme, the present invention provides a multimodal method that integrates visual scene information with audio data. The visual input comes from a video stream and can capture noise sources in the environmental context. This wider range of visual context is processed by the ConMamba architecture, which combines the Conformer module and the Mamba module. The Conformer module focuses on global interactions in the data, while the Mamba module uses selective state space models (SSM) to process time dependencies. This combined architecture enables the system to capture long-range and local features at the same time. When applied in complex noisy environments, it can make the integration of visual and audio frequency data to enhance speech quality particularly effective. Compared with facial or lip movements, the visual clues of environmental information are easier to capture based on noise scenes or background objects that emit noise. It is more practical to use visual environmental clues to provide valuable supplements for speech enhancement. Compared with the way of analyzing and encoding contextual factors through a dedicated encoder and then refining the audio signal using this information, the present invention uses visual clues as a component of the enhancement strategy, which can further improve the effect of speech enhancement.
[0013] A further solution is to extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder, including: using an upsampling layer to match the video feature length and audio feature length of the current scene.
[0014] It can be seen that in the process of extracting visual scene embedding and scene-aware audio embedding, in order to ensure the time synchronization between video and audio, an upsampling layer is set to match the video feature length with the audio feature length to ensure the effect of subsequent feature fusion.
[0015] A further solution is to extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual masked autoencoder, including: extracting the visual scene embedding corresponding to the current scene through a video encoder in the pre-trained contrastive audio-visual masked autoencoder; extracting the scene-aware audio embedding corresponding to the current scene through an audio encoder in the pre-trained contrastive audio-visual masked autoencoder.
[0016] It can be seen that since the pre-trained contrastive audio-visual mask autoencoder adopts a contrastive learning strategy, it captures the temporal and semantic consistency between audio and video, and can extract visual scene embedding and scene-aware audio embedding respectively.
[0017] A further approach is to concatenate visual scene embedding, scene-aware audio embedding and acoustic spectrum embedding to obtain a fused feature representation.
[0018] It can be seen that by fusing visual scene embedding, scene-aware audio embedding and acoustic spectrum embedding, better speech enhancement quality effect can be achieved.
[0019] A further solution is that the ConMamba module is obtained through model training, and the model training includes: setting a data set of noise and corresponding visual scenes and a clean speech data set; dynamically mixing clean speech segments of the clean speech data set with noise segments of the noise and corresponding visual scene data set, and generating noisy mixed segments in real time with a signal-to-noise ratio randomly sampled between -10 and 20 dB; and training the ConMamba module based on the noisy mixed segments.
[0020] A further solution is that when training the model, it also includes: the ConMamba module uses 10 noisy mixed fragments to perform a gradient update.
[0021] This shows that the model training effect can be improved.
[0022] In order to achieve the above-mentioned second purpose, the present invention provides a computer device, including a processor and a memory, wherein: a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned scene-aware audio-visual speech enhancement method is implemented.
[0023] In order to achieve the third objective mentioned above, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, the above-mentioned scene-aware audiovisual speech enhancement method is implemented.
[0024] In order to achieve the fourth objective mentioned above, the present invention provides a computer program product, including computer instructions, wherein: when the computer instructions are executed by a processor, the above-mentioned scene-aware audio-visual speech enhancement method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flowchart of a specific implementation of the scene-aware audiovisual speech enhancement method of the present invention.
[0026] Figure 2 It is a flow chart of an embodiment of the scene-aware audiovisual speech enhancement method of the present invention.
[0027] Figure 3 It is a network structure diagram of an embodiment of the scene-aware audio-visual speech enhancement method of the present invention.
[0028] Figure 4 This is an example diagram of the focus of the video encoder on the input image in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention.
[0029] Figure 5 It is a schematic diagram of the scores of the unprocessed situation in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention, the ConMamba module of this embodiment and other models on the PESQ index and the STOI index.
[0030] Figure 6 It is a schematic diagram of the scores of the ConMamba module of this embodiment on the PESQ indicator in the two cases where the input of the visual scene embedding and the scene-aware audio embedding is reduced and the visual scene embedding and the scene-aware audio embedding are not reduced in the embodiment of the scene-aware audio enhancement method of the present invention.
[0031] Figure 7 1 is a schematic diagram of the scores of the ConMamba module of this embodiment on the STOI indicator in the two cases of reducing and not reducing the input of visual scene embedding and scene-aware audio embedding in the embodiment of the scene-aware audio-visual speech enhancement method of the present invention.
[0032] Figure 8 It is a schematic diagram of the scores on the PESQ index and the STOI index after the ablation experiment in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention.
[0033] Fig. 9 It is a logarithmic spectrum diagram of background noise, clean speech, first processing situation and second processing situation in the embodiment of the scene-aware audio-visual speech enhancement method of the present invention.
[0034] Fig.10It is a schematic diagram of the scores of different fusion strategies on the PESQ index and the STOI index in the embodiment of the scene-aware audiovisual speech enhancement method of the present invention.
[0035] The present invention is further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION
[0036] The scene-aware audiovisual speech enhancement method of the present invention improves the performance of speech enhancement by integrating the visual information of the current scene. Figure 1 , the visual cues in the current scene are encoded by a video encoder (VisualEncoder), the noisy speech mixed with the background noise and clean speech in the current scene is encoded by an audio encoder (Audio Encoder), and then the encoded features are fused (AV Fusion), and the fused features are passed through a speech enhancement backbone network (SE Backbone) based on ConMamba, so as to restore the clean speech as much as possible from the noisy speech. The present invention also provides a computer device, a computer-readable storage medium and a computer program product for implementing the above-mentioned scene-aware audiovisual speech enhancement method.
[0037] Embodiment of scene-aware audio-visual speech enhancement method: See also Figure 2 This embodiment is implemented based on a computer program and specifically includes the following steps: S1: Extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder.
[0038] S2: Use a one-dimensional convolutional layer to encode the noisy speech signal after short-time Fourier transform to obtain the acoustic spectrum embedding.
[0039] S3: Fusion of visual scene embedding, scene-aware audio embedding and acoustic spectrum embedding to obtain fused feature representation.
[0040] S4: The fused feature representation is processed by the ConMamba module to obtain the extended phase-sensitive mask.
[0041] S5: Perform inverse short-time Fourier transform based on the extended phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal.
[0042] In the current scenario, the noisy speech signal It can be expressed as the clean speech signal corresponding to the clean speech The noise signal corresponding to the background noise The superposition of, that is: In this embodiment, through the above steps, the noisy speech signal of the current scene is known. In the case of .
[0043] See also Figure 3 , the pre-trained contrastive audio-visual masked autoencoder (CAV-MAE) includes a video encoder and an audio encoder. The pre-trained contrastive audio-visual masked autoencoder uses contrastive learning and masked data modeling to jointly learn coordinated audio-visual representations. This helps to generate more robust audio features and video features, which is beneficial for understanding multimodal tasks of cross-modal interactions. In addition, in the process of extracting visual scene embedding and scene-aware audio embedding, in order to ensure the time synchronization between video and audio, the pre-trained contrastive audio-visual masked autoencoder of this embodiment is also provided with an upsampling layer to match the video feature length and audio feature length extracted from the visual cues to ensure the effect of subsequent feature fusion.
[0044] In the above step S1, the video features corresponding to the current scene are extracted by the video encoder in the pre-trained contrastive audio-visual mask autoencoder and represented as a visual scene embedding, that is, ,in, is the visual scene embedding, is the video encoder, It is the video input.
[0045] The audio features corresponding to the current scene are extracted by the audio encoder in the pre-trained contrastive audio-visual mask autoencoder and represented as scene-aware audio embedding, i.e. ,in, is the visual scene embedding, is the audio codec, is the Fbank acoustic feature of the noisy speech signal.
[0046] In the above step S2, the noisy speech signal After performing a short-time Fourier transform (STFT) operation, a one-dimensional convolutional layer is used for encoding to obtain the acoustic spectrum embedding, which is expressed as ,in, is the acoustic spectrum embedding, is a one-dimensional convolution, is the noisy speech signal after short-time Fourier transform . The acoustic spectrum is embedded in the one-dimensional convolutional layer. Embedding with visual scenes and scene-aware audio embedding have the same dimensions.
[0047] So far, we get the visual scene embedding , scene-aware audio embedding , Acoustic Spectrum Embedding ,in, , , denote the time steps of visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding, respectively. Represents the feature dimension.
[0048] In the above step S3, the visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding are concatenated to obtain a fused feature representation, which is expressed as ,in, is the fusion feature representation, Represents an embedded concatenation operation.
[0049] In the above step S4, the fused feature representation is input into the ConMamba module to obtain the extended phase sensitive mask (PSM), which is expressed as in, is the extended phase-sensitivity mask, is the fusion feature representation, is a trainable parameter is the output layer (the output layer is a linear layer).
[0050] Continue to see Figure 3 ,The ConMamba module includes the Conformer module and the BiMamba module. The Conformer module is used to capture ,long-distance dependencies, and the BiMamba module is used to capture ,granularity features.
[0051] The ConMamba module includes multiple ConMamba layers and an output layer. A ConMamba layer includes the following parts: (1) Feedforward network (FFN) layer pre-processing: . (2) Bidirectional Mamba layer: (3)Convolutional layer: . (4) Layer normalization: .in, is the input of the i-th ConMamba layer, is the output of the input of the i-th ConMamba module, is the feed-forward network layer (Feed-Forward Network), For the BiMamba module, is the convolutional layer, is the layer normalization function.
[0052] The BiMamba module consists of two Mamba layers, one for processing the input sequence in different time axes (forward and backward). By capturing the complex patterns and dependencies of the input sequence, the BiMamba module can effectively enhance the processing capabilities of speech signals.
[0053] The specific processing process of the bidirectional Mamba layer includes: (1) Forward Mamba layer processing: processing the input sequence The forward part of ,in represents the input sequence features, represents the number of time steps, Represents the feature dimension. (2) Backward Mamba layer processing: processing the input sequence The backward part is achieved by flipping the input sequence Implementation, expressed as ,in, Indicates that the input sequence The input sequence is sorted in reverse order. Represents the output of the backward Mamba layer. (3) Fusion of forward and backward results: The forward and backward outputs are fused by adding each element together to generate a combined bidirectional output , ,in, represents the output of the forward Mamba layer, Represents the output to the Mamba layer (4) Normalize the output layer; combine the bidirectional output and the input sequence Add and normalize through the layer Get the final output , ,in, Representation layer normalization function, Represents the output sequence of the BiMamba layer. This process ensures that the bidirectional Mamba layer can capture the global dependencies in the input sequence, thereby improving the performance of the model. The internal decomposition of the Mamba layer includes: (1) Input decomposition: input features is decomposed into two parallel paths, the first path performs a linear transformation and SiLU activation : in, represents the output of the first path; the second path performs linear transformations in sequence , deep convolution 、SiLU activation , SSM layer and layer normalization , ,in, Represents the output of the second path. (2) Feature merging: multiplying elements one by one The outputs of the two paths are combined and passed through a linear layer to get the final output: ,in Represents an element-by-element multiplication operation, Represents the final output of the Mamba layer.
[0054] In the above step S5, by extending the phase sensitive mask Get the enhanced speech signal: ,in, is the inverse short-time Fourier transform operation, represents an element-by-element multiplication operation, To extend the phase-sensitive mask, is the noisy speech signal after short-time Fourier transform operation The enhanced speech signal is the noisy speech signal. Extract as clean a speech signal as possible .
[0055] The ConMamba module of this embodiment is obtained through model training. In terms of data set settings, for clean speech data, the train-clean-100 subset of the LibriSpeech corpus is used, which includes 28,539 sentences from 251 speakers; for noise and corresponding visual scenes, three commonly used data sets are used: music (MUSIC), audio-visual speech (AVSpeech) and audio set (AudioSet). The MUSIC data set includes specialized high-quality music recordings, the AVSpeech data set involves a variety of interfering speakers, languages, and facial gestures, and can simulate complex multi-speaker scenarios. The AudioSet data set includes different environmental sound samples. These three data sets cover a variety of audio environments and provide a comprehensive evaluation.
[0056] The above-mentioned MUSIC, AVSpeech, and AudioSet datasets are all existing datasets. The MUSIC dataset can refer to "The sound of motions" published in the conference proceedings of the IEEE International Conference on Computer Vision, the AVSpeech dataset can refer to "Seeing through noise: Visually driven speaker separation and enhancement" published in the conference proceedings of the IEEE International Conference on Audio, Speech, and Signal Processing, and the AudioSet dataset can refer to "Audio Set: An ontologyand human-labeled dataset for audio events" published in the conference proceedings of the IEEE International Conference on Audio, Speech, and Signal Processing.
[0057] For the above noise and corresponding visual scene datasets, noise recordings longer than 30 seconds were segmented into segments no longer than 30 seconds, resulting in 2500 segments corresponding to the MUSIC dataset, 3130 segments corresponding to the AVSpeech dataset, and 8560 segments corresponding to the AudioSet dataset.
[0058] 1000 clean speech segments were randomly selected from the train-clean-100 subset of the LibriSpeech corpus, 400 noise segments were randomly selected from the MUSIC dataset after the segmentation, 500 noise segments were randomly selected from the AVSpeech dataset after the segmentation, and 1000 noise segments were randomly selected from the AVSpeech dataset after the segmentation. All the above segments were resampled at 16kHz.
[0059] Each clean speech segment is mixed with random noise segments (in increments of 1 dB) at random signal-to-noise ratios (SNRs) ranging from -10 to 20 dB to create a validation set. Specifically, 200 clean speech segments are randomly selected from the train-clean-100 subset of the above-mentioned LibriSpeech corpus (without replacement), and a total of 200 noise segments are randomly selected from the above-mentioned segmented MUSIC dataset, AVSpeech dataset, and AudioSet dataset. Each clean speech segment selects a noise segment from the above-mentioned randomly selected noise segments for mixing, and the noise segments selected between different clean speech segments are different. Then, degradation processing is performed at five different SNR levels (i.e., -5 dB, 0 dB, 5 dB, 10 dB, and 15 dB), thus generating 1,000 noisy audio and video segments for evaluation.
[0060] For the short-time Fourier transform (STFT) spectrum, a Hanning window of 32 milliseconds (512 samples) and a hop length of 16 milliseconds (256 samples) are used to calculate, thereby obtaining a 257-point amplitude spectrum as the network input. In the time dimension, Indicates the maximum time range within a batch. and They are fixed to 49 and 128 respectively. Feature Dimension Set to 256. For the Mamba layer, use the default hyperparameters, where the state dimension is is 16, where the convolution dimension is is 16, where the expansion factor is 2.
[0061] The noisy mixed segments are generated in real time by dynamically mixing clean speech segments and noise segments, and randomly sampling the signal-to-noise ratio (SNR) between -10 and 20dB (increments of 1dB). The model uses a batch of 10 noisy mixed segments for gradient update training. The initial learning rate is 1e -3 The Adam optimizer was used with the default hyperparameters, β1 = 0.9, β2 = 0.999. A warm-up training strategy with 40,000 warm-up steps was used to adjust the learning rate. Gradient clipping was used to limit the gradient value to the range of -1 to 1. The root mean square error (MSE) was used as the objective function.
[0062] Thus, the ConMamba module is obtained by training the above data set and training conditions. The above specific scene-aware audiovisual speech enhancement method is implemented based on the ConMamba module. Figure 4 , which image regions contribute to speech enhancement through gradient-weighted class activation heatmaps (Grad-CAM), Figure 4 The image contains a row of original video frames, and each original video frame has a corresponding Grad-CAM heat map below it. These heat maps emphasize the key areas of the noise source, namely Figure 4 The heat map shows the ability of the method of this embodiment to focus on contextual elements, such as moving ducks, pedestrians, and guitars.
[0063] In this embodiment, three baseline models are provided for comparison with the ConMamba module: the Conformer-based MP-SENet model, the AViTAR model, and the ExtBi-Mamba model. Among them, the Conformer-based MP-SENet model adopts a codec architecture for SE, in which the encoder and the decoder are connected by a Conformer module; the AViTAR model converts sound from one space to another by changing the scene-driven acoustic features of the sound; the ExtBi-Mamba model applies bidirectional Mamba to monophonic speech enhancement for the first time. The above-mentioned Conformer-based MP-SENet model, AViTAR model, and ExtBi-Mamba model are trained under similar training conditions to the ConMamba module, including using the same data set and training conditions.
[0064] See also Figure 5 For the MUSIC dataset, AVSpeech dataset or AudioSet dataset, in terms of the PESQ (Perceptual evaluation of speech quality) index and the STOI (Short-Time Objective Intelligibility) index, compared with the unprocessed (i.e., Umprocessed in the figure), the above-mentioned MP-SENet model, AViTAR model, ExtBi-Mamba model and the ConMamba module (VC-S 2E) methods all achieve the effect of speech enhancement. The method of using the ConMamba module for speech enhancement in this embodiment shows better results in terms of PESQ and STOI indicators than the methods of using the Conformer-based MP-SENet model, AViTAR model or ExtBiMamba model for speech enhancement on three data sets.
[0065] See also Figure 6 and Figure 7 , Figure 6 and Figure 7The figure shows the scores of the PESQ index and the STOI index when the method of applying the ConMamba module of this embodiment is verified in different categories of the AudioSet dataset. Among them, category "1" represents the "Fire" category in the AudioSet dataset, category "2" represents the "Child speech" category in the AudioSet dataset, category "3" represents the "Male speech" category in the AudioSet dataset, category "4" represents the "Musical instrument" category in the AudioSet dataset, category "5" represents the "Music" category in the AudioSet dataset, category "6" represents the "Water" category in the AudioSet dataset, and category "7" represents the "Female speech" category in the AudioSet dataset. The category "speech" (female speech) in the AudioSet dataset, category "8" represents the "Thunderstorm" category in the AudioSet dataset, category "9" represents the "Vehicle" category in the AudioSet dataset, category "10" represents the "Speech" category in the AudioSet dataset, category "11" represents the "Thunder" category in the AudioSet dataset, category "12" represents the "Windnoise" category in the AudioSet dataset, category "13" represents the "Plucked instrument" category in the AudioSet dataset, category "14" represents the "Bird" category in the AudioSet dataset, and category "15" represents the "Rowboat" category in the AudioSet dataset. When it is determined that there are visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding inputs, better results are shown in both the PESQ indicator and the STOI indicator compared to only inputting the acoustic spectrum embedding input. For example, on category “1” of the AudioSet dataset, when there are visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding inputs, the score of the PESQ indicator is improved from 2.50 to 2.62, and the score of the STOI indicator is improved from 91.61% to 92.57% compared to the input of only acoustic spectrum embedding.
[0066] See also Figure 8 , Figure 8 Figure 2 shows a schematic diagram of the scores of the PESQ index and the STOI index in the ablation experiment. Figure 8In the method of speech enhancement using the ConMamba module of this embodiment, when there are visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding inputs, compared with reducing the visual scene embedding and scene-aware audio embedding inputs, and compared with reducing the scene-aware audio embedding inputs, the PESQ index and the STOI index both show better results. For the method of speech enhancement using the Conformer module, although when there are visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding inputs, compared with reducing the visual scene embedding and scene-aware audio embedding inputs, and compared with reducing the scene-aware audio embedding inputs, the PESQ index and the STOI index both show better results, the overall effect is still not as good as the method of this embodiment using the ConMamba module.
[0067] See also Fig. 9 , Fig. 9 The log-magnitude spectrograms corresponding to the background noise, clean speech, the first processing scenario, and the second processing scenario are shown respectively. The first processing scenario is the corresponding scenario after the ConMamba module of this embodiment is processed when there is visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding input. The second processing scenario is the corresponding scenario after the ConMamba module of this embodiment is processed when only the acoustic spectrum is embedded in the input. It can be seen from the white circle area in the first processing scenario and the second processing scenario that the first processing scenario has better noise reduction capability.
[0068] See also Fig.10 , Fig.10 The corresponding PESQ and STOI scores under different feature fusion strategies are shown. It can be determined that when visual scene embedding, scene-aware audio embedding, and acoustic spectrum embedding splicing are compared with other feature fusion methods, both PESQ and STOI show better results.
[0069] In summary, the present invention provides a multimodal method that integrates visual scene information with audio data. The visual input comes from a video stream and can capture noise sources in the environmental context, such as dog barking or passing vehicles. This wider range of visual context is processed through the ConMamba architecture. The Conformer module focuses on global interactions in the data, while the Mamba module uses a selective state space model (SSM) to process temporal dependencies. This combined architecture enables the system to capture both long-range and local features at the same time. In complex noisy environments, integrating visual and audio frequency data to enhance speech quality is particularly effective. The present invention uses visual cues as part of the enhancement strategy and processes them through the ConMamba architecture for a larger range of visual context to improve the effect of speech enhancement.
[0070] Computer device embodiment: The computer device of this embodiment includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the above-mentioned scene-aware audiovisual speech enhancement method embodiment is implemented.
[0071] The computer device may include but is not limited to a processor and a memory. Those skilled in the art will appreciate that the computer device may include more or fewer components, or a combination of certain components, or different components, for example, the computer device may also include input and output devices, network access devices, buses, etc.
[0072] For example, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microcontroller or any conventional processor, etc. The processor is the control center of a computer device, and uses various interfaces and lines to connect various parts of the entire computer device.
[0073] The memory can be used to store computer programs and / or modules. The controller realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. For example, the memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound receiving function, a sound conversion to text function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0074] Computer readable storage medium embodiment: If the module integrated in the computer device of the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the process of the embodiment of the scene-aware audio-visual speech enhancement method can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the controller, the steps of the above-mentioned scene-aware audio-visual speech enhancement method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0075] Computer program product embodiment: The computer program product of this embodiment includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the various steps of the above-mentioned scene-aware audiovisual speech enhancement method embodiment.
Claims
1. A scene-aware audio-visual speech enhancement method, characterized in that: The following steps are involved: Extract the visual scene embedding and scene-aware audio embedding corresponding to the current scene through a pre-trained contrastive audio-visual mask autoencoder; The noisy speech signal after short-time Fourier transform is encoded using a one-dimensional convolutional layer to obtain an acoustic spectrum embedding; fusing the visual scene embedding, the scene-aware audio embedding, and the acoustic spectrum embedding to obtain a fused feature representation; Processing the fused feature representation through a ConMamba module to obtain an extended phase-sensitive mask; An inverse short-time Fourier transform is performed based on the extended phase-sensitive mask and the noisy speech signal to obtain an enhanced speech signal.
2. A scene-aware audio-visual speech enhancement method as claimed in claim 1, characterized in that: The visual scene embedding and the scene-aware audio embedding corresponding to the current scene are extracted by the pre-trained contrastive audio-visual mask autoencoder, including: using an upsampling layer to match the video feature length and the audio feature length of the current scene.
3. A scene-aware audio-visual speech enhancement method as claimed in claim 2, characterized in that: Extracting the visual scene embedding and the scene-aware audio embedding corresponding to the current scene through the pre-trained contrastive audio-visual mask autoencoder, comprising: extracting a visual scene embedding corresponding to the current scene through a video encoder in the pre-trained contrastive audio-visual mask autoencoder; A scene-aware audio embedding corresponding to the current scene is extracted by an audio encoder in the pre-trained contrastive audio-visual mask autoencoder.
4. A scene-aware audio-visual speech enhancement method as claimed in claim 1, characterized in that: The visual scene embedding, the scene-aware audio embedding and the acoustic spectrum embedding are concatenated to obtain the fused feature representation.
5. A scene-aware audio-visual speech enhancement method according to any one of claims 1 to 4, characterized in that: The ConMamba module is obtained through model training, and the model training includes: Set up a dataset of noise and corresponding visual scenes and a clean speech dataset; Dynamically mix the clean speech segments of the clean speech data set and the noise segments of the noise and corresponding visual scene data sets, and generate noisy mixed segments in real time with a signal-to-noise ratio randomly sampled between -10 and 20 dB; The ConMamba module is trained based on the noisy mixed segment.
6. A scene-aware audio-visual speech enhancement method as claimed in claim 5, characterized in that: When the model is trained, it also includes: the ConMamba module uses 10 noisy mixed fragments to perform a gradient update.
7. A computer device comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the scene-aware audiovisual speech enhancement method described in any one of claims 1 to 6 is implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the scene-aware audiovisual speech enhancement method described in any one of claims 1 to 6 is implemented.
9. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the scene-aware audiovisual speech enhancement method described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Specific word voice processing method and device
CN111613211A
Audio-visual speech enhancement method and system capable of fully utilizing vision and speech connection
CN113470671A
Method and apparatus for speech enhancement using deep learning model in inverse frequency domain
CN117789742A
System and Method for Selective Enhancement Of Speech Signals
US20120134522A1
Audio signal processing method, audio signal processing apparatus, computer device and storage medium
US20240096332A1
Cited By
Unmanned vehicle robust position identification method based on look-around image
CN119131740A
Speech enhancement method and system based on selective state space model Mamba
CN119993176A
Speech enhancement method for low-signal-to-noise-ratio audio signal
CN120164480A
WebRTC speech enhancement system and method based on multi-modal large model
CN121011196A
Speech enhancement method and system based on selective state-space model time-frequency multi-directional scanning
CN122799878A
