An audio-visual speech separation method based on cross-modal multi-scale fusion
By employing a cross-modal, multi-scale fusion method, the problems of insufficient separation performance and high computational complexity in audiovisual speech separation technology under complex scenarios are solved, achieving efficient and real-time speech separation results, which are applicable to fields such as intelligent interaction and industrial inspection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2026-07-01
- Publication Date
- 2026-07-31
AI Technical Summary
Existing audiovisual speech separation technologies have insufficient separation performance in complex scenarios, high computational complexity, and are difficult to meet the practical application requirements of real-time and high precision. Furthermore, they lack effective utilization of speaker identity and semantic information.
A cross-modal, multi-scale fusion approach is adopted, which achieves feature interaction and speech separation of audio and visual features at their respective optimal temporal resolutions through multi-scale audio and visual feature extraction, cross-modal modulation, and global semantic guidance, combined with lightweight convolutional gating and iterative optimization.
It significantly improves the robustness and accuracy of separation in complex scenarios, reduces computational complexity, enhances the adaptability and flexibility of the model, meets real-time processing requirements, and achieves separation accuracy close to that of the original clean speech.
Smart Images
Figure CN122493878A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and audiovisual speech separation technology, and in particular to a cross-modal multi-scale fusion audiovisual speech separation method. Background Technology
[0002] Audiovisual speech separation technology, as a cutting-edge research direction in multimodal artificial intelligence, aims to achieve accurate separation of the target speaker's voice in mixed speech by fusing audio signals and visual information. Currently, the core challenge facing this technology lies in the severe imbalance between performance and efficiency, a fundamental problem that greatly limits its practical application value.
[0003] From a technical implementation perspective, existing solutions mainly fall into two categories: time-domain methods, renowned for their superior audio reconstruction quality and ability to provide high-fidelity output, suffer from large model parameters and high computational complexity, making them unsuitable for real-time applications; while time-frequency domain methods excel in computational efficiency, they have long lagged behind time-domain methods in key separation performance metrics. This technical bottleneck directly results in target speech extraction accuracy remaining at a limited level of 62%-68% in practical applications, especially in complex dialogue environments with three or more participants, far from meeting the real-time application standards required by high-precision fields such as intelligent manufacturing and smart services.
[0004] Traditional pure audio separation techniques based on statistical methods such as ICA and NMF rely solely on the statistical characteristics of audio signals. This results in poor separation performance in multi-speaker scenarios and a complete lack of the crucial ability to identify individuals. While commercial AI solutions widely employ deep learning, most remain confined to single-modal processing, exhibiting significant bottlenecks in separation accuracy within video scenarios where the target person is visible. Although top-tier international academic research models perform exceptionally well in controlled laboratory environments, their excessive computational resource requirements and poor real-time performance hinder their effective deployment in real-world applications.
[0005] Current advanced audiovisual-speech separation methods generally employ a symmetrical multi-scale fusion framework, a design approach with fundamental structural flaws. Audio and visual modalities inherently possess different temporal characteristics: audio features have a higher sampling rate and contain rapidly changing fine-grained patterns; while visual lip movements evolve relatively slowly, conveying more stable semantic cues. Existing symmetrical fusion designs force a perfect match between the two modalities' features in terms of temporal resolution, ignoring this inherent temporal asymmetry, leading to low information fusion efficiency.
[0006] Furthermore, existing methods overemphasize local temporal correspondence modeling, lacking effective mining and utilization of global semantic information at the sentence level, such as speaker identity and pronunciation patterns. This design flaw leads to insufficient semantic consistency of the model in complex scenarios, making it difficult to maintain stable and reliable separation performance over long time sequences. Summary of the Invention
[0007] To overcome the shortcomings of the prior art, this invention provides a cross-modal multi-scale fusion audiovisual speech separation method that effectively separates target speech and significantly improves the robustness and accuracy of separation in complex scenarios.
[0008] To achieve the above objectives, the present invention adopts the following technical solution, including: A cross-modal, multi-scale fusion method for audiovisual speech separation includes: S1, extract initial audio features from the audio waveform of the mixed speech, extract initial visual features from the video of the lip region of the target speaker, construct multi-scale audio features and multi-scale visual features respectively, and obtain audio cumulative features and visual cumulative features; S2, firstly, the audio accumulated features and visual accumulated features are subjected to time average pooling to obtain audio modulation signals and visual modulation signals, then cross-modal modulation is performed through convolutional gating to generate audio global features and visual global features, and then audio deep semantics and visual deep semantics are obtained through a feedforward network. S3 broadcasts deep audio and deep visual semantics to all scales, modulates features at each scale through gated residual connections, and generates modulated multi-scale audio and video features. S4, the modulated multi-scale audio features and multi-scale video features are fused to generate enhanced audio features and visual features; S5, return the enhanced audio and visual features to step S1 as the initial audio and visual features, and iterate through steps S1-S5. Second-rate; S6, via The enhanced audio features output after the next iteration are iteratively processed by the auditory module. Next, refined audio features are generated; S7 generates a speech mask based on the refined audio features, and masks the initial audio features to obtain the masked audio features. The masked audio features are then converted back into an audio waveform to obtain the target speech of the target speaker.
[0009] Preferably, in step S1, the audio encoder consists of a one-dimensional convolutional layer that converts the audio waveform into a frame-level embedding representation to obtain initial audio features; the video encoder uses a pre-trained lip-reading model to extract spatiotemporal embedding representations from the video frame sequence to obtain initial visual features. For the initial audio features, a hierarchical feature pyramid is constructed through multi-level temporal downsampling to extract multi-scale audio features. All shallow audio features are pooled to the deepest temporal scale and then summed to obtain the audio cumulative features. For the initial visual features, the original temporal resolution is maintained, and the receptive field is expanded through convolution to extract multi-scale visual features. All scale visual features are summed to obtain the visual cumulative features.
[0010] Preferably, in step S2, Audio modulation signal and visual modulation signals Cross-modal modulation is performed using lightweight convolutional gating to generate global features of the modulated audio. and visual global features : ; ; in, and It is a 1D convolutional layer. For the Sigmoid function, ⊙ represents element-wise multiplication; Modulated global features and The final deep audio semantics are generated through corresponding feedforward networks. and visual deep semantics : ; in, and These represent the audio feedforward network and the visual feedforward network, respectively.
[0011] Preferably, in step S3, the deep semantics of the audio are... and visual deep semantics Broadcast to all scales, and modulate the features at each scale through gated residual connections to generate modulated multi-scale audio and video features: ; ; in, and The modulated number of Audio and visual features at various scales; and It is a 1D convolutional layer. For the Sigmoid function, ⊙ represents element-wise multiplication; and The first of the multi-scale audio features and multi-scale visual features constructed in step S1 Audio and visual features at various scales.
[0012] Preferably, in step S4, Modulated multiscale audio features middle Upsample deeper audio features to The temporal resolution of the shallowest audio features is calculated, and then concatenated or added to the shallowest audio features along the channel dimension. Finally, these features are integrated through a convolutional layer to generate enhanced audio features. For scale number; Modulated multi-scale video features The features are added together, or the deepest video features are taken and then integrated through a convolutional layer to generate enhanced visual features.
[0013] Preferably, in step S6, the auditory module has a multi-layer network structure.
[0014] Preferably, in step S7, the refined audio features are used to generate a speech mask of the target speaker through a ReLU activation function. Then, the initial audio features are masked using the speech mask to obtain the masked audio features. Finally, the masked audio features are converted back into an audio waveform through an audio decoder to obtain the target speech of the target speaker.
[0015] Preferably, during model training, the loss function Scale-invariant signal-to-noise ratio (SI-SNR) is used as the optimization objective.
[0016] The present invention also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the aforementioned cross-modal multi-scale fusion audiovisual speech separation method.
[0017] The present invention also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned cross-modal multi-scale fusion audiovisual speech separation method.
[0018] The advantages of this invention are: (1) In the prior art, audiovisual speech separation is particularly affected by complex acoustic environments (such as multiple people speaking at the same time, strong background noise) and visual interference from the target person (such as low light, facial occlusion), resulting in low accuracy and easy distortion of the target speech extraction, which makes it difficult to meet the needs of practical applications. This invention combines an asymmetric multi-scale fusion framework with a global guided attention mechanism, enabling the system to accurately locate the target speaker in multimodal noise data, effectively separate the target speech, and significantly improve the robustness and accuracy of separation in complex scenarios.
[0019] (2) In traditional methods, symmetric multi-scale fusion requires forced matching of audio and visual features with temporal resolution, which not only increases the computational burden and prolongs the processing time, but also easily loses key spatiotemporal information of the modality. The asymmetric fusion design of this invention allows audio and visual streams to extract features and interact at their respective optimal temporal resolutions, simplifies redundant computation processes, and efficiently completes feature fusion and speech separation directly from the original multimodal data. It achieves efficient computation while maintaining a lightweight parameter of 3.3M, significantly improving the processing speed.
[0020] (3) Existing technologies have limited generalization ability when dealing with different speaker characteristics (such as timbre, speech rate, and accent) or unknown scenarios, and their performance drops sharply when faced with unseen speakers or extreme environments (such as multiple people overlapping or dynamic occlusion). This invention captures time-invariant semantic information such as speaker identity and pronunciation patterns through a global guided attention mechanism, and combines it with text prior transfer from large language models and action perception feature extraction to learn the essential feature distribution of speech signals. This allows it to better adapt to different individuals and complex, unseen scenarios, and significantly improves the universality and flexibility of the model.
[0021] (4) Traditional methods rely on local time correspondence modeling, which easily loses key amplitude, phase and semantic details, resulting in artificial traces or insufficient naturalness in the separated speech. This invention achieves more refined modeling of speech signals through fine multi-scale feature interaction and global semantic guidance, more accurately simulates and restores the real fluctuations and detailed features of physiological speech, and significantly improves the accuracy and naturalness of the separated speech (PESQ score reaches 3.28, close to the level of the original pure speech).
[0022] (5) This invention aims to provide a more accurate, robust, and efficient audiovisual speech separation solution, providing new technical support for fields such as intelligent interaction, industrial inspection, and security monitoring, especially in application scenarios requiring real-time and accurate speech extraction (such as intelligent conferencing, remote collaboration, and equipment inspection). These improvements help to promote speech separation technology from the laboratory to practical application, making it more suitable for everyday consumer electronics, professional industrial equipment, and clinical-grade interactive environments, laying a key foundation for building a more intelligent and natural human-computer interaction system. Attached Figure Description
[0023] Figure 1 This is a model architecture diagram of the present invention.
[0024] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation
[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] Faced with the limitations of existing technological systems, the field of audiovisual-speech separation urgently needs to achieve technological breakthroughs in the following areas: 1. Innovation of multimodal fusion mechanism A fusion mechanism that fully respects the inherent characteristics of different modalities is needed, especially to address the temporal asymmetry between audio and visual modalities. An ideal solution should allow different modalities to extract features and interact at their respective optimal temporal resolutions, rather than forcibly requiring temporal resolution matching.
[0027] 2. Utilization of global semantic information There is a need to develop new methods that can effectively capture and utilize global semantic information at the statement level. By introducing time-invariant semantic features such as speaker identity and pronunciation patterns, the semantic consistency and long-term stability of the model in complex scenarios can be significantly improved.
[0028] 3. Optimization of computational efficiency While ensuring separation performance, it is necessary to significantly reduce computational complexity through model architecture innovation and algorithm optimization, so that the technology can meet the strict real-time requirements of real-world application scenarios.
[0029] It is against this technological backdrop and driven by these demands that the cross-modal, multi-scale fusion audiovisual-speech separation model has emerged. This technology aims to solve key problems in existing technological systems through innovative multimodal fusion mechanisms and globally guided attention mechanisms.
[0030] The technical positioning of the cross-modal, multi-scale fusion audiovisual-speech separation model is not only to achieve performance breakthroughs, but more importantly, to establish a new technical paradigm that conforms to the essence of multimodal information processing. Its innovative value lies in fundamentally rethinking the fusion of audio and visual modalities, breaking through the limitations of traditional symmetrical design thinking, and opening up a new direction for the development of audiovisual-speech separation technology.
[0031] By introducing text prior knowledge transfer technology driven by a large language model, uncertainty quantification modeling methods, and cross-category general feature extraction strategies, the cross-modal multi-scale fusion audiovisual-speech separation model achieves a significant improvement in computational efficiency while maintaining high performance, laying a solid foundation for the practical application of the technology. This comprehensive technological innovation makes the cross-modal multi-scale fusion audiovisual-speech separation model a promising key breakthrough in driving audiovisual-speech separation technology from the laboratory to practical applications.
[0032] Depend on Figure 1 As shown, the cross-modal multi-scale fusion audiovisual speech separation model of the present invention includes: a multi-scale feature extraction module, a cross-modal feature fusion module, and a feature decoding module.
[0033] The multi-scale feature extraction module consists of two parts: an audio encoder and a video encoder. The audio encoder extracts features from the input mixed speech in real time, converting the audio waveforms of the mixed speech into frame-level embedding representations to obtain initial audio features; the video encoder extracts features from the target speaker's lip region video in real time, extracting spatiotemporal embedding representations of the lip region to obtain initial visual features; for the initial audio features, a hierarchical feature pyramid is constructed through multi-level temporal downsampling to extract multi-scale audio features, and cumulative audio features are obtained based on these multi-scale audio features; for the initial visual features, the original temporal resolution is maintained, and the receptive field is expanded through convolution to extract multi-scale visual features, and cumulative visual features are obtained based on these multi-scale visual features. This design fully respects the temporal differences between different modalities.
[0034] The cross-modal feature fusion module first uses temporal average pooling to accumulate features from each modality, generating a deep cross-modal modulated signal. Then, a lightweight convolutional gating mechanism enables cross-modal attention fusion, allowing the global descriptors of the two modalities to modulate each other, generating modulated global features. Next, a feedforward network (FFN) is used to generate a final deep semantic layer containing the fused visual and speech features. This deep semantic layer is then broadcast to all scales, and features at each scale are modulated using gated residual connections, generating modulated multi-scale audio and video features. Finally, the modulated multi-scale audio features are upsampled and fused to the finest temporal scale to generate enhanced audio features; the modulated multi-scale visual features are processed through a convolutional layer to generate enhanced visual features. This design ensures consistent semantic guidance for features at each scale, enhancing feature consistency.
[0035] Feature decoding module: The auditory module iteratively optimizes the enhanced audio features. Next, the audio features are further purified to generate refined audio features. Then, a speech mask is generated based on the refined audio features, and the initial audio features are masked using the speech mask to obtain the masked audio features. Finally, the masked audio features are converted back into an audio waveform through an audio decoder to obtain the target speech of the target speaker.
[0036] Depend on Figure 2 As shown, the present invention provides a cross-modal multi-scale fusion audiovisual speech separation method, which inputs mixed speech and video of the target speaker's lip region into an audiovisual speech separation model, and outputs the separated clean speech waveform of the target speaker. Specifically, it includes the following steps: S1, the audio waveform of the mixed speech Frame sequence of video of the lip area The inputs are then fed into the multi-scale feature extraction module.
[0037] An audio encoder consists of one-dimensional convolutional layers used to process audio waveforms. Convert to frame-level embedding representation to obtain initial audio features The video encoder employs a pre-trained lip-reading model (such as ResNet-18 combined with 3D convolution) to process the frame sequence. Extracting spatiotemporal embedding representations yields initial visual features. .in, The time step of the audio waveform; This is the time step after audio encoding downsampling, i.e., the time step of the initial audio features; The number of channels (dimensions) for the audio features; This represents the number of frames in the video of the lip region, which is also the time step of the initial visual features; The height and width of the video frame; The number of channels (dimensions) for visual features.
[0038] Based on initial audio features and initial visual features We constructed audio multi-scale representations and visual multi-scale representations respectively, and obtained audio cumulative features and visual cumulative features.
[0039] Constructing Multi-Scale Representations of Audio: Audio Features pass A series of cascaded one-dimensional convolutional layers with a stride of 2 are downsampled to form a depth of The temporal hierarchy structure yields audio feature sets at different time scales. ,in, For the first The audio features at the time scale, i.e., the _th_ time scale Layered audio features; For the first The number of channels in the layer audio features; After (i-1) downsampling steps with a step size of 2, the i-th... The time step of layer audio features.
[0040] All shallow audio features are pooled. ( After unifying to the deepest time scale, the features are summed to obtain the cumulative audio features. : ; in, This represents a pooling operation used to adapt shallow audio features to the deepest time scale.
[0041] Constructing visual multi-scale representations: Unlike audio paths, visual paths maintain the original temporal resolution. Unchanged. Visual features pass Each temporal refinement block (each block containing one-dimensional convolution, layer normalization, and PReLU activation function) is processed sequentially to obtain a multi-level visual feature set. ;in, For the first Level visual features; for The number of channels; for The length of the time dimension (time step).
[0042] By summing the visual features at all scales, we obtain the cumulative visual features. : .
[0043] S2, Audio Accumulation Features and visual cumulative features Input cross-modal feature fusion module.
[0044] To provide sentence-level global semantic guidance, audio accumulated features and visual cumulative features Time-averaged pooling is performed separately to obtain a deep audio modulation signal for cross-modal modulation. and visual modulation signals .
[0045] Modulation signals of two modes and Cross-modal modulation is performed using lightweight convolutional gating to generate global features of the modulated audio. and visual global features : ; ; in, and For 1D convolutional layers with different kernel sizes, This is the Sigmoid function, where ⊙ represents element-wise multiplication.
[0046] Modulated global features and The final deep audio semantics are generated through the corresponding feedforward networks (FFNs). and visual deep semantics : ; in, and These represent the audio feedforward network and the visual feedforward network, respectively.
[0047] S3 will generate deep semantics from the audio. and visual deep semantics Broadcast to all scales, and modulate the audio and visual features at each scale using gated residual connections to generate modulated multi-scale audio and video features: ; ; in, and The modulated number of Audio and visual features at various scales; and It is a 1D convolutional layer. This is the Sigmoid function, where ⊙ represents element-wise multiplication.
[0048] This allows features at each scale to be incorporated into global semantic information.
[0049] S4, modulated multi-scale audio features and multi-scale video features Multi-scale feature fusion is performed to generate enhanced audio features. and visual features .
[0050] Modulated multi-scale audio features By using a custom upsampling process, audio features from all scales are fused to the finest temporal scale to generate enhanced audio features. Specifically, this involves the deeper modulation features (those with lower temporal resolution). ( Upsampling (e.g., transposed convolution or interpolation) is performed to the shallowest layer (i.e., ...). (layer) temporal resolution , and then with The channels are concatenated or added together, and finally integrated through a convolutional layer to generate the final enhanced audio features of this iteration. .
[0051] Modulated multi-scale video features After processing through a convolutional layer, the enhanced visual features are obtained. Specifically, since the temporal resolution of visual features at all levels is the same, the modulated multi-level visual features can be... Directly add them, or take the last level feature. Then, a lightweight convolutional layer is used to integrate the data, generating the final enhanced visual features for this iteration. .
[0052] S5 will enhance audio features and visual features The multi-scale feature extraction module in step S1 is input and iteratively processed according to the processing method of steps S1-S5. Next, gradually optimize cross-modal alignment.
[0053] Steps S1-S5 together constitute an AV Fusion Block. This AV Fusion Block will be executed iteratively as a whole. Each iteration outputs the enhanced features of the previous AV fusion block. and It will be used as the input to the current AV fusion block, replacing the original. and The process involves re-extracting multi-scale features (step S1) and fusing cross-modal features (steps S2-S5) to gradually align and enhance cross-modal features.
[0054] S6, via After the iteration, the enhanced audio features output by the cross-modal feature fusion module Input features are decoded and iteratively processed by the auditory module (multi-layer network). Next, the audio features are further purified to generate refined audio features; the auditory module itself adopts an iterative expansion design and will be processed repeatedly. Next, for more detailed optimization, the output of this module is refined audio features, which will be used for subsequent mask generation and speech waveform reconstruction.
[0055] S7. The refined audio features are used to generate a speech mask for the target speaker through a ReLU activation function. Then use voice masking For initial audio features Perform masking to obtain the masked audio features. Finally, the masked audio features are processed by an audio decoder (one-dimensional transposed convolution). Convert back to audio waveform to obtain the target speaker's target speech. .
[0056] During model training, the loss function Scale-invariant signal-to-noise ratio (SI-SNR) is used as the optimization objective: ; in, It is the pure target voice. It is the estimated target speech. It is a scale-invariant signal-to-noise ratio function.
[0057] The processing procedure for the model inference phase is the same as that for the model training phase.
[0058] The method of this invention respects the inherent temporal differences between visual and audio modalities through a multi-scale framework, and provides consistent semantic guidance across scales by utilizing the deep semantics of the fused visual and speech, thereby achieving efficient and high-performance speech separation.
[0059] The technical effectiveness of the cross-modal multi-scale fusion audiovisual speech separation model of the present invention is analyzed as follows.
[0060] (1) Technical effects of lightweight iterative optimization structure a. Parameter efficiency optimization effect Through iterative processing in steps S1-S5, the AV fusion block, consisting of a multi-scale feature extraction module and a cross-modal feature fusion module, is repeatedly iterated. Second-rate( By sharing parameters, incremental feature optimization is achieved with only a 0.2M increase in parameters. This design keeps the total number of model parameters to 3.3M, which is about 70% less than similar models (such as av-sepformer), while maintaining excellent separation performance.
[0061] b. Improved convergence stability Gradient pruning (maximum L2-norm=5) and learning rate scheduling strategies (reducing by 50% after 20 epochs if no improvement is observed) during iterative optimization ensure training stability. On the LRS2 and VoxCeleb2 datasets, the model achieves stable convergence within 30 epochs, validating loss fluctuations within ±0.5dB.
[0062] c. Reasoning acceleration effect Lightweight auditory refinement module ( Through parameter sharing and simplified operations, approximately 3x speedup is achieved in the inference phase (with each iteration). With a batch size of 3, the single inference time is controlled within 200ms, meeting real-time processing requirements.
[0063] (2) Technical effects of end-to-end joint optimization a. Feature alignment optimization effect By using scale-invariant signal-to-noise ratio (SI-SNR) as the end-to-end optimization objective, the quality of the separated speech is directly optimized rather than intermediate feature matching. This design improves the model's robustness in complex noisy environments by approximately 42%, maintaining stable separation performance within an SNR range of -5dB to 5dB.
[0064] b. Multi-task collaboration effect The global guided attention mechanism simultaneously optimizes both intra-modal refinement and inter-modal fusion subtasks, achieving positive transfer between tasks through shared representation learning. Under challenging conditions such as occlusion and low illumination, the model maintains approximately 89% of its base performance through the cross-modal compensation mechanism.
[0065] (3) Technical effects of the multi-scale fusion framework a. Modal property adaptation effect Traditional symmetric fusion frameworks force audio and visual features to match temporal resolution, ignoring the fundamental difference between audio features having a higher sampling rate (containing rapidly changing fine-grained patterns) and visual lip movement evolving more slowly. This invention allows the audio stream to construct a multi-level temporal hierarchy while maintaining a fixed temporal resolution for the visual stream, enabling feature extraction and interaction across modalities at the optimal temporal scale. This design improves SI-SNRi by 16.5 dB on the LRS2 dataset, approximately 6.3% higher than symmetric methods.
[0066] b. Computational efficiency optimization effect By avoiding unnecessary visual flow temporal downsampling, computational overhead is reduced by approximately 40% while maintaining temporal fidelity. The visual path employs a hierarchical refinement strategy with fixed temporal resolution, expanding the receptive field through convolution without sacrificing temporal details, enabling the model to achieve real-time processing capabilities with minimal computational cost.
[0067] c. Cross-scale information aggregation effect The four-level temporal hierarchy (D=4) constructed by the audio path can simultaneously capture short-term acoustic dynamics and long-term prosodic patterns, while the multi-level representation of the visual path aggregates lip movement information at different scales through receptive field expansion. This multi-scale fusion achieves an SDRi of 14.7 dB in complex VoxCeleb2 scenes, demonstrating its effective ability to capture multi-scale temporal patterns.
[0068] (4) Technical effects of global attention guidance mechanism a. Semantic consistency enhancement effect Each modality is compressed into a compact global descriptor with a time dimension of 1 by temporal average pooling, capturing time-invariant semantic information such as speaker identity and pronunciation patterns. These global descriptors provide semantic guidance across scales, maintain separation consistency in long temporal sequences, and significantly improve the harmonic structure and spectral continuity of the output speech, achieving a PESQ score of 3.28.
[0069] b. Cross-modal interaction optimization effect A lightweight convolutional gating mechanism between global descriptors enables effective interaction of semantic information between modalities. The audio global descriptor enhances its ability to focus on the features of the target speaker by modulating visual information; the visual global descriptor enhances speech-related lip movement features by modulating audio information. This interaction improves the model's target speech locking accuracy by approximately 23% in scenarios with multiple speakers speaking simultaneously.
[0070] c. Multi-scale feature modulation effect The fused visual and speech deep semantics are broadcast to all scales, modulating feature representations at each scale through gated residual connections. This design ensures consistent semantic guidance from fine-grained frame-level features to coarse-grained sentence-level features, improving separation stability by approximately 35% in complex acoustic environments.
[0071] (5) The overall effect of the overall technical solution a. Breakthrough improvement in separation accuracy Through the synergistic effect of asymmetric multi-scale fusion and global semantic guidance, the cross-modal multi-scale fusion audiovisual-speech separation network achieves significant performance improvements in standard benchmark tests. It achieves 16.5 dB SI-SNRi and 16.64 dB SDRi on the LRS2 dataset, and 14.0 dB SI-SNRi and 14.7 dB SDRi on the VoxCeleb2 dataset, representing improvements of 0.5 dB and 0.4 dB respectively over the current state-of-the-art methods. Furthermore, it achieves a score of 3.28 in the PESQ speech quality evaluation, approaching the level of pristine speech.
[0072] b. Optimizing the balance between computational efficiency and performance With a lightweight design featuring 3.3M parameters and 19.8G MACs, significant optimization of computational complexity is achieved while maintaining top-tier separation performance. Experiments on an 8×RTX 4090 GPU show that model training time is reduced by approximately 45% compared to traditional methods, and inference speed meets real-time processing requirements, making deployment on edge devices feasible.
[0073] c. Significantly enhanced cross-scenario generalization capability Ablation experiments demonstrate that the globally guided attention mechanism enables the model to adapt well to unfamiliar speakers, accents, and acoustic environments. In cross-dataset testing, the model trained on LRS2 and tested on VoxCeleb2 showed a performance drop of only 1.2 dB, significantly better than the 2.5 dB drop of the comparison methods.
[0074] d. Improved usability and deployability Modular design supports flexible configuration, allowing the number of iterations to be adjusted according to application scenarios. and depth Balancing performance and efficiency. In real-world tests across scenarios such as smart conferencing, in-vehicle systems, and security monitoring, the system maintained approximately 86% of its baseline performance under complex acoustic environments, demonstrating its strong practical value.
[0075] This invention provides a novel solution for speech separation in complex scenarios through an innovative combination of asymmetric temporal modeling and global semantic guidance. It not only advances the development of multimodal fusion technology at the academic level but also demonstrates broad prospects in application fields such as intelligent interaction, remote communication, and assisted healthcare, laying a technological foundation for achieving a more natural and robust human-computer interaction experience.
[0076] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An audio-visual speech separation method of cross-modal multi-scale fusion, characterized in that, include: S1, extract initial audio features from the audio waveform of the mixed speech, extract initial visual features from the video of the lip region of the target speaker, construct multi-scale audio features and multi-scale visual features respectively, and obtain audio cumulative features and visual cumulative features; S2, firstly, the audio accumulated features and visual accumulated features are subjected to time average pooling to obtain audio modulation signals and visual modulation signals, then cross-modal modulation is performed through convolutional gating to generate audio global features and visual global features, and then audio deep semantics and visual deep semantics are obtained through a feedforward network. S3 broadcasts deep audio and deep visual semantics to all scales, modulates features at each scale through gated residual connections, and generates modulated multi-scale audio and video features. S4, the modulated multi-scale audio features and multi-scale video features are fused to generate enhanced audio features and visual features; S5, return the enhanced audio features and visual features to step S1 as initial audio features and initial visual features, iterate the processing according to steps S1-S5 seconds; S6, output the enhanced audio features after the secondary iteration via the auditory module The enhanced audio features output after the secondary iteration are iteratively processed via the auditory module The refined audio features are generated S7 generates a speech mask based on the refined audio features, and masks the initial audio features to obtain the masked audio features. The masked audio features are then converted back into an audio waveform to obtain the target speech of the target speaker.
2. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, In step S1, the audio encoder consists of a one-dimensional convolutional layer that converts the audio waveform into a frame-level embedding representation to obtain the initial audio features. The video encoder uses a pre-trained lip-reading model to extract spatiotemporal embedding representations from the video frame sequence to obtain initial visual features; For the initial audio features, a hierarchical feature pyramid is constructed through multi-level temporal downsampling to extract multi-scale audio features. All shallow audio features are pooled to the deepest temporal scale and then summed to obtain the audio cumulative features. For the initial visual features, the original temporal resolution is maintained, and the receptive field is expanded through convolution to extract multi-scale visual features. All scale visual features are summed to obtain the visual cumulative features.
3. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, In step S2, Audio modulation signal and visual modulation signals Cross-modal modulation is performed using lightweight convolutional gating to generate global features of the modulated audio. and visual global features : in, and It is a 1D convolutional layer. For the Sigmoid function, ⊙ represents element-wise multiplication; Modulated global features and The final deep audio semantics are generated through corresponding feedforward networks. and visual deep semantics : in, and These represent the audio feedforward network and the visual feedforward network, respectively.
4. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, In step S3, the deep semantics of the audio are... and visual deep semantics Broadcast to all scales, and modulate the features at each scale through gated residual connections to generate modulated multi-scale audio and video features: in, and The modulated number of Audio and visual features at various scales; and It is a 1D convolutional layer. For the Sigmoid function, ⊙ represents element-wise multiplication; and The first of the multi-scale audio features and multi-scale visual features constructed in step S1 Audio and visual features at various scales.
5. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, In step S4, Modulated multiscale audio features middle Upsample deeper audio features to The temporal resolution of the shallowest audio features is calculated, and then concatenated or added to the shallowest audio features along the channel dimension. Finally, these features are integrated through a convolutional layer to generate enhanced audio features. For scale number; Modulated multi-scale video features The features are added together, or the deepest video features are taken and then integrated through a convolutional layer to generate enhanced visual features.
6. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, In step S6, the auditory module has a multi-layer network structure.
7. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, In step S7, the refined audio features are used to generate a speech mask of the target speaker through a ReLU activation function. Then, the initial audio features are masked using the speech mask to obtain the masked audio features. Finally, the masked audio features are converted back into an audio waveform through an audio decoder to obtain the target speech of the target speaker.
8. The cross-modal multi-scale fusion audiovisual speech separation method according to claim 1, characterized in that, During model training, the loss function uses scale-invariant signal-to-noise ratio as the optimization objective.
9. A readable storage medium, characterized in that, It stores a computer program, which, when executed, implements the cross-modal multi-scale fusion audiovisual speech separation method as described in any one of claims 1 to 8.
10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-modal multi-scale fusion audiovisual speech separation method according to any one of claims 1 to 8.