A speech separation method and system based on orthogonal decoupling of audiovisual consistency and complementarity.

Abstract

This invention belongs to the field of speech separation technology and provides a speech separation method and system based on orthogonal decoupling of audiovisual consistency and complementarity. It acquires mixed audio signals and continuous lip-reading video segments as multimodal inputs. The multimodal inputs are orthogonalized by projecting the extracted audio and visual features across modalities, decomposing them into consistency features and complementarity features. The original features and consistency features are used as common inputs for the first-stage separation, where a bidirectional long short-term memory network layer is used to model the temporal dependence of articulation movements, supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The output of the first-stage separation and the complementarity features are used as common inputs for the second-stage separation, where a correction mask estimated from the complementarity features is used to optimize the output of the first-stage separation, yielding the final separation result. This invention improves detection performance in complex acoustic scenarios.

Description

Technical Field

[0001] This invention belongs to the field of speech separation technology, specifically relating to a speech separation method and system based on orthogonal decoupling of audiovisual consistency and complementarity. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The human auditory system exhibits the "cocktail party effect" in noisy environments, referring to its ability to extract target speech from complex acoustic scenes and highlighting its selective attention to noise. This ability enables humans to clearly perceive and understand specific sounds against a noisy backdrop. Subsequent research has shown that this ability stems from the brain's utilization of visual information, thereby enhancing auditory processing to solve the "cocktail party problem." In this process, the brain combines the acoustic features of speech signals with visual cues such as lip movements and facial expressions, achieving effective sound source separation. Based on this neurobiological insight, researchers can construct a more noise-robust speech separation system by combining the time-frequency features of speech signals with visual-motor information from the speech organs. Furthermore, this technological framework has been extended to applications such as hearing aid development and augmented reality interaction.

[0004] Based on this neuroscience foundation, deep learning-based speech separation has evolved from unimodal to multimodal. Early research mainly focused on audio-only speech separation (AOSS), and significant progress was made through techniques such as deep clustering, temporal separation, and time-frequency masking optimization.

[0005] However, the separation quality of single-modal systems deteriorates significantly under noisy conditions because distortion of acoustic features hinders accurate resolution of speech components. To overcome this limitation, researchers introduced audiovisual-speech separation (AVSS), which utilizes noise-invariant visual articulation cues (such as lip trajectories) to compensate for impaired audio features. By combining visual spatiotemporal information with acoustic time-frequency correlations, these multimodal systems achieve enhanced robustness.

[0006] The core challenge of multimodal speech separation lies in designing effective cross-modal collaboration mechanisms to fully exploit the intrinsic connections between visual and auditory modalities. An ideal audiovisual separation system must effectively model two fundamental feature attributes: consistency and complementarity. Consistency features reflect shared information and temporal synchronization patterns, such as the overall alignment of lip movements with the speech acoustic envelope, providing a stable temporal reference for multimodal fusion. Complementarity features, on the other hand, contain unique and non-redundant information that can compensate for the deficiencies of a single modality. For example, the stark visual contrast between / f / and / w / can effectively eliminate speech ambiguity. Such visual information is crucial for resolving speech ambiguity when the acoustic signal is subjected to severe background noise interference. Therefore, leveraging consistency for robust alignment and complementarity for precise semantic optimization constitutes the theoretical foundation for achieving efficient speech separation.

[0007] Despite these requirements, existing integration strategies have failed to clearly balance these attributes. For example... Figure 1 As shown, early fusion, by prematurely mixing raw features, causes the dominant audio signal to mask subtle visual complementary cues. Conversely, late fusion preserves modal characteristics but processes individual streams independently, failing to capture the inherent consistency required for alignment. The most common is intermediate fusion, which attempts deep interaction but, due to a lack of explicit constraints, results in entanglement in the feature space, with shared and unique information indistinguishably mixed together. This entanglement causes two main problems: first, it introduces redundancy because shared information cannot be clearly separated from unique cues; second, it leads to mutual interference, especially when one modality is disturbed by noise. For example, distorted audio features degrade the representation of sharp visual features, thus impairing the overall robustness of the model. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes a speech separation method and system based on audiovisual consistency and complementarity orthogonal decoupling. This invention provides a two-stage speech separation framework. The first stage utilizes consistency features to improve time alignment, and its output serves as the initial input for the second stage. The second stage, guided by complementary features, enhances semantic integrity through error correction, enabling the reconstruction of weak speech components in noise, thereby improving performance in complex acoustic scenarios.

[0009] According to some embodiments, the present invention adopts the following technical solution: A speech separation method based on audiovisual consistency and complementarity orthogonal decoupling includes the following steps: Acquire mixed audio signals and continuous lip-reading video clips as multimodal input; The multimodal input is orthogonalized, and the extracted audio and visual features are projected across modalities to decompose them into consistency features and complementary features. The original features and consistency features are used as common inputs for the first stage of separation. This stage uses a bidirectional long short-term memory network layer to model the temporal dependence of the articulation motion and is supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The output of the first-stage separation and the complementary features are used as common inputs for the second-stage separation. In this stage, the output of the first-stage separation is optimized using the correction mask estimated from the complementary features, and the final separation result is obtained.

[0010] As an alternative implementation, before orthogonalizing the multimodal input, the mixed speech signal is processed by short-time Fourier transform to generate a complex-valued spectrogram. The complex-valued spectrogram retains key phase information by utilizing the imaginary part spectrum. The real and imaginary parts are connected before being processed by the input to the two-dimensional convolutional layer. Global layer normalization is applied along the feature dimension to output audio features. Visual features in consecutive lip-reading video clips are extracted using a pre-trained model, which includes a backbone network for extracting features from image frames and a classification subnetwork for word prediction. Temporally align the extracted visual and audio features.

[0011] As an alternative implementation, the process of cross-modal projection of the extracted audio features and visual features includes: projecting the audio features onto the visual subspace, performing inner product operations along the feature dimensions during the projection process to maintain the independence of the spatiotemporal structure; projecting the visual features onto the audio features to generate shared information and visual-specific features, generating consistent features using the projection, and extracting orthogonal residuals as complementary features.

[0012] As a further defined implementation, during the projection process, the standard deviation loss constraint is used to constrain the bidirectional projection coefficients. By minimizing the standard deviation of the coefficients, the amplitude fluctuations across time steps or frequency ranges are constrained to smooth the projection weight distribution.

[0013] As an alternative implementation, the process of using the original features and consistency features as common inputs for the first-stage separation includes: using a first speech separation network for the first-stage separation, wherein the first speech separation network includes an intra-frame spectrum module, a sub-band time module and a full-band self-attention module, which respectively process local spectrum patterns, time dependencies and global context information.

[0014] As a further defined implementation, the intra-frame spectrum module constructs a global spectrum representation within each frequency band and performs layer normalization. Then, it performs local frequency domain expansion through an expansion operation to obtain an expanded tensor after flattening the frequency dimension. The expanded tensor is input into a bidirectional long short-term memory network, and then the frequency bin dimension is reconstructed using one-dimensional transpose convolution.

[0015] As a further defined implementation, the sub-band timing module is used to capture the time dependence within the frequency band, and sequentially expands its input for operation, bidirectional long short-term memory network and deconvolution processing to recover the frequency band dimension and establish residual connections with the input.

[0016] As a further defined implementation, the full-band self-attention module adopts multi-head self-attention, which generates query vectors, key vectors and value vectors through convolutional layers. The concatenated multi-head outputs are fused through a convolutional layer and then combined with the module input through residual connections. The resulting output is deconvolutionally processed to generate a multi-channel complex spectrum. The time-domain separated signal is reconstructed through inverse short-time Fourier transform.

[0017] As an alternative implementation, the process of performing second-stage separation by using the output and complementary features of the first-stage separation as common inputs includes performing second-stage separation using a second speech separation network, the structure of which is the same as that of the first speech separation network.

[0018] As an alternative implementation, both the first-stage separation and the second-stage separation use scale-invariant signal-to-noise ratio as a constraint to compare the extracted speech with the target speech.

[0019] A speech separation system based on audiovisual consistency and complementarity orthogonal decoupling includes: The multimodal acquisition module is configured to acquire mixed audio signals and continuous lip-reading video clips as multimodal inputs; The cross-modal projection module is configured to orthogonalize the multimodal input, perform cross-modal projection on the extracted audio and visual features, and decompose them into consistency features and complementary features. The first-stage separation module is configured to take the original features and consistency features as common inputs for the first-stage separation. This stage uses a bidirectional long short-term memory network layer to model the temporal dependence of the articulation motion and is supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The second-stage separation module is configured to take the output of the first-stage separation and the complementary features as common inputs to perform the second-stage separation. In this stage, the output of the first-stage separation is optimized using the correction mask estimated by the complementary features to obtain the final separation result.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes an innovative multimodal interaction mechanism based on orthogonal subspace decoupling. Unlike traditional fusion methods (which fuse information in a mixed feature space), this invention applies the principle of orthogonality to separate cross-modal features into two distinct components: consistent features (synchronicity) and complementary features (unique cues). This method effectively solves the problem of intermodal redundancy, thereby creating a more robust and complete feature representation.

[0021] This invention designs a novel two-stage speech separation framework that utilizes decoupling features. The first stage uses consistency features to improve time alignment, and its output serves as the initial input for the second stage. The second stage, guided by complementary features, enhances semantic integrity through error correction. This design enables the network to reconstruct weak speech components in noise, thereby improving performance in complex acoustic scenarios.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This diagram illustrates the limitations of existing multimodal fusion strategies. (a) represents early fusion, where the original audio and visual features are directly mixed in the initial stage of feature extraction. Since audio signals typically dominate in energy and dimensionality, visual complementary cues are masked, and audio features become dominant. (b) represents mid-stage fusion, which attempts to capture intermodal relationships through deep interaction in the middle layers of the model. Consistency and complementarity are inextricably mixed together and cannot be distinguished. When one modality is disturbed by noise, this entanglement reduces the representation quality of the other clear modality, thereby impairing the overall robustness of the system. (c) represents late-stage fusion, which treats the audio and video streams as independent streams and combines them only at the decision layer. Although this approach preserves the characteristics of each modality, the lack of deep interaction between the streams prevents the model from capturing the inherent consistency required for alignment. Figure 2 This is a schematic diagram of a two-stage separation architecture provided in one embodiment; Figure 3 This is a geometric schematic diagram of the feature orthogonalization process provided in one embodiment. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0028] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0029] Example 1 A speech separation method based on audiovisual consistency and complementarity orthogonal decoupling includes the following steps: Acquire mixed audio signals and continuous lip-reading video clips as multimodal input; The multimodal input is orthogonalized, and the extracted audio and visual features are projected across modalities to decompose them into consistency features and complementary features. The original features and consistency features are used as common inputs for the first stage of separation. This stage uses a bidirectional long short-term memory network layer to model the temporal dependence of the articulation motion and is supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The output of the first-stage separation and the complementary features are used as common inputs for the second-stage separation. In this stage, the output of the first-stage separation is optimized using the correction mask estimated from the complementary features, and the final separation result is obtained.

[0030] This embodiment utilizes a model architecture, referred to here as the SCCNet architecture. This architecture consists of a two-stage cascaded framework for audio / video / speech separation. Consistency features are input into the first-stage separator. Subsequently, the second stage optimizes the output by fusing complementary features (Fcomp), which originate from the orthogonal decoupling module.

[0031] like Figure 2 As shown, this end-to-end frame receives mixed audio signals. and continuous lip-reading video clips As a multimodal input, it generates clean speech output through orthogonal projection and a two-stage separation network.

[0032] This model employs a "cascaded separation strategy guided by orthogonal feature decomposition." First, the mixed audio input undergoes Schmitt orthogonalization, decomposing the features into "audiovisual consistency components" and "modality-specific complementary components." This orthogonal projection, while maintaining information integrity, enforces subspace independence. Second, the original audiovisual features and consistency features are jointly input into the first-stage separation network. This stage utilizes a bidirectional Long Short-Term Memory (LSTM) layer to model the temporal dependence of phonological motion, supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. Finally, a cascaded refinement structure is implemented, where the separation output of the first stage and the positive interactive complementary features serve as input to the second-stage separation network. This two-stage collaborative design progressively optimizes the separation quality, with complementary features providing discriminative, fine-grained cues to address residual speaker ambiguity.

[0033] Given a video containing N speakers, the mixed audio signal x(t) can be represented as N independent speech sources. Additive noise Linear combination: ; Where T represents the audio duration, It includes environmental noise and meaningless background sounds. The key is the assumption that the speech mixing result in the video is a linear superposition of the sound source signals. Based on the above formula, the multimodal speech separation model utilizes visual cues from speaker i. To estimate the target speaker's speech from x(t), where H, W, and T v These represent the height, width, and number of video frames, respectively.

[0034] Orthogonal projection The Gram-Schmidt orthogonalization process mathematically guarantees the orthogonality of decoupling features. Specifically, consistency features... and complementary characteristics The condition that their inner product is zero indicates that the orthogonality minimizes the information redundancy between the two feature spaces.

[0035] From a mathematical perspective, this orthogonality minimizes information redundancy: This represents the projected component in the shared subspace, while This corresponds to orthogonal residuals containing unique information. Unlike this inherent decoupling, traditional multimodal fusion methods (such as splicing or attention mechanisms) typically learn audio features F... a and visual features F vThe implicit correlation is pursued using the parameterized projection matrix W between them. The key is that, due to... and After decoupling, they occupy mutually exclusive orthogonal subspaces, and the explicit relationships between them eliminate the need for learning implicit correlations. Forced fusion can lead to cascading problems: splicing can cause dimensionality explosion, doubling the feature size and computational load; attention mechanisms often overfit parameters to noise rather than effective correlations; and linear combination directly destroys the purity of orthogonal features by mixing disjoint information flows.

[0036] To address these challenges, this embodiment proposes Bidirectional Gram-Schmidt Orthogonalization (BGSO)—a geometrically constrained feature decoupling framework. Its core innovation lies in eliminating modal bias through a symmetric projection mechanism while ensuring complete and lossless decoupling. Specifically, BGSO achieves structured feature separation through bidirectional orthogonality constraints. This section first formalizes the mathematical foundation of BGSO based on Gram-Schmidt process extensions, then details its implementation as a differentiable operation in deep architectures, and finally elucidates its end-to-end workflow and integration with multimodal model pipelines.

[0037] The core concept of BGSO multimodal feature decoupling involves decomposing feature vectors from two modalities into a shared subspace (consistent features) and orthogonal complementary subspaces (complementary features), achieved through geometric projection. Crucially, unlike traditional basis construction methods, BGSO directly implements bidirectional orthogonal projection onto the original multimodal feature vectors / matrices. Mathematically, classical Gram-Schmidt orthogonalization constructs orthogonal bases by iteratively eliminating linear dependencies; for example, generating orthogonal bases e1 and e2 from linearly independent vectors u1 and u2 through sequential projection. BGSO, however, geometrically extends this process by simultaneously performing bidirectional projection operators. This fundamentally changes the subspace separation mechanism, transforming it from sequential to parallel cross-modal decoupling.

[0038] ; Second basis vector: ; The orthogonal basis constructed through this process satisfies the following conditions: This enforces strict pairwise orthogonality. In multimodal decoupling scenarios, this principle is applied to audio features Fa and visual features Fv. BGSO uses a bidirectional projection operator to map Fa to the feature space of Fv (and vice versa). This projection generates consistent features ( ), used to capture cross-modal shared patterns, while extracting orthogonal residuals as complementary features ( These features preserve unique, modal-specific information.

[0039] From a deep learning implementation perspective, the projection processing in bidirectional BGSO involves two structured inputs: a mixed speech signal processed by a short-time Fourier transform (STFT) to generate a complex-valued spectrogram. The dimensions of this spectrogram are determined by... The spectrum consists of the real and imaginary parts of the FFT, where B represents the batch size, T represents the number of time frames, and F represents the frequency, defined as half the length of the FFT plus one. These spectrograms preserve key phase information by utilizing the imaginary part spectrum. The real and imaginary parts are concatenated before being processed by the input into a 2D convolutional layer with 3×3 kernels. Subsequently, global layer normalization (GLN) is applied along the feature dimension D. Finally, the output is a spectrum of dimension D. The model extracts mixed speech features. Another input contains cropped lip features. The model consists of a backbone network for extracting features from image frames and a classification subnetwork for word prediction. The backbone network contains a 3D convolutional layer and a standard ResNet-18 architecture. Due to the difference in visual and audio sampling rates (visual data contains 50 frames, audio sampling rate is 16kHz), temporal alignment between visual and audio features requires interpolation of the extracted visual features. This process generates visual features that match the dimensions of the audio features. Subsequently, data from both modalities are fed into a projection module.

[0040] The implementation process is divided into three key stages.

[0041] Cross-modal projection coefficient calculation. To achieve speech separation and combine bidirectional interaction between audio and visual modalities, this paper uses an orange tilted cube to represent the audiovisual feature space, with audio features as an example. The light green plane corresponds to the visual subspace.

[0042] like Figure 3 As shown, the blue plane orthogonal to it represents an audio-specific subspace independent of visual information, which preserves the unique characteristics of vocalization. The magenta dashed rectangle represents the projection of audio features onto the visual subspace (producing consistent features), representing the visually relevant components of the audio features. The blue rectangle represents the unique parts of the audio features, defining the projection coefficient matrix from audio to vision. Directly applying the classic Gram-Schmidt method for unidirectional projection introduces modal bias. Therefore, a bidirectional projection method is adopted. First, the audio features are projected onto the visual subspace, as shown below: ; ; ; The projection coefficient controls the shared information between the audio and visual modalities in the projection components, while the residual components represent audio-specific features. To prevent division by zero errors during projection calculations, this embodiment adds a small constant. =10 -8 To ensure numerical stability, this embodiment performs inner product operations along the feature dimension during projection to maintain the independence of the spatiotemporal structure. Similarly, projecting visual features onto audio features generates shared information and visually specific features. Next, this embodiment will describe how to utilize audiovisual consistency and complementary features. Projection coefficients adjust the projection intensity, but abnormal fluctuations can lead to unstable feature decoupling, especially in the presence of noise or occlusion, thus reducing the projection effect. To address this issue, this embodiment designs a standard deviation loss function, detailed in the loss function section.

[0043] Two-stage separation network The input to the first-stage separation network includes raw mixed speech features, visual features, and audiovisual consistency features obtained through projection. Similarly, research shows that joint representations learned through multimodal autoencoders in the early fusion stage can improve the noise resistance of audiovisual speech recognition (AVSR). This means that such audiovisual consistency features can support stable global alignment during the first-stage processing. The speech separation module adopts the TF-GridNet architecture and utilizes a multi-path block stacking structure to achieve effective separation. Core components include an intra-frame spectral module, a sub-band temporal module, and a full-band self-attention module. These modules handle local spectral patterns, temporal dependencies, and global contextual information, respectively.

[0044] In the intra-frame spectrum module, a global spectrum representation is constructed within each frequency band. The input tensor of the i-th TF-GridNet module... These are considered as T independent sequences of length F. These sequences are first normalized at each layer, and then locally expanded in the frequency domain through an expansion operation, as detailed below: ; A convolution operation with kernel size I and stride J stacks adjacent embeddings along the frequency axis. This process flattens the frequency dimension to obtain a unfolded tensor. This unfolded tensor is then input into a bidirectional long short-term memory network, where D represents the channel dimension of the feature embeddings. Next, a one-dimensional transposed convolution (Deconv1D) reconstructs the frequency box dimension. In this step, residual connections are established with the input.

[0045] ; The sub-band temporal module captures temporal dependencies within the frequency band. It shares a similar computational workflow with the intra-frame spectral module. In this workflow, the input sequentially passes through an unrolling operation, a bidirectional long short-term memory network, and deconvolution. This process restores the frequency band dimension and establishes residual connections with the input. The full-band self-attention module employs multi-head self-attention (MHSA) to address the challenge of modeling global interdependencies. It generates query vectors through 1×1 convolutions. Key vector Sum value vector Among them, N h This indicates the number of attention heads.

[0046] ; The cascaded multi-head outputs are fused through a 1x1 convolutional layer. Then, they are combined with the module inputs via residual connections.

[0047] ; There are P stacked TF-GridNet modules. Within each module, a local module provides fine-grained time-frequency features, while an attention module supplements long-range dependencies, achieving joint "local-global" optimization. Finally, the output of the last module is processed by 3×3 deconvolution to generate a complex spectrum with 2N channels, where N represents the number of speakers. The time-separated signals are then reconstructed using inverse short-time Fourier transform (iSTFT). The second-stage separation module in SCCNet shares the same network architecture as the first-stage separation module, employing the same TF-GridNet module structure to jointly model local time-frequency dependencies and global contextual relationships. However, their roles and input configurations differ. The first stage processes the raw mixed audio, visual features, and consistency features extracted via orthogonal projection, while the second stage focuses on refining the separation output using complementary features.

[0048] Specifically, the input to the second stage includes the complex spectrogram predicted by the first-stage decoder (as a coarse separation result) and audio-visual complementary features extracted from the orthogonal decomposition. These complementary features encode residual modality-specific information, providing crucial guidance for resolving residual interference and recovering missing speech components.

[0049] The fusion of the first-stage output with complementary cues is achieved through a residual refinement strategy. The second-stage separator uses the first-stage estimate as a baseline and applies a correction mask estimated from complementary features to progressively improve spectral resolution. This mechanism helps recover low-energy consonants or speech segments masked by overlapping speakers or background noise. This progressive refinement significantly improves separation quality, with clearer harmonic structures and less leakage observed in the final spectrogram. By structurally mirroring the first stage while shifting the focus to modality-specific enhancements, the second stage enables SCCNet to achieve coarse-to-fine separation and robust reconstruction even under visual occlusion or acoustic degradation conditions.

[0050] loss function This model uses scale-invariant signal-to-noise ratio (SI-SNR) as the loss function, which is the loss function between the estimated signal and the original signal. SI-SNR is a commonly used loss metric in source separation tasks. Both the first and second stages of processing use SI-SNR as a constraint to compare the extracted speech with the target speech. The SI-SNR for each speaker is defined as follows.

[0051] ; ; here, and These represent the original audio and the estimated audio, respectively. Both signals have been normalized to have a mean of zero.

[0052] During projection, standard deviation loss is used to constrain the bidirectional projection coefficients. Significant fluctuations in these coefficients can disrupt the stability of feature decoupling and affect model convergence. In complex scenarios such as noise or occlusion, anomalous input features can cause outliers in the coefficients. For example, c might suddenly spike to its maximum value, causing orthogonal components to fail. By minimizing the standard deviation of the coefficients, amplitude fluctuations across time steps or frequency ranges are constrained, thus smoothing the projection weight distribution. Before computation, the last two dimensions are flattened, and then the standard deviation of the last dimension is calculated, as shown below: ; ; Total loss function: ; Define the audio-to-video projection coefficient matrix Video-to-audio projection coefficient matrix Where B represents the batch size and N represents the total number of elements in the flattened coefficient tensor. This refers to the i-th element of the flattening coefficient vector. It is the mean. , and These are hyperparameters that balance the first-stage separation loss, the second-stage separation loss, and the contribution of standard deviation regularization.

[0053] Example 2 A speech separation system based on audiovisual consistency and complementarity orthogonal decoupling includes: The multimodal acquisition module is configured to acquire mixed audio signals and continuous lip-reading video clips as multimodal inputs; The cross-modal projection module is configured to orthogonalize the multimodal input, perform cross-modal projection on the extracted audio and visual features, and decompose them into consistency features and complementary features. The first-stage separation module is configured to take the original features and consistency features as common inputs for the first-stage separation. This stage uses a bidirectional long short-term memory network layer to model the temporal dependence of the articulation motion and is supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The second-stage separation module is configured to take the output of the first-stage separation and the complementary features as common inputs to perform the second-stage separation. In this stage, the output of the first-stage separation is optimized using the correction mask estimated by the complementary features to obtain the final separation result.

[0054] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0055] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity, characterized in that, Includes the following steps: Acquire mixed audio signals and continuous lip-reading video clips as multimodal input; The multimodal input is orthogonalized, and the extracted audio and visual features are projected across modalities to decompose them into consistency features and complementary features. The original features and consistency features are used as common inputs for the first stage of separation. This stage uses a bidirectional long short-term memory network layer to model the temporal dependence of the articulation motion and is supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The output of the first-stage separation and the complementary features are used as common inputs for the second-stage separation. In this stage, the output of the first-stage separation is optimized using the correction mask estimated from the complementary features, and the final separation result is obtained.

2. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 1, characterized in that, Before orthogonalizing the multimodal input, the mixed speech signal is processed by short-time Fourier transform to generate a complex-valued spectrogram. The complex-valued spectrogram retains key phase information by utilizing the imaginary part spectrum. The real and imaginary parts are connected before being processed by the input two-dimensional convolutional layer. Global layer normalization is applied along the feature dimension to output audio features. Visual features in consecutive lip-reading video clips are extracted using a pre-trained model, which includes a backbone network for extracting features from image frames and a classification subnetwork for word prediction. Temporally align the extracted visual and audio features.

3. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 1, characterized in that, The process of cross-modal projection of the extracted audio and visual features includes: projecting the audio features onto the visual subspace, performing inner product operations along the feature dimensions during the projection process to maintain the independence of the spatiotemporal structure; projecting the visual features onto the audio features to generate shared information and visual-specific features, using the projection to generate consistent features, and extracting orthogonal residuals as complementary features.

4. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 3, characterized in that, During the projection process, the standard deviation loss constraint is used to constrain the bidirectional projection coefficients. By minimizing the standard deviation of the coefficients, the amplitude fluctuations across time steps or frequency ranges are constrained to smooth the projection weight distribution.

5. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 1, characterized in that, The first stage of separation is performed using a first speech separation network, which includes an intra-frame spectrum module, a sub-band temporal module, and a full-band self-attention module, which respectively process local spectrum patterns, temporal dependencies, and global contextual information. The second stage of separation is performed using a second speech separation network, the structure of which is the same as that of the first speech separation network.

6. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 5, characterized in that, The intra-frame spectrum module constructs a global spectrum representation within each frequency band and performs layer normalization. Then, it performs local frequency domain expansion through an expansion operation. After flattening the frequency dimension, it obtains an expanded tensor. The expanded tensor is input into a bidirectional long short-term memory network, and then the frequency bin dimension is reconstructed using a one-dimensional transpose convolution.

7. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 5, characterized in that, The subband timing module is used to capture the time dependence within the frequency band, and sequentially expands its input for operation, bidirectional long short-term memory network and deconvolution processing to restore the frequency band dimension and establish residual connections with the input.

8. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 5, characterized in that, The full-band self-attention module employs multi-head self-attention, generating query vectors, key vectors, and value vectors through convolutional layers. The concatenated multi-head outputs are fused through a convolutional layer and then combined with the module input through residual connections. The resulting output undergoes deconvolution processing to generate a multi-channel complex spectrum. The time-domain separated signal is reconstructed through inverse short-time Fourier transform.

9. The speech separation method based on orthogonal decoupling of audiovisual consistency and complementarity as described in claim 1, characterized in that, Both the first-stage and second-stage separations use scale-invariant signal-to-noise ratio as a constraint to compare the extracted speech with the target speech.

10. A speech separation system based on orthogonal decoupling of audiovisual consistency and complementarity, characterized in that, include: The multimodal acquisition module is configured to acquire mixed audio signals and continuous lip-reading video clips as multimodal inputs; The cross-modal projection module is configured to orthogonalize the multimodal input, perform cross-modal projection on the extracted audio and visual features, and decompose them into consistency features and complementary features. The first-stage separation module is configured to take the original features and consistency features as common inputs for the first-stage separation. This stage uses a bidirectional long short-term memory network layer to model the temporal dependence of the articulation motion and is supplemented by a multi-head attention mechanism to enhance semantic consistency in the temporal dimension. The second-stage separation module is configured to take the output of the first-stage separation and the complementary features as common inputs to perform the second-stage separation. In this stage, the output of the first-stage separation is optimized using the correction mask estimated by the complementary features to obtain the final separation result.

Images