Variable speaker audio-visual speech separation method and system based on multi-task joint

Abstract

The present application relates to the technical field of audio-visual speech separation, and provides a variable speaker audio-visual speech separation method and system based on multi-task combination, which comprises the following steps: acquiring video and mixed speech containing multiple speakers, converting the mixed speech into mixed audio features, and extracting visual features of all persons from video frames; based on the mixed audio features, a first stage separator estimates attractors for each speaker, and uses the estimated attractors to guide separation of the mixed speech to obtain audio components separated initially; an audio-visual matching network is used to match the audio components with the visual features of all persons, and the selected visual features are used to process the audio components in a second stage separator to obtain residuals; and after splicing the residuals with the unmatched audio components, time domain waveforms of the separated signals are reconstructed. While flexibility is retained in the scenario of unknown number of speakers, the robustness and precision of speech separation are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of audiovisual speech separation technology, and particularly relates to a variable speaker audiovisual speech separation method and system based on multi-task collaboration. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Single-channel speech separation is the task of estimating the original speech of one or more target speakers from a single-channel mixed speech signal. Although traditional pure audio speech separation methods have made significant progress, they still struggle to meet the needs of practical applications in situations with low signal-to-noise ratios or severe speaker overlap. In recent years, with the development of deep learning and multimodal learning, researchers have introduced visual modalities into speech separation tasks, utilizing the strong correlation between speaker facial and lip movements and speech content as auxiliary cues. This has led to a significant improvement in the separation performance of audiovisual speech separation techniques.

[0004] However, most mainstream audiovisual-speech separation methods are based on an ideal assumption: that the visual cues used are always clear, complete, and synchronized with the audio. But in real-world scenarios, there are often numerous instances of visual occlusion, blurring, dropped frames, and even situations where the speaker is completely outside the video frame. Existing audiovisual-speech separation methods often use visual modalities as guiding cues during training. When visual input is impaired or missing, unreliable guidance leads to a sharp decline in model performance, even falling below the limits of purely audio-based models that do not use visual information. This "visual dependence" severely limits the robustness of audiovisual-speech separation systems in uncontrolled acoustic environments.

[0005] Furthermore, most current mainstream audiovisual speech separation methods are trained in deterministic acoustic environments, meaning they rely on the prior assumption that the number of speakers is known. As mentioned earlier, most current audiovisual speech separation methods use visual signals as a deterministic reference to guide the speech separation process. This requires that the visual information received by the model must match the identities and numbers of speakers in the mixed speech. Therefore, most of these methods are designed to process mixed speech containing a fixed number of speakers (usually 2-3 people), with a fixed number of output channels. However, in real acoustic scenarios, the target sound source and interference signals are often dynamically changing, making it difficult to satisfy the assumption that the number of speakers is known in advance. Although some works have attempted to solve the multi-speaker separation problem when the number of speakers is unknown, these methods all adopt a pure audio speech separation framework because it is difficult to know in advance which visual signals should be used when the number of speakers is unknown. This sacrifices the robustness of existing methods while achieving flexibility. Summary of the Invention

[0006] To address the technical problems mentioned above, this invention provides a variable speaker audiovisual speech separation method and system based on multi-task collaboration. Through an innovative two-stage architecture design, after initially separating the audio vectors in the first separation stage, visual information and audio vectors are matched. In the second separation stage, the matched visual information is used to optimize the audio vectors, and then the separated signal is reconstructed together with the audio vectors that have not been matched with visual information. This method can still generate a sound source estimation for the speaker in the first separation stage even when the speaker in the mixed speech is outside the video frame. While retaining the flexibility in scenarios with an unknown number of speakers, it significantly improves the robustness and accuracy of speech separation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a variable speaker audiovisual speech separation method based on multi-task collaboration, comprising: Acquire video and mixed speech containing multiple speakers, convert the mixed speech into mixed audio features, and extract the visual features of all speakers from the video frames; Based on the mixed audio features, the first-stage separator estimates attractors for each speaker and uses the estimated attractors to guide the separation of mixed speech, obtaining the initially separated audio components; An audio-visual matching network is used to match audio components with the visual features of all people. The visual features that match the audio components are selected. The selected visual features are then used to process the audio components in the second-stage separator to obtain the residuals. The time-domain waveform of the separated signal is reconstructed by splicing the residual with the unmatched audio components.

[0008] Furthermore, the first-stage separator includes: after the mixed audio features are processed by layer normalization and linear layer, they are divided into several overlapping feature blocks. The audio features of the blocks enter the dual-path block. The audio features processed by the dual-path block are processed by the attractor network of the codec for attractor estimation. The attractor and the output of the dual-path block are multiplied element-wise after tensor broadcasting to form the input of the three-path block. Through the three-path block, the initially separated audio components are obtained. The second-stage separator is constructed based on a dual-path temporal speech separation method and uses cross-modal cross-attention for feature querying.

[0009] Furthermore, the visual features include facial features and lip movement sequences.

[0010] Furthermore, the step of reconstructing the time-domain waveform of the separated signal includes: splicing the residual with the unmatched audio components, the masking network estimating the time-domain mask for each speaker, and performing element-wise multiplication of the time-domain mask with the mixed audio features to obtain the separated audio features; and the audio decoder reconstructing the time-domain waveform of the separated signal in the time domain based on the separated audio features.

[0011] Furthermore, the speech separation model, which includes the first-stage separator, the audio-visual matching network, and the second-stage separator, is trained in three stages: the first-stage separator is trained using signal-to-noise ratio and attractor existence cross-entropy; the second-stage separator is trained using signal-to-noise ratio and cross-modal consistency constraints; and a hybrid reduction loss function is introduced for joint training.

[0012] A second aspect of the present invention provides a variable speaker audiovisual speech separation system based on multi-task collaboration, comprising: The feature extraction module is configured to: acquire video and mixed speech containing multiple speakers, convert the mixed speech into mixed audio features, and extract the visual features of all speakers from the video frames; The first-stage separation module is configured to: based on the mixed audio features, the first-stage separator estimates an attractor for each speaker and uses the estimated attractor to perform guided separation of the mixed speech to obtain the initially separated audio components; The second-stage separation module is configured to: use an audio-visual matching network to match the audio components with the visual features of all people, select the visual features that match the audio components, and use the selected visual features to process the audio components in the second-stage separator to obtain the residuals. The reconstruction module is configured to reconstruct the time-domain waveform of the separated signal by splicing the residual with the unmatched audio components.

[0013] Furthermore, the first-stage separator includes: after the mixed audio features are processed by layer normalization and linear layer, they are divided into several overlapping feature blocks. The audio features of the blocks enter the dual-path block. The audio features processed by the dual-path block are processed by the attractor network of the codec for attractor estimation. The attractor and the output of the dual-path block are multiplied element-wise after tensor broadcasting to form the input of the three-path block. Through the three-path block, the initially separated audio components are obtained. The second-stage separator is constructed based on a dual-path temporal speech separation method and uses cross-modal cross-attention for feature querying.

[0014] Furthermore, the visual features include facial features and lip movement sequences.

[0015] Furthermore, the reconstruction module is specifically configured to: after concatenating the residual with the unmatched audio components, a masking network estimates a temporal mask for each speaker, and the temporal mask is multiplied element-wise with the mixed audio features to obtain the separated audio features; the audio decoder reconstructs the temporal waveform of the separated signal in the temporal domain based on the separated audio features.

[0016] Furthermore, the speech separation model, which includes the first-stage separator, the audio-visual matching network, and the second-stage separator, is trained in three stages: the first-stage separator is trained using signal-to-noise ratio and attractor existence cross-entropy; the second-stage separator is trained using signal-to-noise ratio and cross-modal consistency constraints; and a hybrid reduction loss function is introduced for joint training.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention employs an innovative two-stage architecture design. In the first separation stage, after initially separating the audio vectors, visual information and audio vectors are matched. In the second separation stage, the matched visual information is used to optimize the audio vectors, which are then reconstructed together with the audio vectors that have not been matched with visual information. This allows for the generation of a speaker's sound source in the first separation stage, even when the speaker in the mixed speech is outside the video frame. While maintaining flexibility in scenarios with an unknown number of speakers, this invention significantly improves the robustness and accuracy of speech separation. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0019] Figure 1 This is a flowchart of the variable speaker audiovisual speech separation method based on multi-task collaboration according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the first-stage separator structure of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the dual-path block and triple-path block structures according to Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the LSTM-based EDA network structure in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the second-stage separator structure in Embodiment 1 of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] Example 1 This embodiment provides a variable speaker audiovisual speech separation method based on multi-task collaboration.

[0023] The variable speaker audiovisual speech separation method based on multi-task collaboration provided in this embodiment aims to achieve flexible audiovisual speech separation in scenarios with an unknown number of speakers. At the same time, it overcomes the inherent visual dependence problem of audiovisual speech separation methods to a certain extent, enabling the model to process off-screen speakers in audiovisual scenarios.

[0024] Given a video V containing M speakers, the corresponding single-channel mixed speech for this video is: : ; in, Representative from the The speaker's voice signal Represents noise signal, It represents the number of speakers in the mixed speech.

[0025] In the problem scenario of this embodiment, the speaker in the mixed speech It is unknown; the purpose of this embodiment is to... Under unknown circumstances, from mixed speech signals Estimate the speech signal of each speaker To simplify the problem to some extent, this invention assumes that most of the speakers in the mixed speech will appear in the video frame, and that no large number of new speakers will join during the entire dialogue.

[0026] The variable speaker audiovisual speech separation method based on multi-task collaboration provided in this embodiment adopts a speech separation model, including an audio encoder, a visual front end, a first-stage separator, an audiovisual matching network, a visual encoder, a second-stage separator, a masking network, and an audio decoder.

[0027] The variable speaker audiovisual speech separation method based on multi-task collaboration provided in this embodiment, such as... Figure 1 As shown, it includes the following steps: Step 1: Acquire video and mixed audio data containing multiple speakers (i.e., mixed speech), and crop facial images and lip movement sequences of all speakers from the video frames.

[0028] Step 2: The audio encoder converts the mixed speech into audio features, and the visual frontend converts all speaker facial images and lip movement sequences into speaker visual features for the second-stage separator.

[0029] In this embodiment, the audio encoder and audio decoder follow the classic configuration of existing audiovisual speech separation methods. In the existing technical framework, temporal audio encoding and decoding reconstruction are an inverse operation.

[0030] In the audio encoding stage, this embodiment uses a convolution kernel size of [size missing]. The convolution stride is One-dimensional convolutional layers (Conv1D) with ReLU (Rectified Linear Activation Function) from mixed speech Medium-coded mixed audio features The calculation process can be represented as follows: ; Where D represents the feature dimension of the audio feature and L represents the time dimension of the audio feature.

[0031] Step 3: The first-stage separator estimates the attractor for each speaker in the mixed speech. The estimated attractor vector is used to complete the initial guided separation, resulting in the initially separated audio components. At the same time, a preliminary estimate of the number of speakers in the mixed speech is obtained. The number of speakers will be added as a dimension to the initially separated audio components to facilitate subsequent feature processing.

[0032] like Figure 2 As shown, the first-stage separator adopts the model framework of the classic dual-path temporal speech separation method (Sepformer), and combines it with a method based on codec attractor estimation to achieve speaker number estimation.

[0033] (1) such as Figure 2 and Figure 3 As shown, the mixed audio features output by the audio encoder After entering the first-stage separator, it first goes through a Layer-Norm (layer normalization) layer and a linear layer, and then is divided into segments of size [size missing]. Overlapping feature blocks (chunks) with a 50% overlap rate between feature blocks; segmented audio features (3D audio features). Then it enters the Dual-Path Block, where The number of blocks, For block size, For feature dimensions.

[0034] The dual-path block integrates two Transformer encoder layers for processing intra-block and inter-block paths: the Intra-Chunk Transformer for processing intra-block paths and the Inter-Chunk Transformer for processing inter-block paths.

[0035] The Transformer encoder layer (abbreviated as intra-block Transformer) that processes intra-block paths includes encoder blocks repeated P times and layer normalization operations. Each encoder block consists of two LayerNorm (layer normalization) layers, a multi-head self-attention layer, and a feedforward neural network. Layer normalization standardizes the input features, then captures contextual dependencies through multi-head self-attention. The output of the multi-head attention layer is directly added to the input features, and then normalized again through layer normalization. The feedforward neural network performs independent nonlinear transformations on the features at each location to further extract features. The output of the feedforward neural network is added to the input of the second layer normalization block to complete the computation of one encoder block. Finally, layer normalization (outer layer normalization of the encoder block) globally normalizes the final outputs of N encoder blocks, serving as the output of the entire encoder. The role of LayerNorm is to standardize the input distribution of neurons to prevent gradient vanishing or exploding. The multi-head self-attention layer maps the input to Query, Key, and Value, calculating the relevance weights of the Query and Key, and applying these weights to the Value. Feedforward neural networks typically consist of two fully connected layers with a non-linear activation function (such as ReLU) sandwiched in between. Their main function is to perform non-linear transformation and dimensionality projection on the features extracted by the multi-head self-attention layer, thereby further enhancing the model's representational ability.

[0036] (2) The audio features processed by the dual-path block are then subjected to attractor estimation by the codec attractor (EDA) network.

[0037] The EDA network is specifically designed to estimate the number of sound sources in mixed audio and construct an attractor for each potential sound source, thereby achieving speech separation without needing to predetermine the number of sound sources. Following the configuration of the end-to-end speaker log method (EEND-EDA), this embodiment employs an encoder-decoder framework based on LSTM (Long Short-Term Memory) to estimate global attractors from the frame-level audio embeddings after sequence aggregation. The specific structure of the LSTM-based EDA network is as follows: Figure 4As shown, the process includes: First, the temporal audio feature sequence output by the dual-path block is randomly shuffled. This operation aims to eliminate the interference of the order in which sound sources appear on the time axis, prompting the model to capture global speaker identity features from the perspective of feature space distribution rather than temporal dependence. Then, the shuffled temporal audio feature sequence is input into an attractor encoder based on an LSTM architecture. Through recursive processing, the variable-length observed features are compressed into a fixed-dimensional hidden state vector. This vector serves as a global representation of the entire sound field, integrating prior information from all potential sound sources in the current mixed speech. Following this... The Attractor Decoder (LSTM) uses the hidden state as the initial state and iteratively decouples and extracts multiple attractor vectors in a high-dimensional space from the global representation through autoregression, forming an "identity template" that characterizes the features of different sound sources. Finally, the effectiveness of each generated attractor vector is scored by the attractor existence probability layer, and the actual speaker count is determined based on a threshold pre-configured according to the task characteristics. This transforms the complex variable separation task into a deterministic attractor matching process, greatly improving the robustness and task generalization ability in multi-dynamic speech scenarios.

[0038] During attractor estimation, the LSTM encoder... Update its hidden state according to the following formula With unit state : ;in, Indicates a time step. This represents the input vector.

[0039] In the initial stage of the iteration process, the hidden states and cell states of the LSTM encoder are initialized to zero vectors. The LSTM decoder starts from the zero vector input and decodes the global attractor of each potential speaker based on the final hidden states and cell states of the LSTM encoder. Where s represents the number of iterations, , , , , These represent the hidden state generated by the LSTM decoder at step s and the hidden state generated by the LSTM decoder at step s, respectively. The unit state of the step, the nonlinear mapping function of the LSTM decoder, and the LSTM decoder in the previous iteration step ( The hidden state of ) and the LSTM decoder in the previous iteration step ( The unit state of ).

[0040] Finally, the EDA network uses fully connected layers with an activation function (Sigmoid) to compute the probability of the existence of each global attractor: ;in, , These represent the linear layer and the attractor vector, respectively.

[0041] During the inference phase, following the general approach for classification tasks, the estimated probability of each attractor is compared with a predefined probability threshold. Compare; if Greater than If the probability threshold is 0.5, the attractor is considered to exist; otherwise, it is determined that the attractor does not exist. The probability threshold here is typically configured to 0.5 based on the task conventions of binary classification.

[0042] It should be noted that during the training phase, the LSTM decoder of the EDA network will target data containing... Mixed speech execution by individual speakers Decode the previous step. The first attractor is considered a valid attractor, while the second attractor is considered a valid attractor. Each attractor is considered a negative control sample, which helps the model distinguish whether the attractor actually exists during the inference phase.

[0043] (3) After the attractor estimation is completed, the effective attractor and the output of the two-path block are multiplied element-wise after tensor broadcasting. The speaker dimension is extended on the audio features of the output of the two-path block to obtain four-dimensional audio features. This forms the input for a Triple-Path Block, where... These represent the estimated number of speakers, the number of feature blocks, the length of the feature blocks, and the feature dimension, respectively.

[0044] (4) The three-path block extends the two-path block by adding a speaker-dimensional Transformer encoder layer (Inter-Speaker Transformer, i.e., speaker channel Transformer) to model the distinguishing relationship between speakers. Its structure is as follows: Figure 3 As shown: First, four-dimensional audio features... By reorganizing the feature dimensions, three-dimensional audio features are obtained. Transformer within the block Dimensionally Self-attention is computed, and then the input to the inter-block Transformer is obtained through dimension reorganization. Inter-block Transformer Dimensionally Self-attention is calculated, and then the input to the speaker channel Transformer is obtained through dimension reorganization. Transformer between speaker channels Dimensionally Calculate self-attention; the three-path block finally outputs an audio tensor that can be used for mask estimation. This tensor is then spliced ​​together in the time dimension after eliminating the overlapping parts of all audio features through overlapping addition operations to obtain the J-channel time-domain audio signal (L×D), that is, the initially separated audio components.

[0045] Step 4: The audio-visual matching network matches the results of the first-stage separator (the initially separated audio components) with visual information such as the speaker's facial features and lip movement sequences, and selects the visual information of the actual speaker in the scene.

[0046] The first-stage separator generates a rough separation result of all sound sources in the mixed speech. Based on this result, an audiovisual matching method is used to filter corresponding visual information from the facial features and lip movement images of all speakers acquired according to the separation result of the first stage.

[0047] The detailed structure of the audiovisual matching network is based on the following published literature: the paper "Disentangled speech embeddings using cross-modal self-supervision" published in the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) in 2020.

[0048] The audiovisual matching network filters corresponding visual information by matching visual and audio content. This filtering ability is acquired through contrastive learning. Specifically, an audio segment and its corresponding facial features and lip movement image sequence segments have content relevance and are considered a positive sample because they express the same content. On the other hand, the audio segment and other image sequence segments in the same video constitute negative samples because, although they come from the same speaker, their corresponding speech content is inconsistent.

[0049] The audiovisual matching network used in this embodiment employs cosine similarity to measure the similarity of a set of samples during training. ;in, These represent audio and visual features mapped to the same feature space, respectively.

[0050] After filtering by the audiovisual matching network, according to the problem assumptions of this embodiment, a portion of visual information that can match the sound sources in the mixed speech should be obtained. This information will be used in the second-stage separator to refine the output of the three-path block of the first-stage separator. Obviously, at the same time, there may be some sound sources without corresponding visual information, i.e., off-screen speakers. In this case, the tensor components corresponding to these off-screen speakers will be directly sent to the MaskNet masking network to estimate the temporal mask of these speakers.

[0051] Step 5: The matching results from the audiovisual matching module are sent to the visual front end. Based on the matching results, the visual front end and the visual encoder convert the matched speaker visual information into visual features. ,in Indicates the number of feature frames. The feature dimension is represented; visual features are used in the second-stage separator to refine the corresponding audio components and obtain residuals.

[0052] The second-stage separator is also built based on the classic two-path temporal speech separation method (Sepformer). However, in order to refine the separation results based on visual information, this embodiment adopts a cross-modal cross-attention approach to implement feature querying, and its structure is as follows: Figure 5 As shown. Specifically, the second-stage separator separates the audio from the coarse audio output of the first stage. The log-Mel spectrum is calculated and features are extracted using one-dimensional convolution. Simultaneously aligning audio and visual features in both temporal and feature dimensions; subsequently, the second-stage separator uses visual... As the dominant modality, with audio Cross-modal cross-attention is computed for conditional modality, and visual information is used as an information anchor to calibrate the audio information of the corresponding speaker. Finally, the audiovisual features fused by cross-modal cross-attention are passed through a multi-layer convolutional network (residual generator) to generate the residual between the coarsely separated audio in the first stage and the ideal reference audio, so as to repair and refine the coarsely separated output.

[0053] Step 6: Among the sound sources estimated by the first-stage separator, the separated sound sources that do not match visual information (the audio components corresponding to the off-screen speaker) are spliced ​​with the output (i.e., the residual) of the second-stage separator in the speaker dimension of the three-dimensional features.

[0054] Step 7: The concatenated audio features are passed through a masking network (MaskNet) to estimate a temporal mask for each speaker. The mask is then multiplied element-wise with the original mixed audio features to obtain the separated audio features.

[0055] Step 8: The audio decoder reconstructs the separated audio features into a time-domain audio signal in the time domain.

[0056] After passing through the second-stage separator, all feature embeddings are fed together into a masking network to estimate the temporal mask for all speakers. By multiplying the temporal mask element-wise with the output of the audio encoder, the separated audio feature representation x can be obtained. Finally, the audio decoder reconstructs the temporal waveform of the separated signal in the temporal domain based on the separated audio feature representation.

[0057] In the audio decoding stage, the decoder reconstructs the temporal audio signal from the separated temporal audio features. In this stage, this embodiment uses a one-dimensional transposed convolutional layer (TransposeConv1D), whose convolutional kernel and stride configuration are the same as those of the audio encoder. This process can be represented as follows: ; in, Indicates the first Source estimation of a speaker This represents the audio characteristics after separation.

[0058] The speech separation model training in this embodiment is divided into three stages. Throughout the training process, a scale-invariant signal-to-noise ratio (SISNR) is used to constrain the separation process: During the training of the first-stage separator, it is trained independently and connected to an external audio encoder, masking network and decoder. Its ability to separate mixed speech from an unknown number of speakers is optimized separately. In this process, the permutation-invariant SISNR and attractor existence loss (i.e., the binary cross-entropy between the attractor existence probability and the actual existence label) are used to train the first-stage separator. In the second separation stage, the arrangement problem is naturally solved due to the introduction of visual information. The network is trained to directly refine the separation output of the corresponding sound source from the feature representation of the three-path output through visual refinement. Specifically, in this stage, the aforementioned SISNR loss and cross-modal consistency constraint are used to train the second-stage separator. The cross-modal consistency constraint is a contrastive learning loss that constrains the audio features refined by the second-stage separator to have consistency with the original visual features in terms of content. This constraint is achieved by calculating the cosine similarity between the output audio features after feature mapping and the original visual features. In the third stage, joint training is performed. In this stage, to ensure that the model fully optimizes all separate outputs, a hybrid reduction loss function is introduced on top of the aforementioned loss, namely: ;in, The estimated sound source represents the i-th speaker. This represents the original mixed speech; this loss function requires that the linear superposition of the separated sound sources approximates the original mixed speech, thereby better constraining the improvement of the overall separation performance.

[0059] The variable speaker audiovisual speech separation method based on multi-task collaboration provided in this embodiment, through an innovative two-stage architecture design, can still generate a sound source estimation for the speaker in the first separation stage even when the speaker in the mixed speech is outside the video frame. After the audio vector is initially separated in the first separation stage, an audiovisual matching network is introduced to adaptively match visual information and audio vectors. In the second separation stage, the audio vector output by the first separation stage is optimized using the matched visual information, and then the separated signal is reconstructed together with the audio vector that did not match visual information. While retaining the flexibility in scenarios with an unknown number of speakers, it significantly improves the robustness and accuracy of speech separation.

[0060] The variable speaker audiovisual speech separation method based on multi-task collaboration provided in this embodiment integrates audiovisual matching, multi-speaker separation, and mixed speech speaker counting into a unified multi-task collaborative learning framework, thereby achieving mutual promotion and information sharing among various tasks.

[0061] To meet the application and deployment requirements of speech separation technology, the variable speaker audiovisual speech separation method based on multi-task collaboration provided in this embodiment adopts a lightweight structural design, which minimizes the number of model parameters and computational resource requirements.

[0062] Example 2 The variable speaker audiovisual speech separation system based on multi-task collaboration provided in this embodiment includes: The feature extraction module is configured to: acquire video and mixed speech containing multiple speakers, convert the mixed speech into mixed audio features, and extract the visual features of all speakers from the video frames; The first-stage separation module is configured to: based on the mixed audio features, the first-stage separator estimates an attractor for each speaker and uses the estimated attractor to perform guided separation of the mixed speech to obtain the initially separated audio components; The second-stage separation module is configured to: use an audio-visual matching network to match the audio components with the visual features of all people, select the visual features that match the audio components, and use the selected visual features to process the audio components in the second-stage separator to obtain the residuals. The reconstruction module is configured to reconstruct the time-domain waveform of the separated signal by splicing the residual with the unmatched audio components.

[0063] Furthermore, the first-stage separator includes: after the mixed audio features are processed by layer normalization and linear layer, they are divided into several overlapping feature blocks. The audio features of the blocks enter the dual-path block. The audio features processed by the dual-path block are processed by the attractor network of the codec for attractor estimation. The attractor and the output of the dual-path block are multiplied element-wise after tensor broadcasting to form the input of the three-path block. Through the three-path block, the initially separated audio components are obtained. The second-stage separator is constructed based on a dual-path temporal speech separation method and uses cross-modal cross-attention for feature querying.

[0064] Furthermore, the visual features include facial features and lip movement sequences.

[0065] Furthermore, the reconstruction module is specifically configured to: after concatenating the residual with the unmatched audio components, a masking network estimates a temporal mask for each speaker, and the temporal mask is multiplied element-wise with the mixed audio features to obtain the separated audio features; the audio decoder reconstructs the temporal waveform of the separated signal in the temporal domain based on the separated audio features.

[0066] Furthermore, the speech separation model, which includes the first-stage separator, the audio-visual matching network, and the second-stage separator, is trained in three stages: the first-stage separator is trained using signal-to-noise ratio and attractor existence cross-entropy; the second-stage separator is trained using signal-to-noise ratio and cross-modal consistency constraints; and a hybrid reduction loss function is introduced for joint training.

[0067] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A variable speaker audiovisual speech separation method based on multi-task collaboration, characterized in that, include: Acquire video and mixed speech containing multiple speakers, convert the mixed speech into mixed audio features, and extract the visual features of all speakers from the video frames; Based on the mixed audio features, the first-stage separator estimates attractors for each speaker and uses the estimated attractors to guide the separation of mixed speech, obtaining the initially separated audio components; An audio-visual matching network is used to match audio components with the visual features of all participants, selecting the visual features that match the audio components. These selected visual features are then used to process the audio components in a second-stage separator to obtain residuals. The second-stage separator further separates the audio components from the coarse audio data obtained in the first stage. The log-Mel spectrum is calculated and features are extracted using one-dimensional convolution. Simultaneously, audio and visual features are aligned in both temporal and feature dimensions; Subsequently, the second-stage separator uses vision. As the dominant modality, with audio Cross-modal cross-attention is computed for conditional modality, and visual information is used as an information anchor to calibrate the audio information of the corresponding speaker. Finally, the audiovisual features fused by cross-modal cross-attention are passed through a multi-layer convolutional network to generate the residual between the coarsely separated audio in the first stage and the ideal reference audio. The time-domain waveform of the separated signal is reconstructed by splicing the residual with the unmatched audio components. The steps for reconstructing the time-domain waveform of the separated signal include: splicing the residual with the unmatched audio components, estimating the time-domain mask for each speaker using a masking network, and performing element-wise multiplication of the time-domain mask with the mixed audio features to obtain the separated audio features. The audio decoder reconstructs the time-domain waveform of the separated signal based on the separated audio characteristics in the time domain.

2. The variable speaker audiovisual speech separation method based on multi-task collaboration as described in claim 1, characterized in that, The first-stage separator includes: after the mixed audio features are processed by layer normalization and linear layer, they are divided into several overlapping feature blocks. The audio features of the blocks enter the dual-path block. The audio features processed by the dual-path block are processed by the attractor network of the codec for attractor estimation. The attractor and the output of the dual-path block are multiplied element-wise after tensor broadcasting to form the input of the three-path block. The audio components that are initially separated are obtained through the three-path block. The second-stage separator is constructed based on a dual-path temporal speech separation method and uses cross-modal cross-attention for feature querying.

3. The variable speaker audiovisual speech separation method based on multi-task collaboration as described in claim 1, characterized in that, The visual features include facial features and lip movement sequences.

4. The variable speaker audiovisual speech separation method based on multi-task collaboration as described in claim 1, characterized in that, The speech separation model, which includes a first-stage separator, an audio-visual matching network, and a second-stage separator, is trained in three stages: the first-stage separator is trained using signal-to-noise ratio (SNR) and attractor existence cross-entropy; the second-stage separator is trained using SNR and cross-modal consistency constraints; and a hybrid reduction loss function is introduced for joint training.

5. A variable speaker audiovisual speech separation system based on multi-task collaboration, characterized in that, include: The feature extraction module is configured to: acquire video and mixed speech containing multiple speakers, convert the mixed speech into mixed audio features, and extract the visual features of all speakers from the video frames; The first-stage separation module is configured to: based on the mixed audio features, the first-stage separator estimates an attractor for each speaker and uses the estimated attractor to perform guided separation of the mixed speech to obtain the initially separated audio components; The second-stage separation module is configured to: use an audio-visual matching network to match audio components with the visual features of all individuals, select visual features that match the audio components, and use the selected visual features to process the audio components in the second-stage separator to obtain residuals; the second-stage separator further separates the audio components from the coarse audio components obtained in the first stage. The log-Mel spectrum is calculated and features are extracted using one-dimensional convolution. Simultaneously, audio and visual features are aligned in both temporal and feature dimensions; Subsequently, the second-stage separator uses vision. As the dominant modality, with audio Cross-modal cross-attention is computed for conditional modality, and visual information is used as an information anchor to calibrate the audio information of the corresponding speaker. Finally, the audiovisual features fused by cross-modal cross-attention are passed through a multi-layer convolutional network to generate the residual between the coarsely separated audio in the first stage and the ideal reference audio. The reconstruction module is configured to reconstruct the time-domain waveform of the separated signal by splicing the residual with the unmatched audio components. The steps for reconstructing the time-domain waveform of the separated signal include: splicing the residual with the unmatched audio components, estimating the time-domain mask for each speaker using a masking network, and performing element-wise multiplication of the time-domain mask with the mixed audio features to obtain the separated audio features. The audio decoder reconstructs the time-domain waveform of the separated signal based on the separated audio characteristics in the time domain.

6. The variable speaker audiovisual speech separation system based on multi-task collaboration as described in claim 5, characterized in that, The first-stage separator includes: after the mixed audio features are processed by layer normalization and linear layer, they are divided into several overlapping feature blocks. The audio features of the blocks enter the dual-path block. The audio features processed by the dual-path block are processed by the attractor network of the codec for attractor estimation. The attractor and the output of the dual-path block are multiplied element-wise after tensor broadcasting to form the input of the three-path block. The audio components that are initially separated are obtained through the three-path block. The second-stage separator is constructed based on a dual-path temporal speech separation method and uses cross-modal cross-attention for feature querying.

7. The variable speaker audiovisual speech separation system based on multi-task collaboration as described in claim 5, characterized in that, The visual features include facial features and lip movement sequences.

8. The variable speaker audiovisual speech separation system based on multi-task collaboration as described in claim 5, characterized in that, The speech separation model, which includes a first-stage separator, an audio-visual matching network, and a second-stage separator, is trained in three stages: the first-stage separator is trained using signal-to-noise ratio (SNR) and attractor existence cross-entropy; the second-stage separator is trained using SNR and cross-modal consistency constraints; and a hybrid reduction loss function is introduced for joint training.

Images