Cross-modal speaker extraction method and system based on multiple consistency

Abstract

The invention provides a cross-modal speaker extraction method and system based on multiple consistency. The method comprises the following steps: acquiring a mixed voice signal containing target speaker data and video data; performing audio coding on the obtained mixed voice signal to obtain audio features; performing consistent visual feature extraction on the acquired video data to obtain two visual features consistent with the target voice in content and identity; fusing the visual features and the audio features which are consistent in content, and performing speaker extraction on the fused features in a cross-modal system embedded into Mama to obtain separated voices; according to the method, the visual features consistent in identity are used for carrying out identity constraint on the separated voice, a final extraction result is obtained, an extra text marking data set is not needed for pre-training, and information corresponding to the voice content in the visual clues and identity information of the speaker can be effectively utilized.

Description

Technical Field

[0001] The present invention belongs to the field of speech extraction, and in particular relates to a cross-modal speaker extraction method and system based on multi-consistency. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In noisy environments or environments with multiple people speaking, humans have the ability to distinguish speech. Speech separation technology can be used as a speech front-end technology to separate the speech of each speaker from the mixed speech. The separated speech can be applied to tasks such as speech recognition and speaker recognition. However, speech separation requires knowing the number of speakers in advance, which limits the application scenarios. In most cases, this embodiment only needs to pay attention to and listen to the speech of the target speaker at the same time, which is the extraction of the target speaker. Target speaker extraction refers to the extraction of the speech of a single target speaker in the presence of multiple speakers. It has a wide range of applications, such as digital hearing aids, speech monitoring, speech recognition, lyrics transcription, etc.

[0004] Target speaker extraction requires the use of additional information about the target speaker as clues, such as pre-recorded reference speech. Deep neural networks can learn specific features related to the target speaker from pre-recorded reference speech, such as pitch, intonation, identity information, etc. Therefore, it is possible to learn speaker embeddings similar to the target speaker by using pre-recorded reference speech of the target speaker and apply them to the target speaker extraction process. However, the need for pre-recorded reference speech limits its application scenarios. Figure 1 As shown in Figure 2, due to the lack of pre-recorded reference speech, the speech of the newly added target speaker cannot be extracted during the meeting. In addition, the human-computer interaction process cannot handle the newly added speaker.

[0005] Some researchers have proposed target speaker extraction methods that don't require pre-recording the target speaker's speech, but instead use the speaker's visual information as clues. For example, using visual cues typically relies on pre-trained speech recognition models to extract visual features, but this requires an additional, labeled speech-to-text dataset for pre-training, which is quite scarce. Furthermore, neither of these methods considers identity consistency between the target speech and visual features, nor does it consider consistent visual features with respect to content.

[0006] On the other hand, time domain modeling of long-sequence speech data requires processing speech sequences with a large number of time steps. Some existing models can extract good local features but have difficulty capturing contextual information; some can capture global features but easily forget details and cannot be parallelized; some support parallelization but increase costs and reduce inference speed. Summary of the Invention

[0007] In order to solve the above problems, the present invention proposes a cross-modal speaker extraction method and system based on multi-consistency. The present invention does not require additional text labeling datasets for pre-training and can effectively utilize identity information in visual cues.

[0008] According to some embodiments, the present invention adopts the following technical solutions: A cross-modal speaker extraction method based on multi-consistency includes the following steps: Obtaining mixed speech signals and video data containing target speaker data; Performing audio encoding on the obtained mixed speech signal to obtain audio features; Consistent visual features are extracted from the acquired video data to obtain two visual features that are consistent with the target speech in terms of content and identity. Fuse the visual features and audio features with consistent content, extract the speaker from the fused features, and obtain the separated speech; The separated speech is constrained using identity-consistent visual features to obtain the final extraction result.

[0009] As an optional implementation, the process of performing audio encoding on the obtained mixed speech signal includes performing audio encoding using an audio encoder, where the audio encoder includes a one-dimensional convolution layer and a linear activation layer to encode the one-dimensional audio signal into a two-dimensional audio feature.

[0010] As an optional implementation, the process of extracting consistent visual features from the acquired video data includes: extracting using a consistent visual feature extractor, the consistent visual feature extractor including an audio-visual matching network and stacked VTCNs, wherein the audio-visual matching network is pre-trained through a frame-level audio-visual matching task, including a face subnet and an audio subnet, each subnet including six convolutional layers, and the output of each subnet is processed by two independent one-dimensional convolutional layers, which are used to obtain identity features and content features, respectively.

[0011] As a further step, in the process of learning content consistency, positive and negative samples are used to annotate and learn video clips to form content consistency constraints to capture consistent content features from audio and visual information.

[0012] Furthermore, during the identity consistency learning process, voices and faces from the same speaker are considered as positive samples, and voices and faces from different speakers are considered as negative samples, forming an identity consistency constraint to capture consistent identity features from audio and visual information.

[0013] As an optional implementation, before fusing the visual features and audio features with consistent content, the visual features with consistent content are reduced in dimension by an MLP module, and then further extracted through multiple repeated VTCNs. In the VTCNs, BatchNorm is used for normalization to obtain more advanced visual features, which are then fused with the audio features.

[0014] As an optional implementation method, during the process of speaker extraction of the fused features, the audio features are layer-normalized and linearly processed, and then connected with visual features with consistent content to obtain fused audio-visual features. Features are obtained through a linear layer, and the obtained features are divided into multiple overlapping blocks, each with the same length. Intra-block and inter-block processing are then performed separately to obtain a mask of the target speaker, which is multiplied by the audio features to obtain the estimated speech signal of the target speaker.

[0015] The same network structure is used within and between blocks, including a linear layer, a one-dimensional convolutional layer, and a selective state space model. The selective state space model is bidirectional, and the processing between blocks is performed along the dimension of the number of blocks.

[0016] As an optional implementation, the identity constraint process includes: the estimated speech signal passes through an audio-visual matching network to obtain identity features, and the identity features and identity-consistent visual features are respectively passed through a fully connected layer for feature dimensionality reduction, the weights are not shared between fully connected layers, and multiple losses are used to enforce identity consistency.

[0017] Furthermore, the multiple losses adopt scale-invariant signal-to-noise ratio loss and identity consistency loss to constrain the speaker extraction process, and the goal is to minimize the scale-invariant signal-to-noise ratio loss and identity consistency loss, minimize the distance between the estimated speech and the target speech, and maximize the difference between the estimated speech and the interfering speech.

[0018] As an optional implementation, during the identity constraint process, content-consistent visual features are optimized through reverse gradient to form an inter-loop learning.

[0019] A multi-consistency-based cross-modal speaker extraction system, comprising: a data acquisition module configured to acquire a mixed speech signal and video data including target speaker data; an audio encoding module configured to perform audio encoding on the acquired mixed speech signal to obtain audio features; a consistent visual feature extraction module configured to extract consistent visual features from the acquired video data to obtain two visual features that are consistent with the target speech in terms of content and identity; A fusion extraction module is configured to fuse visual features and audio features with consistent content, perform speaker extraction on the fused features, and obtain separated speech; The identity constraint module is configured to use the identity-consistent visual features to perform identity constraints on the separated speech to obtain the final extraction result.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses a pre-trained audio-visual matching network to extract visual features that are consistent with the target speech in content and identity. During the speaker extraction process, visual features with consistent content are used for feature fusion, and then visual features with consistent identity are used to constrain the identity of the separated speech. During the identity constraint process, the extraction of visual features with consistent content can be continuously optimized. Better visual features with consistent content facilitate better extraction of the target speech. The more accurate the estimated speech, the better the visual features with consistent content will be. This reciprocal learning mechanism links identity consistency and content consistency, fully utilizing the advantages of multiple consistency.

[0021] The present invention uses a pre-trained audio-visual matching network to match faces and voices at the frame level to learn content features from audio-visual signals. It can learn more fine-grained content features without the need for additional text datasets. The mutual loop learning mechanism based on multiple consistency can better alleviate the semantic gap between different modes and obtain more fine-grained content and identity information from visual signals.

[0022] The consistent visual feature extractor proposed in the present invention can alleviate the gap between cross-modal semantics and modalities, help alleviate cross-modal heterogeneity, extract content and identity information from visual clues, and does not require a text-labeled dataset.

[0023] This invention extracts identity features from the separated speech and visual signals, enabling the separated speech to learn the correct identity. This process helps the consistent visual feature extractor further extract visual features more relevant to the target speech content. This optimized visual feature also facilitates the extraction of the target speech. Effective extraction of the target speech also optimizes the visual features. This mutually reinforcing learning mechanism effectively integrates multiple consistencies to jointly assist in the extraction of the target speaker.

[0024] The present invention provides a cross-modal dual-path Mamba structured speaker extractor that combines a visual feature processing module and an audio-visual fusion module. The speaker extractor can capture long-term dependencies in sequences while effectively processing local information, thereby reducing the computational complexity of long sequence models while maintaining linear complexity.

[0025] The present invention can effectively alleviate the semantic and modality gap between speech and visual modalities without the need for pre-recording speaker information, making it applicable to a wide range of scenarios.

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 This is a schematic diagram of a conference scene in the prior art; Figure 2 is an overall architecture diagram of the MC-Mamba model in an embodiment; Figure 3 is a structural diagram of a consistent visual feature extractor in an embodiment; Figure 4 is a diagram of the Matching Net structure in an embodiment; FIG5( a ) is a structural diagram of a speaker extractor in one embodiment; Figure 5(b) is a diagram of the BiMamba structure in one embodiment; Figure 6 1 is a schematic diagram of target speech extraction results when the same mixed speech is input into USEV, USEV(C) and the proposed MC-Mamba model respectively in an embodiment; Figure 7 is a visual feature output by different networks in an embodiment. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0033] Example 1 A cross-modal speaker extraction method based on multi-consistency includes the following steps: Obtaining mixed speech signals and video data containing target speaker data; Performing audio encoding on the obtained mixed speech signal to obtain audio features; Consistent visual features are extracted from the acquired video data to obtain two visual features that are consistent with the target speech in terms of content and identity. Fuse the visual features and audio features with consistent content, extract the speaker from the fused features, and obtain the separated speech; The separated speech is constrained using identity-consistent visual features to obtain the final extraction result.

[0034] The following is a detailed introduction: The above method is mainly implemented through a model. The model of this embodiment is the MC-Mamba model. Figure 2 As shown, it mainly includes audio encoder, consistent visual feature extractor, speaker extractor and audio decoder.

[0035] First, the mixed speech signal is passed through an audio encoder to obtain audio features. Consistent visual feature extraction is then performed on the face sequence to obtain visual features. In the consistent visual feature extractor, a pre-trained matching network (Matching Net) can obtain two visual features that are consistent with the target speech in terms of content and identity.

[0036] This embodiment feeds consistent visual features into the MLP block and visual TCN (VTCN) to generate higher-level visual features more suitable for the target speaker extraction task. The consistent visual features are then fed into the speaker extractor and fused with audio features. The speaker extractor employs a cross-modal dual-path Mamba architecture to efficiently process long sequences of data both locally and globally. The speaker extractor outputs a mask of the estimated speech, which is then passed through the audio decoder to obtain the estimated speech. Finally, a pre-trained matching network is used to extract identity features for the separated speech and calculate a consistency loss with the visual identity features.

[0037] The identity consistency loss minimizes the discrepancy between the separated speech and the target speech, while maximizing the discrepancy between the separated speech and other audio signals. During identity learning, the consistent visual feature extractor can be optimized using reverse gradients. Optimizing visual features with consistent content is more conducive to extracting the target speech. The more accurate the target speech, the more accurate the extracted identity. This recurrent learning mechanism considers multiple coherences between audio and visual coherences and connects these different coherences. This approach effectively mitigates heterogeneity between different modalities and provides more accurate visual cues.

[0038] First, the audio encoder uses the audio signal in the time domain As input, where T represents the length of the audio. The encoder consists of a one-dimensional convolutional layer and a linear activation layer to encode the one-dimensional audio signal into a two-dimensional audio feature.

[0039] (1) The convolution kernel of the encoder is K and the stride is S. The output of the audio encoder is , in represents the time vector, , represents the feature vector.

[0040] In order to alleviate the heterogeneity between audiovisual cross-modalities and ensure that audiovisual features are consistent in terms of identity and content, this embodiment designs a consistent visual feature extractor. Figure 3 As shown, the main structure consists of an audio-visual matching network (Matching Net) and stacked VTCNs.

[0041] Matching Net is pre-trained on a frame-level audio-visual matching task, enabling the matching network to extract consistent features of identity and content from audio and visual information. These audio-visual features represent the shared representation of audio and vision.

[0042] like Figure 4As shown in Figure 1, the network consists of a face subnet and an audio subnet. Each subnet is derived from the VGG-M architecture, which consists of six convolutional layers. The output of each subnet is processed through two independent one-dimensional convolutional layers with a kernel size of 1, which are used to obtain identity features and content features, respectively. In a video, even if the same target speaker expresses different content at different times, the identity of the target speaker remains unchanged at any moment in the video. Therefore, over time, identity features in audiovisual information change slowly, while content features change rapidly.

[0043] A 2-second video of a speaker is divided into 10 audio clips and 50 face images, where every 5 face images correspond to a 0.2-second audio clip. An audio clip and its corresponding five face images are considered positive samples, meaning that the content they express is consistent. This audio clip and other face clips from the same video form negative samples, which are considered to express inconsistent content. This content consistency constraint allows consistent content features to be captured from audio and visual information. Therefore, this frame-level processing of positive and negative sample pairs can better learn content features. The content feature learning process is similar to the lip reading task, but does not require a text dataset as labels.

[0044] During identity learning, voices and faces from the same speaker, whether or not they are temporally aligned, are considered positive samples, and only voices and faces from different speakers are considered negative samples. This identity constraint allows consistent identity features to be captured from both audio and visual information.

[0045] Matching Net is subject to the constraints of content consistency loss and identity consistency loss. The detailed network structure is shown in [1]. This embodiment uses cosine similarity loss to calculate the similarity between audio and visual features, as shown in Equation (2).

[0046] (2) Where, represents the cosine similarity loss. and Represent the embedded features of audio and face respectively. Indicates content or identity embedding features. Cosine similarity measures the similarity between audio and visual features in terms of direction alignment, indicating whether the face embedding features and audio embedding features are similar. This embodiment uses To measure the similarity of the content and identity embeddings learned from audio and faces, the matching network maps audio and visual features into identity and content embeddings, respectively. The learned embeddings represent shared features of audio and visual information, capturing their inherent correspondence.

[0047] In this example, a pre-trained Matching Net is used to extract content and identity representations consistent with the target speech from visual information for target speaker extraction. The pre-trained Matching Net learns low-level features using a convolutional network with fewer layers, as these low-level features can serve as common features for various high-level tasks.

[0048] In order to further optimize the learned consistent visual features, this embodiment uses VTCN to further learn them. Figure 3 As shown, the visual features are first reduced in dimension through the MLP module to obtain 256-dimensional visual features. . MLP consists of two one-dimensional convolutional layers, an activation function, and batch normalization (BatchNorm). MLP can perform high-dimensional feature mapping to lower dimensions to achieve efficient mapping. Visual features are then further extracted through 5 repeated VTCNs. VTCNs are more capable of modeling visual features. In the model proposed in this embodiment, VTCNs use depthwise separable one-dimensional convolution (DS Conv1d), which is an effective one-dimensional convolution method that can learn visual features well. In VTCNs, this embodiment also uses BatchNorm for normalization. Finally, the consistent visual feature extractor outputs visual features consistent with the target speech content. .

[0049] In time-domain speaker extraction, the model's ability to process long sequences is crucial. The system needs to be able to learn long-term dependent speech sequences and process local features. Therefore, this example establishes a time-domain cross-modal speaker extraction model based on the dual-path Mamba.

[0050] As shown in Figure 5(a), the detailed structure of the speaker extractor is given. The output features of the consistent visual feature extractor are upsampled to obtain The audio features y(t) output by the audio encoder are processed by layer normalization and linear layer. Then they are combined with the visual features Connect to get the fused audio-visual features. Then get the features through the linear layer .first, is divided into J overlapping blocks , the length of each block is L. Then the intra-block and inter-block processing are performed separately.

[0051] (3) in, represents the processing of the jth block, , represents the normalized output. This embodiment uses RMSNrom as the normalization method. The output of intra-block processing undergoes dimensionality transformation before inter-block processing. The intra-block and inter-block processes have the same network structure, except that the inter-block processing (BiMamba) is performed along the J dimension.

[0052] As shown in Figure 5(b), BiMamba uses linear layers, one-dimensional convolutions, and selective SSMs to process sequences within or between blocks, with the Sigmoid Linear Unit as the activation function. The original state-space model (SSM) is very similar to an RNN, where the output at the next moment is related to the hidden state at the previous moment and the current input.

[0053] (4) in, Represents the hidden state at time t, that is, the potential state representation, h(t) represents the input at the current moment, represents the output of SSM, Represents the hidden state at the next moment. Represents the state transfer matrix, which stores the essence of all historical information before time t and is used to update the hidden state at the next moment. Represents the input projection matrix, which affects the current input h(t). Represents the output projection matrix, transitioning from state to output.

[0054] Since SSM is a continuous-time mapping, the zero-order holder approximation method is used to discretize the SSM.

[0055] (5) Where, and is a discretized matrix, and the discretization process is shown by formula (6).

[0056] (6) Where Δ is a learnable parameter that indicates the degree of attention or neglect to the current state and input. The parameters can be updated based on the input at each time step, dynamically sensing the input and thus obtaining a selective state space model. The Selective SSM can selectively focus on or ignore certain input features at each time step, improving the model's efficiency in memorizing valid information and reducing computational effort.

[0057] As shown in Figure 5(b), in order to utilize both past and future sequence information, BiMamba uses a bidirectional Selective SSM. h represents the forward sequence, Represents the backward sequence. The backward sequence is obtained by reversing the forward sequence.

[0058] (7) Where, and Represent the Selective SSM output of the forward and backward sequences respectively. Represents gate control unit.

[0059] (8) Finally, after the linear layer mapping, the final output of BiMamba is obtained After inter-block and intra-block processing, the target speaker’s mask m is obtained through the linear layer and OverlapAdd, and multiplied by the encoder’s output y(t) to obtain the target speaker’s estimated speech signal. .

[0060] The decoder processes the estimated features of the speech to obtain a waveform.

[0061] (9) In the formula is the estimated speech signal. The decoder is a transposed one-dimensional convolutional layer of the same size as the encoder. The estimated speech waveform has the same size as the target speech waveform.

[0062] The target speaker's speech is extracted using visual information related to the speech content. This ensures that the estimated speech signal is closer to the target speech in terms of content. In order to ensure that the estimated speech signal contains the target speaker's identity information, this embodiment uses identity consistency constraints during the training process. Figure 2 As shown, the estimated speech signal is input into MatchingNet to obtain the identity feature The Matching Net in the consistent visual feature extractor extracts identity features from visual information . and Each layer is passed through a fully connected layer for feature dimensionality reduction. Weights are not shared between fully connected layers. To ensure that the estimated speech signal is consistent with the identity from the visual cues and distinguished from other speakers, this embodiment uses a ternary loss to enforce identity consistency. In the ternary loss, the distance between positive (negative) sample pairs is measured using the cosine similarity distance in Equation (2).

[0063] (10) Where, Indicates the A mixed voice, represents the identity features extracted from the estimated speech, represents the identity features extracted from the visual information of the target speaker. represents the identity features extracted from the visual information of the disturbed speaker.

[0064] Furthermore, during training, the scale-invariant signal-to-noise ratio (SISNR) loss is used to evaluate the quality of the separated speech.

[0065] (11) (12) (13) Where, Denotes the target speech signal. Before calculation, the target speech signal and the estimated speech signal are normalized to zero mean.

[0066] During training, SISNR loss and identity consistency loss are used simultaneously to constrain the speaker extraction process. The goal is to minimize SISNR loss and identity consistency loss, minimize the distance between the estimated speech and the target speech, and maximize the difference between the estimated speech and the interfering speech. The overall loss function is shown in Equation (14).

[0067] (14) Where, is a hyperparameter representing the weight of identity consistency loss.

[0068] Throughout the loss function constraint process, identity learning also optimizes the target speaker's speech and content-consistent visual feature extraction processes. The better the content-consistent visual feature extraction, the more accurate the target speech extraction, and the stronger the identity consistency constraint. This reciprocal learning mechanism effectively links content consistency and identity consistency, fully leveraging the strengths of each consistency and maximizing them for target speaker extraction.

[0069] To validate the effectiveness of the proposed method, this example conducts experiments on the Voxceleb2 dataset, a large-scale audio-visual dataset from YouTube videos. The training set contains 5,994 characters and over 1 million clips. The test set contains 118 characters and over 30,000 clips.

[0070] This example only uses utterances longer than 4 seconds. Utterances were randomly selected and mixed from the 800 speakers in the original training set, resulting in a training set of 20,000 mixed utterances and a validation set of 5,000 mixed utterances. Additionally, 6,000 utterances were randomly sampled from the test set of the original Voxceleb2 dataset, forming a test set of 3,000 mixed utterances. During the mixing process, the interfering speech was mixed with the target speech at a signal-to-noise ratio (SNR) of -10dB to -10dB. The utterances in the test set differ from those in the training and validation sets, indicating that independent speaker experiments were conducted.

[0071] (2) LRS2 dataset: The LRS2 dataset is used to verify the generalization ability of the proposed model. The LRS2 dataset is an audiovisual dataset collected from the BBC in the UK, which contains videos of different lengths. It provides the speaker's facial trajectory and the corresponding speech transcript. However, this embodiment does not use text data. This embodiment extracts synchronized speech from the original LRS2 dataset without any additional video preprocessing. Similar to processing the VoxCeleb2 dataset, this embodiment randomly mixes 3000 speech from the original LRS2 test set. This embodiment trains the model proposed in this embodiment on the VoxCeleb2 dataset and tests it on the LRS2 dataset.

[0072] To fully demonstrate the effectiveness and superiority of the MC-Mamba model, this example compares it with other models: TDSE, MUSE, USEV, reentry, and AV-sepformer, which only consider content consistency, and ActivteExtract, which only considers identity consistency. These models cover typical and state-of-the-art architectures for extracting different visual features and speaker extractors.

[0073] (1) TDSE: TDSE is the first time-domain speaker extraction model. They pre-trained the lip-reading model on words and phonemes to extract visual features related to the speech content. However, this pre-trained lip-reading model requires a text dataset containing sentence transcriptions. The MC-Mamba model proposed in this example can also extract visual features related to the speech content, but it does not require a text dataset. TDSE uses stacked TCN blocks as a speaker extractor, which is also a typical method of extracting the target speaker using a convolutional network.

[0074] (2) MUSE: MUSE is similar to TDSE and uses a lip reading model as a visual feature extractor. MUSE uses multiple speaker encoders and mask estimators to extract features of the target speaker. During the mask estimation process, visual features are input to each speaker encoder and mask estimator. This iterative approach optimizes and fully utilizes the visual features in the extraction process, making the separated speech closer to the visual cues.

[0075] (3) Reentry: Reentry is similar to MUSE in its iterative evolutionary approach. Instead of using a lip-reading model to extract visual features, reentry uses a self-supervised pre-trained model to learn synchronization between audio and visual features. Like MC-Mamba in this paper, it does not require a speech transcription dataset to extract content-consistent visual features. The pre-trained audio-visual synchronization model extracts visual features by learning synchronization features for the speaker extraction process.

[0076] (4) USEV: USEV is an architecture that improves the DPRNN structure. Visual information is integrated into a unimodal model to form an audiovisual speaker extraction model. USEV replaces the TCN block in TDSE with a DPRNN block. Under the same visual feature extraction model, DPRNN outperforms TCN in processing long time domain sequences. To demonstrate the effectiveness of this embodiment, this embodiment replaces the visual front-end features of USEV with the proposed consistent visual features, which can be referred to as USEV (c).

[0077] (5) AV-sepformer: AV-sepformer uses the same visual front-end as TDSE, MUSE, and USEV, but it designs a cross-modal transformer to better extract the target speech. In addition, the structure of AV-sepformer is similar to the MC-Mamba model proposed in this paper, both using a dual-path speaker extraction architecture. AV-sepformer replaces the LSTM module with a transformer in the dual-path architecture, while this example uses Mamba instead of the LSTM module.

[0078] (6) ActiveExtract: The speaker extractor of ActiveExtract adopts the same transformer structure as AV-sepformer. ActiveExtract uses the active speaker detection (ASD) task to learn identity-consistent audio and visual features that can be used in the speaker extraction process.

[0079] This embodiment uses the Scale-Invariant Signal-to-Distortion Ratio (SISDR), Scale-Invariant Signal-to-Distortion Ratio Improvement (SISDRi), and Signal-to-Distortion Ratio Improvement (SDRi) to evaluate the performance improvement of separated speech quality restoration. The Perceptual Evaluation of Speech Quality Improvement (PESQi) and Short-Term Objective Intelligibility Improvement (STOIi) are used to indicate the improvement in the perceived quality and intelligibility of the separated speech. Higher levels of these evaluation metrics indicate better model performance.

[0080] The MC-Mamba model proposed in this example is implemented in PyTorch, and all experiments are performed using NVIDIA GeForce RTX GPUs. The speech sampling rate is 16kHz, and the visual information consists of a sequence of facial images with a video frame rate of 25fps. The initial learning rate is set to 1.5e-4 and the batch size is 1. The model is trained up to 100 times. During training, if there is no improvement on the validation set for 5 consecutive epochs, the learning rate is halved. If there is no improvement after 10 consecutive trainings, the training is stopped, which helps prevent overfitting. The kernel size of the audio encoder is 16 and the stride is 8. Weights Set to 0.1.

[0081] This example conducts two sets of experiments on the VoxCeleb2 and LRS2 datasets. We compare different visual front-ends and evaluate the performance of TCN, DPRNN, Transformer, and Mamba architectures in the target speaker extraction task. This example also compares model baselines across different datasets. We then conduct ablation experiments to verify the effectiveness of the proposed multi-consistency method.

[0082] This example compares the results of different models from different perspectives. To ensure a fair comparison, all models are trained and tested on the same dataset using the same data processing method. Since the original USEV used a different train dataset and processing method, this example replicated USEV and USEV(c) using the same dataset and processing method as this study. Since the dataset processing is the same and the test set is the same, the results of reentry, TDSE, and MUSE are from reentry. This example retests the ActiveExtract model using the same test set to maintain its fairness. To conveniently demonstrate the effectiveness of the proposed model, this example proposes AV-DPMamba1, which uses a pre-trained lip reading model (such as MUSE) to extract content-consistent visual features without considering the identity consistency proposed in this article.

[0083] This example analyzes the performance of different models from three perspectives.

[0084] Comparison of different visual front-ends: For the extraction of visual front-end features, this embodiment proposes three categories: content-consistent visual features, identity-consistent visual features, and multiple-consistent visual features.

[0085] Content-consistent visual features can be categorized into two types. TDSE, USEV, MUSE, and AV-sepformer all use the same visual front-end, employing a pre-trained lip-reading model to extract content-related visual features. Reentry, on the other hand, uses a pre-trained audio-visual synchronization model to extract visual features, and the extracted visual features are generally content-consistent.

[0086] Among the identity-consistent visual features, ActiveExtract uses the ASD task to extract identity-related visual features for audio-visual feature fusion in the speaker extraction process.

[0087] This embodiment uses the proposed consistent visual feature extractor to extract visual features consistent with the target speech content and identity. Content-consistent visual features are used for audiovisual feature fusion. Identity-consistent visual features are used to constrain the identity of the separated speech.

[0088] Evaluation Metric Analysis: As shown in Table 1, in the visual front-end, "vsr" represents a visual feature extractor pre-trained using the speech recognition task, "sync" represents a visual feature extractor pre-trained using the audio-visual synchronization task, "asd" represents visual features extracted using the active speaker detection task, and "proposed" represents a proposed consistent visual feature extractor. In terms of SISDRi, the MC-Mamba model outperforms TDSE by 3.14 dB, MUSE by 2.11 dB, USEV by 1.87 dB, USEV(C) by 1.57 dB, reentry by 1.18 dB, and ActiveExtract by 1.54 dB. Compared to Av-sepformer, this method achieves a 1.58 dB improvement in SISDR. The proposed MC-Mamba model utilizes both identity consistency and content consistency, outperforming models that use only one consistency across various evaluation metrics. This demonstrates the effectiveness of the multi-consistency approach in the work presented in this example.

[0089] In addition, USEV is compared with USEV(c) to better demonstrate the superiority of the proposed consistent visual feature extractor. Compared with USEV(c) using the consistent visual feature extractor, SISDR is improved by 0.33dB. This shows the superiority of the consistent visual feature extractor and that this embodiment does not rely on a transcribed speech-to-text dataset during pre-training.

[0090] Feature distribution analysis: Figure 6 As shown in Figure 2, this embodiment uses t-SNE to visualize the feature distribution. This embodiment randomly selects a video clip from the test dataset. The audio and face image sequences extracted from the video are input into the matching network and other visual front-end networks respectively to obtain audio and visual features respectively. The distribution of audio and video features output by the pre-trained matching network, lip reading network, synchronization network and ASD network is shown in Figure 2. Figure 6 As shown in the figure, this example shows that the visual features obtained by the pre-trained matching network are closer to the audio features in terms of feature distribution. This indicates that the content-consistent visual features extracted from the matching network correspond to the target speech, and this example mitigates the heterogeneity between different modalities. It is worth noting that this example does not require an additional text dataset for pre-training, which is more practical.

[0091] Time domain waveform analysis: A mixed speech is randomly selected from the test set, and then the target speech is extracted using USEV, USEV(c) and the proposed MC-Mamba model. Figure 7 As shown, the waveform and details of the speech intuitively illustrate that the proposed MC-Mamba outperforms USEV and USEV(c). This indicates that considering multiple consistencies can better extract the target speaker than considering only a single consistency. The fact that USEV(C) outperforms USEV directly proves that the proposed content consistency visual feature is more effective in extracting the target speech.

[0092] (2) Comparison of different speaker extractors: When considering the structure of the speaker extractor, this embodiment uses AV-DPMamba 1 To compare different speaker extractors, such as TCN (TDSE, MUSE), DPRNN (USEV) and Transformer (AV-sepformer). As shown in Table 1, AV-DPMamba 1 The SISDRi of AV-DPMamba1 is 2.80 dB higher than TDSE and 1.77 dB higher than MUSE. USEV uses a DPRNN as a speaker extractor, while AV-sepformer uses a transformer to accomplish the same task. Their architectures are similar to the dual-path architecture in this paper. AV-DPMamba1's SISDRi is 1.53 dB higher than USEV, and SISDR is 1.25 dB higher than AV-sepformer. These results demonstrate that the Mamba network exhibits superior performance in processing time-domain speech signals.

[0093] In addition, since AV-DP Mamba 1The AV-DPMamba model shares the same size of the audio encoder as the AV-sepformer model, so the length of the speech sequences processed by the speaker extractor remains the same. 1 The model parameters of AV-DPMamba are 11.27m smaller than those of AV-sepformer. 1 The learning speed during training is faster than that of AV-sepformer and it can reach convergence faster under the same experimental conditions.

[0094] (3) Comparison on other datasets: In order to verify the generalization performance of the proposed model, this embodiment was tested on the LRS2 dataset. The model was trained on the Voxceleb2 dataset and tested on the LRS2 dataset. As shown in Table 2, compared with other models, MC-Mamba improves by 2.38 dB over TDSE, 1.78 dB over MUSE, 1.97 dB over USEV, 1.74 dB over USEV(C), 1.33 dB over reentry, and 1.27 dB over ActiveExtract in terms of SISDRi. In addition, in terms of SISDRi of the LRS2 dataset, USEV(C) increases by 0.23 dB over USEV. The test results on the LRS2 dataset show that the performance of MC-Mamba not only does not decrease, but also improves. The results show that the model still has good performance on unknown datasets and has good generalization and robustness.

[0095] In order to verify the effectiveness of the mutual cyclic multi-consistency, this example conducts ablation experiments on the Voxceleb2 and LRS2 datasets, as shown in Table 3. 2 It means that the proposed consistent visual feature extractor is used to extract visual features, but identity consistency is not used to constrain the extraction network. MC-Mamba refers to the proposed multi-consistency model, which considers both content consistency and identity consistency.

[0096] Comparing MC-Mamba vs. AV-DP Mamba 2 When ∑ i = 1, the SISDRi on the Voxceleb2 dataset increased by 0.23 dB. On the LRS2 dataset, the SISDRi increased by 0.45 dB. This demonstrates that the identity consistency constraint proposed in this paper can better extract the target speaker's speech. Multiple consistency is more beneficial for extracting the target speaker's speech than single consistency. Test experiments on the LRS2 dataset yielded good results. This further demonstrates the effectiveness and strong generalization of multiple consistency.

[0097] Table 1

[0098] Table 2

[0099] Table 3

[0100] Example 2 A multi-consistency-based cross-modal speaker extraction system, comprising: a data acquisition module configured to acquire a mixed speech signal and video data including target speaker data; an audio encoding module configured to perform audio encoding on the acquired mixed speech signal to obtain audio features; a consistent visual feature extraction module configured to extract consistent visual features from the acquired video data to obtain two visual features that are consistent with the target speech in terms of content and identity; A fusion extraction module is configured to fuse visual features and audio features with consistent content, perform speaker extraction on the fused features, and obtain separated speech; The identity constraint module is configured to use the identity-consistent visual features to perform identity constraints on the separated speech to obtain the final extraction result.

[0101] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of one or more computer-usable storage media (including but not limited to disk storage, CD - ROM , optical storage, etc.).

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made by those skilled in the art that fall within the spirit and principles of the present invention and do not require creative effort are intended to be within the scope of protection of the present invention.

Claims

1. A cross-modal speaker extraction method based on multi-consistency, characterized by: The following steps are involved: Obtaining mixed speech signals and video data containing target speaker data; Performing audio encoding on the obtained mixed speech signal to obtain audio features; Consistent visual features are extracted from the acquired video data to obtain two visual features that are consistent with the target speech in terms of content and identity. Fuse the visual features and audio features with consistent content, extract the speaker from the fused features, and obtain the separated speech; The separated speech is constrained using identity-consistent visual features to obtain the final extraction result.

2. The method for cross-modal speaker extraction based on multi-consistency according to claim 1, wherein: The process of performing audio encoding on the obtained mixed speech signal includes performing audio encoding using an audio encoder, where the audio encoder includes a one-dimensional convolution layer and a linear activation layer to encode the one-dimensional audio signal into a two-dimensional audio feature.

3. The cross-modal speaker extraction method based on multi-consistency according to claim 1, characterized in that: The process of extracting consistent visual features from the acquired video data includes: extracting using a consistent visual feature extractor, wherein the consistent visual feature extractor includes an audio-visual matching network and stacked VTCNs, wherein the audio-visual matching network is pre-trained through a frame-level audio-visual matching task, including a face subnet and an audio subnet, each subnet includes six convolutional layers, and the output of each subnet is processed by two independent one-dimensional convolutional layers, which are used to obtain identity features and content features respectively.

4. The method for cross-modal speaker extraction based on multi-consistency according to claim 3, wherein: In the process of learning content consistency, frame-level audio-visual matching between positive and negative samples is learned to form content consistency constraints to capture consistent content features from audio and visual information. In this process, no annotated text dataset is required as labels. Alternatively, in the identity consistency learning process, voices and faces from the same speaker are considered positive samples, and voices and faces from different speakers are considered negative samples, forming an identity consistency constraint to capture consistent identity features from audio and visual information.

5. The method for cross-modal speaker extraction based on multi-consistency according to claim 1, wherein: Before fusing the content-consistent visual features with the audio features, the visual features with consistent content are reduced in dimension by the MLP module, and then further extracted through multiple repeated VTCNs. In the VTCNs, BatchNorm is used for normalization to obtain more advanced visual features, which are then fused with the audio features.

6. A cross-modal speaker extraction method based on multi-consistency according to claim 1 or 5, comprising a cross-modal speaker extractor embedded in a Mamba structure, characterized in that: During speaker extraction from the fused features, the audio features are normalized and processed with a linear layer. They are then concatenated with visual features of the same content to obtain fused audiovisual features. The features are then processed through a linear layer and split into multiple overlapping blocks of the same length. Intra-block and inter-block processing is then performed separately. The long sequence features are processed using the Mamba module to obtain a mask for the target speaker. This mask is then multiplied with the audio features to obtain the estimated speech signal of the target speaker. Or further, the same network structure is used within and between blocks, and the network structure includes a linear layer, a one-dimensional convolutional layer, and a selective state space model, and the selective state space model is bidirectional, and the processing between blocks is performed along the dimension of the number of blocks.

7. The method for cross-modal speaker extraction based on multi-consistency according to claim 1, wherein: The identity constraint process includes: the estimated speech signal passes through an audio-visual matching network to obtain identity features, and the identity features and identity-consistent visual features are respectively passed through a fully connected layer for feature dimensionality reduction. The weights are not shared between the fully connected layers to facilitate the retention of specific modal features, and the identity consistency is enforced using a ternary loss.

8. The method for cross-modal speaker extraction based on multi-consistency according to claim 7, wherein: The multiple losses use scale-invariant signal-to-noise ratio loss and identity consistency loss to constrain the speaker extraction process, and the goal is to minimize the scale-invariant signal-to-noise ratio loss and identity consistency loss, minimize the distance between the estimated speech and the target speech, and maximize the difference between the estimated speech and the interfering speech.

9. The method for cross-modal speaker extraction based on multi-consistency according to claim 1, wherein: In the process of identity constraint, reverse gradient optimization is used to optimize the visual features with consistent content to better extract the target speech, thereby achieving further learning constraints on the identity and forming an inter-cyclic learning method.

10. A cross-modal speaker extraction system based on multi-consistency, characterized by: include: a data acquisition module configured to acquire a mixed speech signal and video data including target speaker data; an audio encoding module configured to perform audio encoding on the acquired mixed speech signal to obtain audio features; a consistent visual feature extraction module configured to extract consistent visual features from the acquired video data to obtain two visual features that are consistent with the target speech in terms of content and identity; A fusion extraction module is configured to fuse visual features and audio features with consistent content, perform speaker extraction on the fused features, and obtain separated speech; The identity constraint module is configured to use the identity-consistent visual features to perform identity constraints on the separated speech to obtain the final extraction result.

Images