Audio-visual speech separation method and system based on visual information redistribution

Abstract

The application relates to the technical field of speech separation, and provides an audio-visual speech separation method and system based on visual information redistribution, which aims to cope with the problem of cross-modal information interference by efficiently modeling cross-modal information. The audio-visual speech separation method based on visual information redistribution comprises the following steps: extracting time-domain audio features of all speakers through an audio encoder from a mixed speech signal; capturing lip information of the speakers in a video stream by using a visual coding module and extracting visual features consistent with the dimension of the time-domain audio features from the visual coding module; transmitting the time-domain audio features and the visual features to a separator for separation processing to obtain speech separation features of each speaker and then transmitting the speech separation features to an audio decoder to obtain speech signals of each speaker, which can realize accurate use of effective visual clues by optimizing the distribution mechanism of cross-modal information, and significantly improve the performance of a speech separation model.

Description

Technical Field

[0001] This invention relates to the field of speech separation technology, and in particular to an audiovisual speech separation method and system based on visual information redistribution. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Speech separation technology aims to simulate the human auditory system to separate the speech of all speakers from mixed speech. Speech separation technology can be widely applied to downstream tasks such as speaker recognition, speech recognition, and emotion recognition, thus attracting extensive research. Deep learning-based speech separation methods can be divided into single-modal speech separation methods based on audio signals and cross-modal speech separation methods based on audiovisual signals. Single-modal speech separation methods typically employ end-to-end time-domain signal separation; however, in high-noise environments, the performance of single-modal speech separation models is limited, and the separated speech cannot establish a definite connection with faces in a video. Visual signals can provide modal cues independent of audio, significantly improving the performance of speech separation models.

[0004] In utilizing visual signals through audiovisual feature fusion, a cascaded strategy is typically employed to achieve cross-modal fusion. Audio and video features are extracted separately by an audio encoder and a visual encoder, then the audiovisual features are concatenated, and temporal convolutional attention is used to learn the sequential dependencies of the context. This feature-concatenated fusion method concatenates the visual information of all speakers in a predetermined order and inputs it into the separation system. This mechanism establishes an explicit sequential correspondence between the output separated speech and the input visual features, effectively solving the label arrangement problem in single-modal speech separation methods. However, this fusion method is prone to cross-modal fusion errors, leading to misallocation of visual information. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an audiovisual speech separation method and system based on visual information redistribution. By optimizing the cross-modal information allocation mechanism, it can achieve accurate utilization of effective visual cues and significantly improve the performance of the speech separation model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides an audiovisual speech separation method based on visual information redistribution.

[0007] In one or more embodiments, an audiovisual speech separation method based on visual information redistribution is provided, comprising: The mixed speech signal is processed by an audio encoder to extract the temporal audio features of all speakers; The visual encoding module is used to capture the speaker's lip information in the video stream and extract visual features consistent with the temporal audio feature dimension. Both temporal audio features and visual features are sent to a separator for separation processing to obtain the final speech separation features of each speaker. The final speech separation features of each speaker are processed by an audio decoder to obtain the speech signal of each speaker; The separator comprises a feature fusion module, a first separation network, a visual information redistribution module, and a second separation network connected in series. The feature fusion module concatenates the temporal audio features and visual features of each speaker along the feature dimension to obtain fused audiovisual features. The first separation network divides the fused audiovisual features into several blocks of the same size, and sequentially extracts intra-block features within each block and inter-block features between each block to obtain the initial speech separation features of each speaker. The visual information redistribution module filters out interfering cross-modal information from the initial speech separation features of each speaker, and then provides correct visual information for the speech separation features to obtain corresponding de-interference speech separation features. The second separation network extracts the final speech separation features from the de-interference speech separation features.

[0008] As one implementation, a two-dimensional positional code is added to each block to represent the position of the feature dimension, thereby enhancing the ability to perceive the global position of long sequences and ensuring the positional alignment of audiovisual features in the visual information redistribution module.

[0009] In one implementation, the first separation network and the second separation network each contain several identical separation sub-networks. Each separation sub-network includes two feature extraction paths, which extract intra-block features within each block and inter-block features between each block, respectively. These two feature extraction paths have the same structure and both use bidirectional LSTM, fully connected layers, layer normalization, and residual connections to perceive contextual information.

[0010] In one implementation, the visual information redistribution module includes a subtraction filtering module and an addition supplementation module; the subtraction filtering module is used to filter out interfering cross-modal information, and the addition supplementation module is used to provide correct visual information for the separated speech.

[0011] As one implementation method, the process by which the subtraction filtering module filters out interfering cross-modal information is as follows: The current speech separation features of each speaker are mapped into audio self-query vectors, key vectors, and value vectors; visual features are injected with temporal position information and projected to generate visual query vectors. The self-attention score is obtained by calculating the dot product of the audio self-query vector and the key vector to maintain the semantic consistency of speech features; the subtraction attention score is obtained by calculating the dot product of the visual query vector and the audio key vector to quantify erroneous cross-modal information. The attention score is normalized by Softmax to obtain the enhancement weight; the subtractive attention score is first inverted and then normalized by Softmax to obtain the penalty weight, and then averaged with the enhancement weight to obtain the final weight matrix; The final weight matrix is ​​multiplied by the audio value vector to obtain the cross-modal information without filtering out interference.

[0012] As one implementation method, the process by which the additive supplementation module provides correct visual information for the separated speech is as follows: The features filtered by subtraction are mapped to audio key vectors and audio value vectors, respectively; at the same time, the correct visual features are used to generate visual query vectors through linear projection. The cross-attention score is obtained by calculating the dot product of the visual query vector and the audio key vector, and then generated as a feature enhancement weight by Softmax normalization. The weight matrix is ​​multiplied by the audio value vector to obtain the enhanced feature containing visual complementary information. The enhanced feature is fused with the original input through residual connection to obtain the final output of the cross-modal redistribution module.

[0013] A second aspect of the present invention provides an audiovisual speech separation system based on visual information redistribution.

[0014] In one or more embodiments, an audiovisual speech separation system based on visual information redistribution includes: An audio encoder, used to extract temporal audio features of all speakers from a mixed speech signal; A visual encoding module is used to capture the lip information of a speaker in a video stream and extract visual features consistent with the temporal audio feature dimension. The separator is used to transmit both temporal audio features and visual features to the separator for separation processing, so as to obtain the final speech separation features of each speaker; An audio decoder is used to decode the speech features of each speaker to obtain the speech signal of each speaker. The separator comprises a feature fusion module, a first separation network, a visual information redistribution module, and a second separation network connected in series. The feature fusion module concatenates the temporal audio features and visual features of each speaker along the feature dimension to obtain fused audiovisual features. The first separation network divides the fused audiovisual features into several blocks of the same size, and sequentially extracts intra-block features within each block and inter-block features between each block to obtain the initial speech separation features of each speaker. The visual information redistribution module filters out interfering cross-modal information from the initial speech separation features of each speaker, and then provides correct visual information for the speech separation features to obtain corresponding de-interference speech separation features. The second separation network extracts the final speech separation features from the de-interference speech separation features.

[0015] In one implementation, the audio encoder, visual encoder module, separator, and audio decoder together constitute a speech separation model, which uses scale-invariant signal-to-distortion ratio loss to optimize the entire speech separation model.

[0016] A third aspect of the present invention provides a computer-readable storage medium.

[0017] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the audiovisual-speech separation method based on visual information redistribution as described above.

[0018] A fourth aspect of the present invention provides an electronic device.

[0019] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the audiovisual-speech separation method based on visual information redistribution as described above.

[0020] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes an audiovisual speech separation method based on visual information redistribution. It is built on a temporal speech separation framework and optimizes the cross-modal information allocation mechanism to achieve accurate utilization of effective visual cues, significantly improving the performance of the speech separation model. The invention designs a visual information redistribution process and embeds a cross-modal subtraction filtering module and a cross-modal addition supplementation module into the separation network to actively detect erroneous visual information in the separated speech and make full use of the guiding role of visual information. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 It is a cross-modal fusion error; Figure 2 This is a schematic diagram of the audiovisual speech separation method based on visual information redistribution according to an embodiment of the present invention; Figure 3 This is a structural diagram of the visual encoding module according to an embodiment of the present invention; Figure 4 This is the separator architecture of an embodiment of the present invention; Figure 5 This is a structural diagram of the visual information redistribution module according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in the embodiments of the invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] As can be seen from the background technology, combined with Figure 1 Speaker B's visual features may be mistakenly incorporated into speaker A's separated speech, causing feature interference. This erroneous visual guidance not only introduces invalid information and reduces the learning efficiency of deep neural networks, but also limits the performance of speech separation models.

[0027] This invention proposes an audiovisual speech separation algorithm based on visual information redistribution, aiming to address cross-modal information interference by efficiently modeling cross-modal information. The model mainly consists of an audio encoder and decoder, a visual encoding module, and a separator. The audio encoder and decoder convert the audio signal between the temporal waveform and the feature space; the visual encoding module captures the speaker's lip information in the video stream to extract visual features consistent with the target speech; the separator includes a DPRNN (Dual-Path RNN) module and a visual information redistribution module. The separation network obtains preliminary features of each sound source, which are then optimized by visual information redistribution and fed into subsequent separation networks for deep feature optimization. The visual information redistribution module can also be embedded in other multi-layer separation networks, such as ConvTasNet or other stacked network structures. Figure 2 The diagram shows the overall architecture of the model.

[0028] The proposed visual information redistribution module regulates the allocation of visual information in audio features through a process of subtraction filtering and addition supplementation, so that visual information is used to assist the correct speaker, thereby alleviating the problem of cross-modal information interference in audiovisual speech separation.

[0029] Figure 2 A schematic diagram of the audiovisual-speech separation method based on visual information redistribution according to an embodiment of the present invention is provided. Figure 2 The audiovisual-speech separation method based on visual information redistribution in this embodiment may include the following steps: Step 1: Extract the temporal audio features of all speakers from the mixed speech signal using an audio encoder.

[0030] Mixed speech signals The audio features in the time domain are obtained after processing by the audio encoder. ;in, and These represent the two dimensions of the audio features. The audio encoder consists of a one-dimensional convolutional layer (Conv1d) and the ReLU activation function, with the convolutional kernel being... Step size is .

[0031] (1); Step 2: Use the visual encoding module to capture the speaker's lip information in the video stream and extract visual features that are consistent with the temporal audio feature dimension.

[0032] like Figure 3As shown, the structure of the visual encoding module is similar to previous work. The lip sequence is processed through 3D convolutional layers and ResNet18 to extract lip embedding features. Then, a dimensionality reduction layer maps the lip embedding features to a dimensionality space consistent with the speech features, and stacked TCNs (Temporal Convolutional Networks) are used to further optimize the features, resulting in the visual features of the input separator. ,in .

[0033] Among them, ResNet18 is a deep residual network. As one of the lightest models in the ResNet family, it effectively solves the gradient vanishing problem in the training of deep neural networks by introducing "skip connections" and has become a classic backbone network in the field of computer vision.

[0034] Step 3: Transmit both temporal audio features and visual features to the separator for separation processing to obtain the speech separation features of each speaker.

[0035] The separator comprises a feature fusion module, a first separation network, a visual information redistribution module, and a second separation network, connected in series. Temporal audio features and visual features are concatenated along the feature dimension by the feature fusion module to obtain fused audiovisual features. The first separation network cuts the received features into several equal-sized blocks, sequentially extracting intra-block features within each block and inter-block features between blocks to obtain the current speech separation features for each speaker. The current speech separation features for each speaker are first filtered for interfering cross-modal information in the visual information redistribution module before providing correct visual information for the separated speech. The second separation network further optimizes and separates the output features of the visual information redistribution module.

[0036] Taking two-speaker speech separation as an example, the mixed audio features are... and the visual characteristics of all speakers , concatenated along the feature dimension, and passed through a linear layer Transform feature dimensions The resulting fused audiovisual features ;in, and The two dimensions of the audiovisual features are denoted as .

[0037] (2); Audiovisual features In the separator, the mask signals of each speaker are obtained through a separation network and a visual information redistribution module. Its structure is as follows: Figure 4As shown, the dual-path structure of DPRNN exhibits excellent performance in temporal separation models. This embodiment of the invention uses stacked DPRNN modules as the main separation network. First, Cut into Blocks of the same size The length of each block is .in, , There is a 50% overlap between adjacent blocks, which ensures the continuity and smoothness of the signal and prevents the loss of intermediate transition information. In order to enhance the model's ability to perceive the global position of long sequences and ensure the positional alignment of audiovisual features in the subsequent visual information redistribution process, a two-dimensional positional encoding is added to the original structure, as shown in Equation (3).

[0038] (3); In the formula, Represents two-dimensional position encoding; This indicates the feature after adding position encoding; This is the function corresponding to two-dimensional position encoding.

[0039] Then, each DPRNN layer has two paths, processing the segmented features intra-block and inter-block respectively. Each path uses a bidirectional LSTM (Bi-LSTM), fully connected layers, layer normalization, and residual connections to perceive contextual information. The Bi-LSTM perceives past and future information through forward and backward directions respectively, and concatenates the two to give the network a global perspective. Intra-block processing aims to capture local details, and this process is fixed. Dimension, focusing on the length within each block. The sequence is modeled. The output of the Bi-LSTM within the block is used. Intermediate features are obtained after processing with a fully connected layer (FC) and layer normalization (LN). As shown in equation (4).

[0040] (4); and Residual connections are made to obtain the output within each block, with each block sharing network weights. After intra-block processing, the dimensions are transformed for inter-block processing. Inter-block processing uses the same network structure but focuses on establishing global dependencies through fixed dimensions. Regarding length The block sequence is modeled holistically. Features are output between blocks. The estimated features for all speakers are obtained after passing through ReLU and a 2D convolutional layer. These separated features are then fed into a visual information redistribution module, which aims to further optimize the separated features by detecting and suppressing erroneous cross-modal interference. The features optimized in the first round then enter the next layer of the DPRNN and are processed again by this redistribution module after the output.

[0041] Combination Figure 5 The visual information redistribution module is based on an attention mechanism and includes a subtraction filtering module and an addition supplementation module. The separation features output from the two-layer DPRNN in the first separation network are filtered by the subtraction filtering module to remove interfering cross-modal information, and then pass through the addition supplementation module to provide the correct visual information for the separated speech. The second separation network contains a four-layer DPRNN network for deep separation of the optimized separated speech features.

[0042] All speaker-segmented speech undergoes a redistribution process, which shares the network's weights. Taking speaker A's speech segmentation optimization process as an example, we will introduce the specific flow of visual information redistribution. Figure 4 As shown, speech features After three independent linear layers , and Mapped to audio self-query vector Key vector Sum value vector The calculation process is shown in equation (5).

[0043] (5); In order to align the spatiotemporal features, inject temporal location information into the speaker features without location information, and achieve more accurate attention mechanism query, the visual features of the two speakers were encoded with a location similar to that in Equation (3), as shown in Equation (6).

[0044] (6); in, and These are the location-encoded visual information of speaker A and speaker B, respectively. This is the integer division operator; To take half the block size; Indicates the length of the block.

[0045] Then, the location-encoded visual information of the non-target speaker (speaker B) Visual query vectors are generated through linear layer projection. In the subtraction filtering module, Used as a self-attention mechanism, it aims to capture semantic dependencies within speech; As a distraction query vector, it aims to locate and separate residual erroneous visual distractions in speech. The subtractive filtering process integrates self-attention and subtractive attention learning; the former calculates... and The dot product yields the self-attention score. To maintain the semantic consistency of speech features; the latter is achieved through computation. and The dot product yields the subtraction attention score. This is used to quantify cross-modal information about errors. During the weight generation phase, The enhanced weights are obtained after Softmax normalization; and First, the weights are inverted, then processed using Softmax to apply a negative penalty to highly matching interfering features. The final weight matrix is ​​obtained by averaging the two sets of weights. ;in, This represents the scaling factor, which is a constant.

[0046] (7); Then, AND value vector Multiplication yields the output of the subtraction filtering module. This achieves effective filtering of erroneous visual interference and purification of target features.

[0047] (8); In the addition supplementation module, the features are filtered by subtraction. They are fed into separate linear layers and mapped to audio key vectors respectively. Sum value vector Simultaneously, the correct visual features are used to generate a query vector through linear projection. This aims to provide precise positive guidance for speech feature recovery. Through computation... and The dot product yields the cross-attention score. The feature enhancement weights are generated through Softmax normalization. This weight matrix and value vector... Multiplication yields enhanced features that incorporate visual complementary information. Finally, the enhanced features are fused with the original input through residual connections to obtain the final output of the cross-modal redistribution module. .

[0048] (9); The speech features of speaker B were also optimized using the same steps to obtain the optimized features. Optimized features The signal is then processed by a separation network, which is further optimized by subsequent layers of stacked DPRNNs. Finally, the output of the separator is subjected to overlapping summation, one-dimensional convolution, and a ReLU function to obtain the speaker's mask. .

[0049] It should be noted that the additive supplementation module can be embedded in a separator composed of stacked network layers. It is not limited to the dual-path DPRNN structure, but can also be used in the single-path separator structure of stacked network layers, such as ConvTasNet.

[0050] The subtraction filtering module of this invention uses visual features as queries to guide the model to locate interference components of non-target speakers in the audio feature space, and removes non-target speaker information from the features through reverse subtraction, thereby purifying the target speech representation. The additive supplementation module of this invention effectively fuses corresponding audiovisual features by utilizing a cross-modal attention mechanism, and strengthens the intrinsic connection between modalities by coordinating the consistency information of audiovisual signals, thereby making full use of visual information to effectively supplement the target speech.

[0051] Step 4: The final speech separation features of each speaker are processed by an audio decoder to obtain the speech signal of each speaker.

[0052] This invention demonstrates experiments conducted on the VoxCeleb2 dataset. Speech was randomly selected and mixed from 800 speakers in the original training set, resulting in a training set containing 20,000 mixed utterances and a validation set containing 5,000 mixed utterances. Furthermore, this invention uses a test set configuration consistent with the CTCNet method to evaluate the proposed method. The dataset used has a video frame rate of 25 fps, a speech sampling rate of 16 kHz, and each audio sample is uniformly truncated to 3 seconds. During speech mixing, a signal-to-noise ratio (SNR) ranging from -5 dB to 5 dB is used. This invention uses Signal-to-Distortion Ratio (SDR) and Scale-Invariant Signal-to-Noise Ratio Boost (SISDRi) to evaluate the similarity between the separated speech and the real speech. Perceptual Speech Quality Assessment (PESQ) and Short-Term Objective Intelligibility (STOI) are used to evaluate the speech's clarity and intelligibility. Higher values ​​for these metrics indicate better separation performance.

[0053] Comparative experiment: To fully demonstrate the superiority of the method proposed in this embodiment of the invention, it is compared and analyzed with representative advanced speech separation algorithms in recent years. To ensure the fairness of the evaluation and the comparability of the results, the method proposed in this embodiment of the invention and the compared cross-modal speech separation methods are tested using the same test set and strictly adhere to the same speaker identity. Detailed performance comparison results are shown in Table 1.

[0054] Table 1 compares the performance of the proposed method with other state-of-the-art methods;

[0055] Table 1 presents the experimental results, which visually demonstrate the significant advantages of the proposed method in various key evaluation metrics. Specifically, in terms of SDR (Speed-Reduction) performance, this method achieves a substantial improvement of 3.57 dB compared to the two-stage fusion strategy. In terms of SISDRi (Speed-Reduction Difference), it achieves performance gains of 3.14 dB, 3.04 dB, 1.71 dB, and 0.54 dB, respectively, compared to VisualVoice (single-stage cascaded fusion), AVLIT-8, and CTCNet (attention mechanism fusion). It is noteworthy that the benchmarks include the two major time-frequency domain speech separation paradigms. The proposed method demonstrates superiority in comparison with both of these algorithms, fully validating its broad applicability and robustness. The visual information redistribution mechanism within the model actively identifies and suppresses irrelevant interference information in the separated speech, effectively alleviating the mismatch problem in cross-modal fusion, thereby overcoming the performance bottleneck of existing methods.

[0056] VisualVoice is an audio-visual speech separation neural network model that can accurately extract the speech signal corresponding to a specific face from a video containing multiple speakers and background noise.

[0057] AVLIT-8 is a deep learning model architecture for audio-visual speech separation, which improves the quality of speech separation in complex environments by fusing auditory and visual information.

[0058] CTCNet is a model architecture that has been named multiple times in different fields. The version involving attention mechanism fusion mainly refers to the Cortical-Thalamic-Cortical Network (CTCNet), proposed by the Tsinghua University team and inspired by brain science, for the purpose of implementing multimodal speech separation tasks.

[0059] To verify the effectiveness of the visual information redistribution mechanism proposed in this invention, and to explore the independent contributions of the subtraction screening module and the additive supplementation module, a systematic ablation experiment was conducted. Three variants were designed for comparative analysis: "Proposed-None" (without any redistribution module embedded) was used to determine the performance benchmark; "Proposed-Subtraction" (embedded only with the cross-modal subtraction screening module) was used to evaluate the effectiveness of the proposed method in suppressing interfering information; and "Proposed-Addition" (embedded only with the cross-modal additive supplementation module) was used to evaluate the effect of feature enhancement. The evaluation results of the different variants are shown in Table 2.

[0060] Table 2 shows the ablation experimental results of the proposed method;

[0061] As shown in the table above, compared to the baseline model "Proposed-None", "Proposed-Addition" and "Proposed-Subtraction" achieved performance improvements of 0.43 dB and 0.80 dB, respectively. "Proposed", which integrates a complete cross-modal redistribution module, achieved a gain of 1.04 dB, fully demonstrating the module's significant contribution to separation performance. Further analysis revealed that the performance gain from "Proposed-Subtraction" is higher than that from "Proposed-Addition". This difference reveals that interference caused by misallocation of cross-modal information is one of the bottlenecks restricting performance in the audiovisual fusion process, and also strongly verifies the effectiveness of the subtraction screening process in suppressing interference information.

[0062] In one or more embodiments, an audiovisual speech separation system based on visual information redistribution is also provided, which can be implemented in software. The audiovisual speech separation system based on visual information redistribution includes the following software modules: An audio encoder, used to extract temporal audio features of all speakers from a mixed speech signal; A visual encoding module is used to capture the lip information of a speaker in a video stream and extract visual features consistent with the temporal audio feature dimension. The separator is used to transmit both temporal audio features and visual features to the separator for separation processing, so as to obtain the final speech separation features of each speaker; An audio decoder is used to decode the speech features of each speaker to obtain the speech signal of each speaker. The separator comprises a feature fusion module, a first separation network, a visual information redistribution module, and a second separation network connected in series. The feature fusion module concatenates the temporal audio features and visual features of each speaker along the feature dimension to obtain fused audiovisual features. The first separation network divides the fused audiovisual features into several blocks of the same size, and sequentially extracts intra-block features within each block and inter-block features between each block to obtain the initial speech separation features of each speaker. The visual information redistribution module filters out interfering cross-modal information from the initial speech separation features of each speaker, and then provides correct visual information for the speech separation features to obtain corresponding de-interference speech separation features. The second separation network extracts the final speech separation features from the de-interference speech separation features.

[0063] Specifically, the audio encoder, visual encoder module, separator, and audio decoder together constitute the speech separation model, which uses scale-invariant signal-to-distortion ratio loss to optimize the entire speech separation model.

[0064] The speaker's mask is multiplied by the output of the audio encoder to obtain the estimated speaker features, which are then passed through the audio decoder to obtain the estimated speaker signal. The audio decoder is a transposed one-dimensional convolution with the same size as the encoder. This embodiment of the invention uses Scale-Invariant Signal-to-Distortion Ratio (SISDR) loss to optimize the entire separation network, specifically expressed as: (10); (11); (12); In the formula, The loss function; and These are the estimated and target speech signals, respectively. This represents the projection of the estimated signal onto the target signal, thus eliminating the scaling effect. The remaining noise signal is used. To eliminate the impact of signal amplitude scaling differences on the evaluation and to measure the correlation between clean speech and separated speech at the same scale, the model performs zero-mean normalization on both the target speech signal and the separated speech signal before calculating the loss. During the training phase, SISDR constrains the speech separation process, guiding the iterative optimization of network parameters. The training objective is to minimize... .

[0065] It should be noted that each module in the audiovisual speech separation system based on visual information redistribution in this embodiment corresponds one-to-one with each step in the audiovisual speech separation method based on visual information redistribution in the above embodiment, and their specific implementation processes are the same, so they will not be repeated here.

[0066] The structure of the electronic device according to an embodiment of the present invention will be described in detail below. The electronic device provided in this embodiment includes at least one processor, a memory, a user interface, and at least one network interface. The various components in the audiovisual-voice separation system based on visual information redistribution are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The user interface 303 may include a display, keyboard, mouse, trackball, click wheel, buttons, a touchpad, or a touch screen, etc.

[0067] It is understood that the memory can be volatile memory or non-volatile memory, or both. The memory in this embodiment of the invention is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0068] In some embodiments, the audiovisual-speech separation system based on visual information redistribution provided in this invention can be implemented using a combination of hardware and software. As an example, the audiovisual-speech separation system based on visual information redistribution provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the audiovisual-speech separation method based on visual information redistribution provided in this invention. For example, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0069] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where a general-purpose processor can be a microprocessor or any conventional processor, etc.

[0070] As an example of the hardware implementation of the audiovisual speech separation system based on visual information redistribution provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the audiovisual speech separation method based on visual information redistribution provided in this embodiment of the invention.

[0071] The memory in this embodiment of the invention is used to store various types of data to support the operation of the audiovisual-speech separation system based on visual information redistribution, or to store program code for executing the audiovisual-speech separation method based on visual information redistribution. Examples of such data include any executable instructions for operating on the audiovisual-speech separation system based on visual information redistribution, such as executable instructions, and programs implementing the audiovisual-speech separation method based on visual information redistribution of this invention may be included in the executable instructions.

[0072] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing an audiovisual-speech separation method based on visual information redistribution. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs the various functions defined in the apparatus of this application.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for audiovisual speech separation based on visual information redistribution, characterized in that, include: The mixed speech signal is processed by an audio encoder to extract the temporal audio features of all speakers; The visual encoding module is used to capture the speaker's lip information in the video stream and extract visual features consistent with the temporal audio feature dimension. Both temporal audio features and visual features are sent to a separator for separation processing to obtain the final speech separation features of each speaker. The final speech separation features of each speaker are processed by an audio decoder to obtain the speech signal of each speaker; The separator includes a feature fusion module, a first separation network, a visual information redistribution module, and a second separation network connected in series. The feature fusion module is used to concatenate the temporal audio features and visual features of each speaker along the feature dimension to obtain fused audiovisual features. The first separation network is used to cut the fused audiovisual features into several blocks of the same size, and sequentially extract the intra-block features within each block and the inter-block features between each block to obtain the initial speech separation features of each speaker. The visual information redistribution module includes a subtraction filtering module and an addition supplementation module. The subtraction filtering module is used to filter out interfering cross-modal information. The process is as follows: the current speech separation features of each speaker are mapped into audio self-query vectors, key vectors, and value vectors; visual features are injected with temporal position information and projected to generate visual query vectors; the dot product of the audio self-query vector and the key vector is calculated to obtain a self-attention score to maintain the semantic consistency of speech features; the dot product of the visual query vector and the audio key vector is calculated to obtain a subtraction attention score, which is used to quantify erroneous cross-modal information; the self-attention score is normalized by Softmax to obtain enhancement weights; the subtraction attention score is first inverted and then normalized by Softmax to obtain penalty weights, which are then averaged with the enhancement weights to obtain the final weight matrix; the final weight matrix is ​​multiplied by the audio value vector to obtain the cross-modal information without interference. The additive supplementation module provides correct visual information for speech separation, thereby obtaining corresponding interference-free speech separation features. The process is as follows: the features filtered by subtraction are mapped to audio key vectors and audio value vectors, respectively; simultaneously, the correct visual features are used to generate a visual query vector through linear projection; the dot product of the visual query vector and the audio key vector is calculated to obtain the cross-attention score, and then normalized by Softmax to generate feature enhancement weights; this weight matrix is ​​multiplied by the audio value vector to obtain enhanced features containing visual complementary information; the enhanced features are fused with the original input through residual connections to obtain the final output of the cross-modal redistribution module. The second separation network is used to extract the final speech separation features from the denoised speech separation features.

2. The audiovisual speech separation method based on visual information redistribution as described in claim 1, characterized in that, Two-dimensional positional encoding is added to each block to represent the position of the feature dimension, thereby enhancing the ability to perceive the global position of long sequences and ensuring the positional alignment of audiovisual features in the visual information redistribution module.

3. The audiovisual-speech separation method based on visual information redistribution as described in claim 1, characterized in that, The first and second separation networks each contain several identical separation sub-networks. Each separation sub-network includes two feature extraction paths, which extract intra-block features within each block and inter-block features between each block, respectively. These two feature extraction paths have the same structure and both use bidirectional LSTM, fully connected layers, layer normalization, and residual connections to perceive contextual information.

4. An audiovisual speech separation system based on visual information redistribution, characterized in that, The audiovisual-speech separation method based on visual information redistribution as described in any one of claims 1-3 includes: An audio encoder, used to extract temporal audio features of all speakers from a mixed speech signal; A visual encoding module is used to capture the lip information of a speaker in a video stream and extract visual features consistent with the temporal audio feature dimension. The separator is used to transmit both temporal audio features and visual features to the separator for separation processing, so as to obtain the final speech separation features of each speaker; An audio decoder is used to decode the speech features of each speaker to obtain the speech signal of each speaker. The separator comprises a feature fusion module, a first separation network, a visual information redistribution module, and a second separation network connected in series. The feature fusion module concatenates the temporal audio features and visual features of each speaker along the feature dimension to obtain fused audiovisual features. The first separation network divides the fused audiovisual features into several blocks of the same size, and sequentially extracts intra-block features within each block and inter-block features between each block to obtain the initial speech separation features of each speaker. The visual information redistribution module filters out interfering cross-modal information from the initial speech separation features of each speaker, and then provides correct visual information for the speech separation features to obtain corresponding de-interference speech separation features. The second separation network extracts the final speech separation features from the de-interference speech separation features.

5. The audiovisual-speech separation system based on visual information redistribution as described in claim 4, characterized in that, The audio encoder, visual encoder module, separator, and audio decoder together constitute the speech separation model, which uses scale-invariant signal-to-distortion ratio loss to optimize the entire speech separation model.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the audiovisual speech separation method based on visual information redistribution as described in any one of claims 1-3.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the audiovisual speech separation method based on visual information redistribution as described in any one of claims 1-3.

Images