Two-stage speech separation method and system based on visual guidance

Abstract

The application provides a two-stage speech separation method and system based on visual guidance. In the first stage, the obtained mixed speech is separated into speaker speech in the time domain. In the second stage, the independent speech features with speaker information are extracted by means of the coarsely separated speech in the first stage, and the potential correlation features and complementary features between the visual and audio modalities are mined and fused and separated, so that the pure target speech is finally obtained. The independent speech features of the speaker are extracted by the first stage, the introduction of pure reference sound is avoided, the speech separation performance and robustness of visual guidance are improved, and the label arrangement problem is solved. The weight of the two-stage model is dynamically adjusted to further improve the speech separation quality, and the disclosed speech separation system is suitable for most application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of speech separation technology, and relates to a two-stage speech separation method and system based on visual guidance. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Speech separation refers to extracting one or more target speech signals from mixed speech produced by multiple speakers. The speech separation problem stems from the "cocktail party effect," which describes the phenomenon where, in a noisy indoor environment, such as a cocktail party, many different types of sound sources exist simultaneously. Even so, people can focus their attention on one person's conversation while ignoring other background conversations or noise. This is a kind of auditory selective ability. We hope to achieve this speech selection and filtering ability through machine learning. Speech separation has wide applications and is a fundamental and important part of many downstream speech tasks. Separating high-quality, clean speech is essential for better application in speech recognition [TORFIA, IRANMANESH SM, NASRABADI N, et al. 3D Convolutional Neural Networks for CrossAudio-Visual Matching Recognition[J].IEEE Access,2017][T.AFOURAS JSC, A. SENIOR, O. VINYALS, AND A. ZISSERMAN. Deep Audio-Visual Speech Recognition[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2018], human-computer interaction, and other scenarios.

[0004] Traditional speech separation algorithms include Computational Auditory Scene Analysis (CASA), Nonnegative Matrix Factorization, and Hidden Markov Models, which usually require certain assumptions and prior knowledge [JOSH H, MCDERMOTT. The cocktail party problem[J]. Current Biology, 2009]. They also have poor performance when separating similar speakers' speech [Rivet B, Wang W, Naqvi SM, et al. Audiovisual speech source separation: An overview of key methodologies[J]. IEEE Signal Processing Magazine, 2014, 31(3):125-134.]. With the development of deep learning, neural networks have also achieved good results in the field of speech separation. Neural networks can learn the complex mapping relationship between mixed speech signals and target speech signals. Common pure audio methods have the problem of label arrangement when training the separation network. Although the correct matching can be selected through permutation-invariant training (PIT), the calculation is relatively complex. In addition, speech is greatly affected by the environment and noise, resulting in poor robustness of the separation system.

[0005] In real-world scenarios, people aid their auditory perception by observing the speaker; hearing is easier when the speaker's face or lips are visible. Furthermore, studies in psychology, physiology, and psychiatry [Golumbic EZ, Cogan GB, Schroeder CE, et al. Visual input enhances selective speech envelope tracking in auditory cortex at a 'cocktail party'[J]. Journal of Neuroscience, 2013, 33(4):1417-1426.] have demonstrated that visual information helps people understand speech. Compared to speech, visual information about the speaker, such as lip movements and facial appearance, is more stable. Simultaneously, visual information possesses identity characteristics, allowing for the matching of correct speaker labels during the separation of mixed speech. The separation model proposed by Ephrat in 2017 only incorporates static images, which, while reducing data dimensionality, loses temporal information about the visual information and compromises separation performance.

[0006] Some studies [AFOURAS TCJS, ZISSERMAN A. My lips are concealed Audio-visual speech enhancement through obstructions[J]. Interspeech, 2019, 4295-9.][OCHIAI T, DELCROIX M, KINOSHITA K, et al. Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-Visual Speaker Clues[M]. Interspeech 2019, 2718-22.][R.Gu, S.-X.Zhang, Y.Xu, L.Chen, Y.Zou, and D.Yu, “Multi-modal multi-channel target speech separation,” IEEE Journal of Selected Topics in Signal Processing, 2020] have proposed using additional reference speech features extracted from the speaker to improve separation performance. Wang et al. used an x-vector for speaker recognition in their separation model, while Luo et al. introduced an i-vector of the speaker's clean speech to assist in separation. However, this method has two drawbacks. First, it requires the speaker's clean reference speech to be recorded in advance before the separation model can be trained. Second, when the model is trained and applied to a real separation scenario, the speaker's clean speech must be available for separation. Therefore, it has great limitations in real-world applications. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes a two-stage speech separation method and system based on visual guidance. In the first stage, the invention extracts the speaker's unique speech features that distinguish the speaker's identity to assist in speech separation. Furthermore, it enhances speech separation performance by extracting and fusing features from both visual and audio modal information.

[0008] According to some embodiments, the present invention adopts the following technical solution:

[0009] A two-stage speech separation method based on visual guidance includes the following steps:

[0010] In the first stage, the acquired mixed speech is separated in the time domain to obtain coarsely separated speaker speech;

[0011] In the second stage, independent speech features with speaker information are extracted using the pure audio separation results from the first stage. Then, potential related and complementary features between the visual and audio modalities are mined. The visual features and speech time-frequency domain features are fused and then separated. The weights of the two stages are dynamically adjusted to finally obtain the pure target speech.

[0012] As an alternative implementation, the first stage specifically includes the following process:

[0013] The acquired mixed speech is encoded using an encoder to extract mixed speech features;

[0014] The mixed speech features are separated to obtain the mask of the target speech, the target speech features are determined, and the target speech features are decoded to obtain the coarsely separated target speech time-domain signal.

[0015] As a further limitation, the specific process of separating mixed speech features includes processing the mixed speech features using a first separation network. The first separation network is a temporal convolutional network structure, including a normalization layer and multiple identical stack modules, wherein each stack module consists of a fully convolutional layer, a dilated convolutional layer, and a residual module. The output of the last stack module passes through a convolutional layer and a PReLU activation layer to obtain the separated target mask.

[0016] As a further limitation, the target speech feature is calculated by multiplying the mask of the mixed speech and the target speech.

[0017] As an alternative implementation, the second stage specifically includes the following process:

[0018] The mixed speech is transformed to obtain the complex spectrum of the mixed speech, and the complex spectrum mask of the real clean speech is obtained from it;

[0019] The time-domain signal of the target speech obtained in the first stage is transformed to obtain the complex spectrograms of each speaker after separation; the complex spectrograms are then processed by the ResNet-18 independent speech feature extraction network to extract the independent speech features of each speaker.

[0020] Visual information of the speaker synchronized with the mixed speech is acquired and preprocessed. Static visual features and dynamic visual features are extracted from the preprocessed visual images. Static visual features contain distinctive speaker identity information and are similar to voice features such as timbre. Dynamic visual features contain speech content information and are similar to voice features such as phonemes. Combining these two types of visual features can obtain more speech-related features while processing less dimensional information.

[0021] The hybrid speech feature extraction network extracts speech features from the time-frequency domain information of the hybrid speech, performs multimodal feature fusion, and the separation network 2 separates the multimodal features to obtain the mask of the separated target speech. The mask is multiplied with the complex spectrogram of the hybrid speech and then inversely transformed. The time-domain speech signal of the target speaker is obtained by joint training and dynamic optimization of the weights of the two-stage separation.

[0022] As a further limitation, the specific process of transforming the mixed speech to obtain the complex spectrum of the mixed speech includes performing a short-time Fourier transform on the mixed speech signal, and then calculating the real part and imaginary part to obtain the complex spectrum, which contains the amplitude and phase information of the speech.

[0023] As a further constraint, the coarsely separated speech in the first stage is transformed in the time-frequency domain to obtain a complex spectrogram; then, the independent speech feature extraction network ResNet-18 is used to extract independent speech features from the complex spectrograms of each speaker; then, the independent speech features are transformed in the time dimension to achieve dimensional consistency between the audio and video modal features.

[0024] As a further limitation, the specific process of acquiring and preprocessing the speaker's visual information synchronized with the mixed speech includes reading the video file, extracting a video of a set length to obtain a multi-frame image sequence, randomly selecting a facial image as static visual information, then cropping each frame image sequence, selecting a lip region of a set size to reduce data dimensionality, and generating a lip sequence file as dynamic visual information.

[0025] As a further limitation, the specific process of extracting visual features includes normalizing and data filling of the lip image. The preprocessed lip data is then passed through a dynamic visual feature extraction network, which includes a 3D convolutional layer, ShuffleNet v2, and a temporal convolutional network to extract time-series features, which can better fit the content information of the speech, and finally obtain the lip features.

[0026] The facial images are standardized and resized, and then the ResNet-18 static visual feature extraction network is used to extract features containing speaker identity information. The facial features are then transformed in the time dimension to ensure that the transformed features have the same time dimension as the lip sequence features.

[0027] As a further limitation, the specific process of multimodal feature fusion includes firstly, passing the mixed sound spectrogram through a mixed speech feature extraction network to obtain mixed speech features, then concatenating the speaker's visual features, independent speech features, and mixed speech features in a cascade manner to finally obtain fused multimodal features.

[0028] As a further limitation, the multimodal features are separated using a second separation network, which is an upsampling network layer of U-Net.

[0029] As a further constraint, the weights of the loss function of the two-stage separation network are dynamically adjusted to maximize the use of the independent speech features of the first stage to assist the separation in the second stage.

[0030] A two-stage speech separation system based on visual guidance includes:

[0031] The first separation module is configured to separate the acquired mixed speech in the time domain in the first stage to obtain coarsely separated speaker speech;

[0032] The second separation module is configured to extract sound features with speaker information using the pure audio separation results of the first stage in the second stage, while mining the potential correlation and complementary features between the visual and audio modalities, and then separating the two modalities of visual features and speech time-frequency domain features. The weights of the two stages are dynamically adjusted to finally obtain the separated target speech.

[0033] The dynamic weighting module dynamically adjusts the weights based on the performance of the separation model in both stages, so as to maximize the use of the independent speech features extracted in the first stage to assist the second stage and achieve clean target speaker speech separation.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] (1) The two-stage speech separation scheme based on visual guidance proposed in this invention can extract the speech features of a single speaker to assist speech separation in the case of mixed speech, thus avoiding the introduction of additional clean reference speech.

[0036] (2) This invention utilizes dynamic visual features containing speech content information and static visual features containing identity information to explore the potential correlation and complementarity between the two modalities of vision and audio. It also solves the label arrangement problem in pure audio speech separation, avoids the computational complexity of loss function, and improves the separation effect and the robustness of the separation system.

[0037] (3) This invention proposes a method for dynamically adjusting the weights of the loss function for two-stage speech separation. While simultaneously optimizing two training objectives, it maximizes the use of independent speech features extracted from the first stage to assist the separation in the second stage, ultimately obtaining clean speech with high performance indicators. Attached Figure Description

[0038] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0039] Figure 1 This is a flowchart of a two-stage speech separation method based on visual guidance according to the present invention.

[0040] Figure 2 This is a flowchart of a method for extracting visual features according to the present invention.

[0041] Figure 3 This is a flowchart of a method for dynamically adjusting the weight coefficients of a loss function according to the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] This invention proposes a two-stage speech separation method and system based on visual guidance. The first stage extracts the speaker's speech features to assist speech separation, and the second stage uses feature extraction and fusion of multimodal information to assist speech separation.

[0046] The specific process includes:

[0047] I. First Stage: Pure Audio Time Domain Separation

[0048] 1. Acquire mixed speech. Randomly select and read clean WAV files of two speakers (taking two-person mixed speech separation as an example), extract a fixed length (2.55s as an example), sample the read time-domain speech signal at a sampling rate of 16kHz, and normalize it, denoted as x. A ,x B Adding two clean speech samples together yields a mixed speech sample x. mix =x A +x B .

[0049] 2. Mixed speech x mix The encoder extracts mixed speech features. The encoder uses a single-layer one-dimensional convolutional network.

[0050] 3. The mixed speech features are processed by Separation Network 1 to obtain the mask of the separated target speech. Separation Network 1 uses a Temporal Convolutional Network (TCN) structure. The TCN consists of a normalization layer (performing group normalization and one-dimensional convolution sequentially), n identical stack modules, where each stack module consists of a fully convolutional layer, a dilated convolutional layer, and a residual module. The output of the last stack module is processed by a convolutional layer and a PReLU activation layer to obtain the separated target mask.

[0051] 4. Mix speech x mix Multiply each feature by the mask of the target speech to obtain the target speech features of the corresponding speaker.

[0052] 5. The target speech features obtained above are processed by a decoder to obtain the time-domain signal of the target speech, denoted as x'. A ,x' B The decoder uses a single-layer one-dimensional transposed convolutional network.

[0053] II. Multimodal time-frequency domain separation in the second stage

[0054] 6. Obtain the time-frequency domain information of the mixed speech – the complex ideal ratio mask (cIRM), denoted as S. mix The complex spectrum is an ideal ratio mask in the complex domain. It can represent three-dimensional information of time, frequency, and energy in a two-dimensional plane. The real and imaginary parts avoid the loss of phase information caused by the simple amplitude spectrum, which leads to a decrease in separation quality. First, a short-time Fourier transform (STFT) is performed on the mixed speech signal, and then the real and imaginary parts are calculated to obtain the complex spectrum. The STFT is shown in formula (1).

[0055]

[0056] Where x(m) is the input speech signal, w(m) is the window function, and X(n,ω) is a two-dimensional function of time n and frequency ω. In this invention, the length of the window function window_size can be set to 400, the number of audio samples hop_size between adjacent STFT columns can be set to 160, that is, the number of overlapping audio samples between adjacent windows is 160, the length of the windowed signal after zero padding is n_fft to 512, and the final size of the complex spectrogram is 2*F*T, that is, 2*257*256, where 2 represents the real and imaginary parts, and F and T represent the frequency and time dimensions, respectively.

[0057] 7. The clean speech in the time domain is transformed by STFT (as shown in formula (1)) to obtain the corresponding complex spectrogram. The complex spectral mask M of the real clean speech is obtained based on the complex spectrogram of the mixed speech. A M B .

[0058]

[0059] Among them, Y r and Y i S represents the real and imaginary components of the complex spectrogram of mixed speech, respectively. r and S i Let represent the real and imaginary components of the complex spectrogram of clean speech, respectively. Based on the ideal complex mask and mixed speech, the clean speech can be obtained using the following formula:

[0060] S=M*Y (3)

[0061] Here, * represents negative number multiplication.

[0062] 8. Extract independent speech features from the time-domain signal output of the first stage. First, extract the independent speech features from the separated speech x' of the first stage. A ,x' B Perform time-frequency domain conversion, and obtain the complex spectrograms S' of each speaker after separation by STFT transformation (as shown in formula (1)). A ,S' B Then, the speech features of each speaker are obtained through a ResNet-18 network, denoted as α. A α B The dimension is 128*1. The speech features extracted here come from the separated speech, so they represent the speaker's identity characteristics to a certain extent and can provide effective identity information for the second stage of separation.

[0063] 9. Acquire and preprocess the speaker's visual information synchronized with the mixed speech. First, read the video file and extract a 2.55s long segment. The video sampling rate is 75 frames per second, resulting in a 64-frame image sequence. Since the speaker's face, except for the lip area, doesn't change significantly within a speech range, a single facial image is randomly selected as static visual information to preserve speaker identity while reducing data processing complexity. Then, the 64-frame image sequence is cropped, and an 88*88 pixel lip area is selected to generate an h5 file containing the lip sequence as dynamic visual information.

[0064] 10. Extract static visual features containing identity information and dynamic visual features containing speech content information, respectively. First, normalize and pad the lip image to improve the accuracy and stability of the feature extraction model. The preprocessed lip data is then passed through a 3D convolutional layer, a ShuffleNet v2 network, and finally a TCN to extract time-series features. The final lip feature dimension is 512*1*64, denoted as f. lip_A f lip_B Then, the facial images are standardized and resized to accelerate the convergence of the feature extraction model. Since the facial images are color images with three channels, the preprocessed facial data size is 3*224*224. A ResNet-18 residual network is used to extract features of dimension 128*1. To fuse the lip and facial features, the facial features need to be copied in the time dimension to ensure that the converted features have the same time dimension as the lip sequence features, i.e., 128*1*64, denoted as f. face_A f face_B .

[0065] 11. Obtain the speech features of the mixed speech and perform multimodal feature fusion. First, obtain the mixed sound complex spectrogram S. mix The mixed speech feature 'mix' is obtained after the U-Net downsampling network layer, with dimensions of 512*1*64. Then, the speech features 'α' extracted from the separated speech are... A α B Transformation is performed in the time dimension to maintain consistency with the visual feature dimension, and the speaker's visual features are processed in a cascaded manner. lip_A f lip_B f face_A f face_B and sound characteristics α A α B α mix After splicing, the final fused multimodal feature dimension is 2048*1*64.

[0066] 12. The above multimodal features are processed by separation network 2 to obtain the mask M” of the separated target speech. A M” B Separation network 2 consists of upsampled network layers of U-Net.

[0067] 13. Set the target mask M” A M” B and mixed speech complex spectrum S mix Multiplying them separately yields the speaker's complex spectrum S after the second stage of separation. A ,S” B To S” A ,S” B Performing an inverse short-time Fourier transform (iSTFT) yields the time-domain speech signal x” of the target speaker. A ,x” B The calculation formula is as follows:

[0068] S” A =M” A *S mix (4)

[0069] S” B =M” B *S mix (5)

[0070] x” A =iSTFT(S”) A (6)

[0071] x” B =iSTFT(S”) B (7)

[0072] III. Dynamically Adjusting the Weights of the Two-Stage Loss Function

[0073] In this invention, the two-stage separation process and the simultaneous training of each network module are used to achieve the optimization objective. The loss function for the overall network architecture is defined as follows:

[0074] loss=λ1loss1+λ2loss2 (8)

[0075]

[0076] loss2=||M A -M' A ||+||M B -M' B || (10)

[0077] Among them, loss1 and loss2 are the training loss functions in two stages respectively, and λ1 and λ2 are the training weights of the two loss functions. For loss1, x target is defined as x noise is defined as s’ - x target , where s’ represents the separated speech signal and s represents the clean speech signal.

[0078] In order to obtain effective speaker speech features from the separated speech in the first stage and further improve the separation effect in the second stage, the present invention proposes a method for dynamically adjusting the weights of the two-stage loss function. Since at the initial state of training, the quality of the speech separated in the first stage is poor, and the quality of the corresponding speaker speech features extracted is also relatively poor. Therefore, at this time, the weights of the loss functions in the two stages are set to λ1 = λ2 = 1, and the separation networks in the two stages are trained and optimized simultaneously. As the quality of the separated speech provided by the first stage improves, the corresponding speech features are more discriminative. Therefore, at this time, the separation effect in the second stage can be significantly improved. When the separation effects in the two stages reach a threshold relationship, the weights are set to λ1 = 1 and λ2 = 2 respectively, and the separation network in the second stage is trained intensively. The threshold relationship at this time can be judged by the separation losses in the two stages. Assuming that the loss of single-stage pure audio speech separation is the negative value of the source-to-distortion ratio (SDR), defined as e0, the loss of speech separation in the first stage is the negative value of the SDR of the separated speech, defined as e1, and the loss of speech separation in the second stage is the negative value of the SDR of the speech separated in the second stage, defined as e2. When e1 - e0 < e1 - e2, it indicates that the speech separated in the first stage improves the separation effect in the second stage, and the speech features at this time are effective for the second stage. Therefore, the weight allocation is adjusted. The definition of SDR is as follows:

[0079]

[0080] Among them, s target , e interf , e noise and e artif respectively represent the speech of the target speaker, the interference generated by other speakers, the interference of noise, and the interference generated during other artificial processing.

[0081] Embodiment 2

[0082] A two-stage speech separation system based on visual guidance, comprising:

[0083] A first separation module, configured to separate the acquired mixed speech in the time domain in the first stage to obtain independent speaker speech;

[0084] The second separation module is configured to extract independent speech features with speaker information using the pure audio separation results of the first stage in the second stage. At the same time, it mines the potential correlation and complementary features between the visual and audio modalities, performs fusion of visual features and speech time-frequency domain features, and then separates them to finally obtain the separated target speech.

[0085] The dynamic weighting module dynamically adjusts the weights based on the performance of the separation model in both stages, so as to maximize the use of the independent speech features extracted in the first stage to assist the second stage and achieve clean target speaker speech separation.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0091] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A two-stage speech separation method based on visual guidance, characterized in that, Includes the following steps: In the first stage, the acquired mixed speech is separated in the time domain to obtain the time domain signal of the coarsely separated target speech. The time domain signal of the acquired target speech is then transformed to obtain the complex spectrograms of each speaker after separation, and the independent speech features of each speaker are extracted. In the second stage, the mixed speech is transformed to obtain the complex spectrum of the mixed speech, and the complex spectrum mask of the real pure speech is obtained based on it. The speaker's visual information, which is time-synchronized with the mixed speech, is acquired and preprocessed. The facial image in the preprocessed visual image is used as static visual information, and the lip data is used as dynamic visual information. Static visual features and dynamic visual features are extracted respectively. Multimodal feature fusion is performed on the speaker's static visual features, dynamic visual features, independent speech features, and mixed speech features to obtain fused multimodal features. The multimodal features are separated to obtain a mask of the separated target speech. The mask of the target speech is multiplied with the complex spectrogram of the mixed speech and then inversely transformed to obtain the clean speech signal of the target speaker. During the second stage of training, the source distortion ratio of the second stage is calculated based on the complex spectral mask of the real clean speech, and the loss function for speech separation in the second stage is determined. The separation process of the two stages is pre-trained, and the weights of the loss function of the two stages are dynamically adjusted during the training process.

2. The two-stage speech separation method based on visual guidance as described in claim 1, characterized in that, The first stage, specifically, includes the following processes: The acquired mixed speech is encoded using an encoder to extract mixed speech features; The mixed speech features are separated to obtain the mask of the target speech, the target speech features are determined, and the target speech features are decoded to obtain the time-domain signal of the coarsely separated target speech.

3. The two-stage speech separation method based on visual guidance as described in claim 2, characterized in that, The specific process of separating the mixed speech features in the first stage includes processing the mixed speech features using a first separation network. The first separation network is a temporal convolutional network structure, which includes a normalization layer and multiple identical stack modules. Each stack module consists of a fully convolutional layer, a dilated convolutional layer, and a residual module. The output of the last stack module passes through a convolutional layer and a PReLU activation layer to obtain the separated target mask.

4. The two-stage speech separation method based on visual guidance as described in claim 1, characterized in that, The specific process of converting the time-domain signal of the target speech obtained in the first stage includes: firstly, coarsely separating the speech and performing time-frequency domain conversion to obtain a complex spectrogram; then using the ResNet-18 independent speech feature extraction network to extract independent speech features from the complex spectrograms of each speaker; and finally, performing time-dimensional conversion on the independent speech features to achieve dimensional consistency between the audio and video modal features.

5. The two-stage speech separation method based on visual guidance as described in claim 1, characterized in that, The specific process of acquiring and preprocessing the speaker's visual information synchronized with the mixed speech includes reading the video file, extracting a video of a set length to obtain a multi-frame image sequence, randomly selecting a facial image as static visual information, then cropping each frame of the image sequence, selecting a lip region of a set size, and generating a lip sequence file as dynamic visual information.

6. The two-stage speech separation method based on visual guidance as described in claim 5, characterized in that, The specific process of extracting visual features includes normalizing and data filling of the lip image. The preprocessed lip data is then passed through a 3D convolutional layer, a ShuffleNet v2 network, and a temporal convolutional network structure to extract time-series features. Finally, dynamic visual features are obtained, which contain the content information of the speech. The facial images are standardized and resized, and then static visual features are extracted using a ResNet-18 residual network. These static visual features contain discriminative speaker identification information. The static visual features are then transformed to have the same time dimension as the dynamic visual sequence features.

7. The two-stage speech separation method based on visual guidance as described in claim 1, characterized in that, The specific process of multimodal feature fusion includes first passing the mixed sound spectrogram through a U-Net downsampling network layer to obtain mixed speech features, then concatenating and splicing the speaker's visual features, independent speech features, and mixed sound features to finally obtain fused multimodal features; The multimodal features are separated using a second separation network, which is an upsampling network layer of U-Net.

8. The two-stage speech separation method based on visual guidance as described in claim 1, characterized in that, The weights of the loss function for the two-stage speech separation are dynamically adjusted to maximize the use of the independent speech features from the first stage to assist the separation in the second stage.

9. A two-stage speech separation system based on visual guidance, characterized in that, include: The first separation module is configured to separate the acquired mixed speech in the time domain in the first stage to obtain the time domain signal of the coarsely separated target speech, transform the acquired target speech time domain signal to obtain the complex spectrogram of each speaker after separation, and extract the independent speech features of each speaker. The second separation module is configured to transform the mixed speech in the second stage to obtain the complex spectrum of the mixed speech, and obtain the complex spectrum mask of the real clean speech based on it. The speaker's visual information, which is time-synchronized with the mixed speech, is acquired and preprocessed. The facial image in the preprocessed visual image is used as static visual information, and the lip data is used as dynamic visual information. Static visual features and dynamic visual features are extracted respectively. Multimodal feature fusion is performed on the speaker's static visual features, dynamic visual features, independent speech features, and mixed speech features to obtain fused multimodal features. The multimodal features are separated to obtain a mask of the separated target speech. The mask of the target speech is multiplied with the complex spectrogram of the mixed speech and then inversely transformed to obtain the clean speech signal of the target speaker. During the second stage of training, the source distortion ratio of the second stage is calculated based on the complex spectral mask of the real clean speech, and the loss function for speech separation in the second stage is determined. The separation process of the two stages is pre-trained, and the weights of the loss function of the two stages are dynamically adjusted during the training process.

Images