A target speech extraction method, system and medium

Abstract

The application discloses a target speech extraction method and system and a medium, and belongs to the technical field of multi-modal speech signal processing; the method comprises the following steps: given a mixed speech and a target speaker's lip video, extracting a speech log power spectrum, a cross-channel phase difference and a visual time sequence feature; inputting the improved DPCRN network after splicing the multi-modal features, estimating a speech mask and deriving a noise mask; calculating beam forming weights through a covariance matrix and generalized eigenvalue decomposition to obtain beam features of the speech and the noise; fusing the log power spectrum, the visual features and the beam features to construct high-dimensional representations of the speech and the noise; utilizing a frame-level cross-attention mechanism to realize feature mutual exclusion enhancement, eliminating residual noise from the speech features and removing leaked speech from the noise features; finally, generating an optimized mask through a decoder and outputting pure target speech through spectral reconstruction; the application can accurately separate target speech in a complex noise environment.

Description

Technical Field

[0001] This invention relates to a method, system, and medium for accurately separating target speech in complex noisy environments. Background Technology

[0002] As the core carrier of human-computer interaction and remote communication, the transmission quality of voice signals directly affects the performance of downstream tasks and user experience. In practical applications, voice signals are highly susceptible to various types of noise interference, including environmental noise such as white noise, impulse noise, traffic noise, and industrial noise, as well as interference speech in multi-speaker scenarios. These noises are mixed into the target speech through superposition or convolution, resulting in a decrease in voice quality and intelligibility, which seriously affects the reliability of speech recognition, voice interaction and other systems.

[0003] To address these issues, speech purification technology has been continuously developing. With the maturity of deep learning technology, data-driven single-channel speech enhancement algorithms have become the mainstream research approach. These algorithms automatically learn the feature patterns of speech and noise through deep models, demonstrating good adaptability in complex noisy scenarios. However, these methods still have the following limitations: On the one hand, most existing algorithms rely solely on single audio modal features, lacking correlation modeling of the essence of speech pronunciation, making it difficult to accurately distinguish target speech from noise in low signal-to-noise ratio or strong interference scenarios. On the other hand, even though some studies have introduced visual information, they have still failed to fully achieve effective alignment and interaction of audiovisual features, and have not designed a dedicated feature separation mechanism for beamforming features. This makes it difficult to completely remove residual noise in the target speech, and the speech signal leaked from the noise components cannot be effectively filtered, ultimately affecting the precision of speech purification. In addition, existing models also suffer from insufficient generalization ability and poor adaptability to unfamiliar noise scenarios, limiting their widespread application in practical scenarios such as intelligent voice interaction, remote conferencing, and public safety monitoring.

[0004] Therefore, developing a technical solution that can fully integrate multimodal information, enhance feature interaction and separation capabilities, and thus achieve high-precision speech purification in complex noise environments has become an urgent technical problem to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a target speech extraction method that can accurately separate target speech in complex noisy environments.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a target speech extraction method, comprising the following steps: Acquire multi-channel mixed speech signals and lip video of the target speaker; extract the logarithmic power spectrum features and cross-channel phase difference features of the multi-channel mixed speech signals, and the temporal visual features of the lip video; concatenate the above three features along the channel dimension to obtain multimodal fusion features; perform audiovisual alignment processing on the multimodal fusion features to output a speech mask, and derive a noise mask from the speech mask; calculate the covariance matrices of speech and noise based on the speech mask and noise mask, respectively, and solve the optimal weights of the covariance matrices using a generalized eigenvalue decomposition beamformer to obtain speech beam features and noise beam features; The logarithmic power spectrum feature, the temporal visual feature, and the speech beam feature are fused to transform into a high-dimensional speech feature, and the logarithmic power spectrum feature, the temporal visual feature, and the noise beam feature are fused to transform into a high-dimensional noise feature. Based on the cross-attention principle, a frame-level mutually exclusive interaction logic is designed for the beam feature, so that the high-dimensional speech feature and the high-dimensional noise feature are cross-calculated at the frame level. Based on the mutual exclusion between features, residual noise in the speech feature is removed, and leaked speech components in the noise feature are removed, outputting the cleaned speech feature and noise feature. Based on the purified speech and noise features, a final ideal speech mask and noise mask are generated; the final ideal speech mask is applied to the original speech spectrum, and through time-frequency domain masking and reconstruction, a clean target speech is output.

[0007] In some optional embodiments, based on the principle of cross-attention, frame-level mutually exclusive interaction logic is designed for beam features, enabling cross-calculation of high-dimensional features on the speech side and high-dimensional features on the noise side at the frame level. Based on the mutual exclusion between features, residual noise in the speech features is removed, and leaked speech components in the noise features are removed, outputting cleaned speech features and noise features. The specific steps are as follows: Extract the first decoupling feature from the high-dimensional features on the speech side to calculate the speech side attention; extract the first decoupling feature from the high-dimensional features on the noise side to calculate the speech side attention; extract the second decoupling feature from the high-dimensional features on the speech side to calculate the noise side attention; extract the second decoupling feature from the high-dimensional features on the noise side to calculate the noise side attention. The first decoupled feature on the speech side and the first decoupled feature on the noise side are interacted, and then passed through the SoftMax layer to quantify the degree of interference of the noise feature on the speech feature, thus obtaining the speech side attention; the second decoupled feature on the speech side and the second decoupled feature on the noise side are interacted, and then passed through the SoftMax layer to quantify the discriminative power of the speech feature on the noise feature, thus obtaining the noise side attention.

[0008] A speech-side suppression mask is generated from speech-side attention to accurately label the noise components that need to be suppressed in speech features; a noise-side suppression mask is generated from noise-side attention to accurately label the speech components that need to be suppressed in noise features. Subtracting the result of element-wise multiplication of the high-dimensional features on the noise side and the noise suppression mask from the high-dimensional features on the speech side removes residual noise from the speech features and outputs the cleaned speech features; subtracting the result of element-wise multiplication of the high-dimensional features on the speech side and the speech suppression mask from the high-dimensional features on the noise side removes leaked speech components from the noise features and outputs the cleaned noise features.

[0009] In some optional embodiments, the audiovisual alignment processing of the multimodal fusion features is performed using a multimodal mask estimation network. The multimodal mask estimation network is based on the DPCRN network and a cross-modal interaction module is added to enhance the audiovisual feature alignment.

[0010] In some optional embodiments, the temporal visual features of the lip video are extracted by a Video Extractor, the Video Extractor network structure of which consists of a spatiotemporal convolutional module, a ResNet-16 network and a two-layer gated recurrent unit.

[0011] In some optional embodiments, the fusion of the logarithmic power spectrum features, the temporal visual features, and the speech beam features to transform them into high-dimensional speech features, and the fusion of the logarithmic power spectrum features, the temporal visual features, and the noise beam features to transform them into high-dimensional noise features, specifically includes: The logarithmic power spectrum features, the temporal visual features, and the speech beam features are concatenated along the channel dimension, and feature extraction and fusion are performed through an encoder and a DPRNN network to transform them into high-dimensional speech features. The logarithmic power spectrum features, the temporal visual features, and the noise beam features are concatenated along the channel dimension, and then feature extraction and fusion are performed through an encoder and a DPRNN network to transform them into high-dimensional noise-side features.

[0012] In some optional embodiments, the formulas for calculating the speech-side attention and noise-side attention are as follows: ; ; In the formula, For speech-side attention; This refers to the first decoupling feature on the speech side; This represents the first-path decoupling characteristic on the noise side. For noise-side attention; This refers to the second decoupling feature on the speech side. This is the second-path decoupling feature on the noise side; This is the feature dimension scaling factor.

[0013] In some optional embodiments, the formulas for calculating the purified speech features and noise features are as follows: ; ; In the formula, These are the purified speech features; High-dimensional features on the speech side; For speech-side suppression mask, High-dimensional features on the noise side; For noise suppression mask; The noise characteristics after purification.

[0014] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the target speech extraction method described above.

[0015] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when run by a processor, is capable of executing the above-described target speech extraction method.

[0016] The target speech extraction method provided by this invention has at least the following beneficial effects: This invention constructs a dual-modal feature space by fusing audio logarithmic power spectrum, cross-channel phase difference, and lip movement visual temporal features, thereby enhancing the distinguishability between target speech and noise from the fundamental level of speech generation. Based on the DPCRN network, a multimodal mask estimation network is innovatively designed to accurately output the ideal ratio mask of target speech and noise, providing a precise basis for subsequent separation. A cross-beam attention mechanism is used to achieve frame-level mutual exclusion interaction between speech and noise dual-branch features, removing residual noise in the speech branch and leaking speech in the noise branch, outputting purified target speech features and enhanced noise features. This forms a complete technical chain of "signal acquisition, feature extraction, fusion processing, mask generation, and speech reconstruction," which can flexibly adapt to diverse noise scenarios, ultimately significantly improving the signal-to-noise ratio and clarity of the speech signal. It meets the high-fidelity speech application needs in various scenarios such as intelligent voice interaction and remote conferencing, possessing strong practicality and generalization capabilities. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0018] Figure 1 This is a flowchart of a target speech extraction method according to an embodiment of the present invention; Figure 2 This is a diagram of a voice purification architecture provided according to an embodiment of the present invention; Figure 3 This is a noise spectrum diagram provided according to an embodiment of the present invention; Figure 4 This is a clean spectrum diagram provided according to an embodiment of the present invention; Figure 5 This is a spectrum diagram after SN-2 level-L processing provided according to an embodiment of the present invention; Figure 6 This is a spectrum diagram after SNI-2stage-v3 processing provided according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] One embodiment of the present invention relates to a target speech extraction method. The implementation details of the target speech extraction method of this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0021] The specific process of the target speech extraction method in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Acquire multi-channel mixed speech signals and lip video of the target speaker; extract the logarithmic power spectrum features and cross-channel phase difference features of the multi-channel mixed speech signals, and the temporal visual features of the lip video; concatenate the above three features along the channel dimension to obtain multimodal fusion features; perform audiovisual alignment processing on the multimodal fusion features to output a speech mask, and derive a noise mask from the speech mask; calculate the covariance matrices of speech and noise based on the speech mask and noise mask, respectively, and solve the optimal weights of the covariance matrices using a generalized eigenvalue decomposition beamformer to obtain speech beam features and noise beam features; The temporal visual features of the lip video are extracted using a Video Extractor network structure consisting of a spatiotemporal convolutional module, a ResNet-16 network, and a two-layer gated recurrent unit.

[0022] The calculation formula for beamformers is as follows: In the formula, Here, t represents the frequency point, and t represents the time frame index. For voice soft masking, For target speech in frequency The covariance matrix at that location, For noise at frequency The covariance matrix at that location.

[0023] The formula for calculating the optimal weight is as follows: In the formula, for The conjugate transpose of . This is the optimal weight vector for the beamformer.

[0024] Step 102: The logarithmic power spectrum feature, the temporal visual feature, and the speech beam feature are fused to transform into a high-dimensional feature on the speech side. The logarithmic power spectrum feature, the temporal visual feature, and the noise beam feature are fused to transform into a high-dimensional feature on the noise side. Based on the cross-attention principle, a frame-level mutually exclusive interaction logic is designed for the beam feature, so that the high-dimensional features on the speech side and the high-dimensional features on the noise side are cross-calculated at the frame level. Based on the mutual exclusion between features, residual noise in the speech feature is removed, and the leaked speech components in the noise feature are removed. The purified speech feature and noise feature are then output. The audiovisual alignment processing of the multimodal fusion features is performed using a multimodal mask estimation network. The multimodal mask estimation network is based on the DPCRN network structure, with an added cross-modal interaction module to enhance the audiovisual feature alignment.

[0025] The logarithmic power spectrum features, the temporal visual features, and the speech beam features are concatenated along the channel dimension, and feature extraction and fusion are performed through an encoder and a DPRNN network to transform them into high-dimensional speech features. The logarithmic power spectrum feature, the temporal visual feature, and the noise beam feature are concatenated along the channel dimension, and feature extraction and fusion are performed through an encoder and a DPRNN network to transform them into high-dimensional noise-side features. Extract the first decoupling feature from the high-dimensional features on the speech side to calculate the speech side attention; extract the first decoupling feature from the high-dimensional features on the noise side to calculate the speech side attention; extract the second decoupling feature from the high-dimensional features on the speech side to calculate the noise side attention; extract the second decoupling feature from the high-dimensional features on the noise side to calculate the noise side attention. The first decoupled feature on the speech side and the first decoupled feature on the noise side are interacted, and then passed through the SoftMax layer to quantify the degree of interference of the noise feature on the speech feature, thus obtaining the speech side attention; the second decoupled feature on the speech side and the second decoupled feature on the noise side are interacted, and then passed through the SoftMax layer to quantify the discriminative power of the speech feature on the noise feature, thus obtaining the noise side attention.

[0026] A speech-side suppression mask is generated from speech-side attention to accurately label the noise components that need to be suppressed in speech features; a noise-side suppression mask is generated from noise-side attention to accurately label the speech components that need to be suppressed in noise features. Subtracting the result of element-wise multiplication of the high-dimensional features on the noise side and the noise suppression mask from the high-dimensional features on the speech side removes residual noise from the speech features and outputs the cleaned speech features; subtracting the result of element-wise multiplication of the high-dimensional features on the speech side and the speech suppression mask from the high-dimensional features on the noise side removes leaked speech components from the noise features and outputs the cleaned noise features.

[0027] The formulas for calculating speech-side attention and noise-side attention are as follows: ; ; In the formula, For speech-side attention; This refers to the first decoupling feature on the speech side; This represents the first-path decoupling characteristic on the noise side. For noise-side attention; This refers to the second decoupling feature on the speech side. This is the second-path decoupling feature on the noise side; This is the feature dimension scaling factor.

[0028] The formulas for calculating the purified speech features and noise features are as follows: ; ; In the formula, These are the purified speech features; High-dimensional features on the speech side; For speech-side suppression mask, High-dimensional features on the noise side; For noise suppression mask; The noise characteristics after purification.

[0029] Step 103: Based on the purified speech features and noise features, generate the final ideal speech mask and noise mask; apply the final ideal speech mask to the original speech spectrum, and output the clean target speech through time-frequency domain masking and reconstruction.

[0030] Based on the purified speech features and noise features, the final ideal speech mask and noise mask are generated by the decoder; Voice purification architecture diagram as follows Figure 2 As shown in the figure, the speech separation technology framework integrating audio and lip movement visual information is divided into two stages: Stage 1 initially separates speech noise through multimodal feature extraction and beamforming; Stage 2 utilizes a cross-beam attention mechanism to achieve mutual exclusion suppression of speech and noise features, ultimately outputting clean speech. For logarithmic power spectrum characteristics, For cross-channel phase difference characteristics, For temporal visual features, For multimodal fusion features, For speech masking, 1- For noise mask, For speech beam characteristics, Noise beam characteristics, The fusion of the logarithmic power spectrum features, the temporal visual features, and the speech beamform features. This is a fusion of the logarithmic power spectrum features, the temporal visual features, and the noise beam features. For high-dimensional features on the speech side, For high-dimensional features on the noise side, The first decoupling feature on the speech side. This represents the first-path decoupling characteristic on the noise side. This is the second decoupling feature on the speech side. This represents the second-path decoupling characteristic on the noise side. For speech-side attention, For noise-side attention, For speech-side suppression mask, For noise suppression mask, For high-dimensional features on the speech side, The purified speech features, For an ideal speech mask, For an ideal noise mask.

[0031] To verify the effectiveness of this invention, we conducted experiments on a real-world dataset, with the following specific settings: The system is designed for smart interactive devices (such as ATMs) and collects user audio and video signals. The device is equipped with a 1080×768 pixel camera and a 4.25 cm radius circular microphone array. Training data includes 50 hours of clean speech (recorded by 50 speakers in an anechoic chamber), 200 two-minute segments of ambient noise (covering 50 scenarios such as airports and cafes), and 200 sets of room impulse response (RIR) simulations to simulate acoustic environments with spacing from 0.5 to 2 meters. The signal-to-noise ratio (SNR) range is set to [-10, -5,…, +10] dB, and the average sound pressure level (SDR) is approximately 0 dB. All signals are stored at a 16 kHz sampling rate and 16-bit precision, with a total training time of 300 hours.

[0032] The initial learning rate was 10⁻³, decreasing by 3% every 3 epochs, with a maximum training epoch of 150 epochs. The video frame rate was 25fps, and the audio sampling rate was 16 kHz. The visual input consisted of a 60×60 pixel lip region. The visual encoder was pre-trained and fixed based on the LipReading 2 dataset. The training batch size was 25.

[0033] The experimental results based on the speech sanitization architecture diagram are shown in Table 1. Specifically, "baseline" indicates that the multimodal speech separation task is completed only through the first-stage AV-DPCRN network, supplemented by GEVD beamforming. Next, "SN-2stage-S" indicates that the second-stage architecture is the same as the first stage, using beamformed speech as input features, namely logarithmic power spectrum features, temporal visual features, speech beamforming features, and noise beamforming features. For fair comparison, "SN-2stage-L" adopts the same architecture as "SN-2stage-S", but with a similar number of parameters to our proposed network. It should be noted that for "SN-2stage-S" and "SN-2stage-L", the second-stage network does not derive the interaction relationship between speech and noise features. Furthermore, the second-stage model of "SNINet" employs a dual-branch neural network architecture combined with a cross-attention mechanism.

[0034] Table 1 Experimental results based on the speech purification architecture diagram Compared to the framework using only the baseline system, the two-stage framework significantly improves accuracy, thanks to the superior performance of the second-stage network. Table 1 shows a comparison of word error rate (WER%) on the dataset for the single-stage framework, the second-stage network with a similar number of parameters to the single-stage network, the second-stage framework with a similar number of parameters to the interactive network, and the two-stage framework using the proposed interactive strategy. P: number of parameters, MBF: multiple beams, FC: feature concatenation, CBA: cross-beam attention;

[0035] To suppress residual noise, this improvement is particularly significant in low signal-to-noise ratio (SNR) scenarios. For example, comparing the "baseline" and "SN-2stage-L," under a 1m-10dB SNR condition, the noise level decreased from 39.88% to 25.62%, a reduction of 35.75%. Secondly, the effect further improves with increasing model parameters—a comparison between "SN-2 stage-S" and "SN-2stage-L" shows that the noise level at a 1m-10dB SNR decreased from 25.62% to 22.24%, a reduction of 13.19%. Finally, when the second-stage network introduces the interaction between speech and noise components, the accuracy of the backend speech recognition system significantly improves, strongly demonstrating the necessity of designing an interactive dual-branch neural network to simultaneously estimate speech and noise signals. Taking "SN-2stage-L" and "SNINet" as examples, under a signal-to-noise ratio of 1m-10dB, the noise level decreased from 22.24% to 21.12%, a reduction of 5.03%. The proposed method outperforms the baseline AV speech separation method in terms of average word error rate (AVG), with a relative reduction in WER of 28.23% (from 28.40% to 20.38%).

[0036] Table 2 shows the word error rates of the multiple versions of the interactor dataset in the cross-attention architecture. To further explore the impact of the interactor network topology on the overall framework performance, we designed ablation experiments for the interactor structure. Version 3 architecture and... Figure 2 (c) Maintaining consistency, version 1 is a simplified design of version 3. The feature representations of both speech and noise components are obtained through the same cross-attention matrix, which can be represented as:

[0037] A N =F N1FTS 1 Compared to version 1, version 2 replaces fully connected layers with convolutional layers and adds a frequency-dimensional GRU layer to the mask generator. As shown in the table, version 2 with convolutional layers performs comparably to version 2.

[0038] Table 2. Word error rates of the multi-version dataset of the interactor in the cross-attention architecture. Table 2 shows the word error rate (WER%) for multiple versions of the interactor dataset. Here, "FE" represents the feature extractor, "MG" represents the mask generator, "FC" represents the fully connected layer, and "TT" indicates whether the interactor uses a dual-tower structure.

[0039] Noise spectrum diagram as follows Figure 3As shown in the figure, the overall composition presents a rather messy red and black area. The red part is unevenly distributed and has blurred edges, showing the irregular distribution characteristics of the noise signal at different frequencies and times.

[0040] Cleaning spectrum diagram as follows Figure 4 As shown, compared to Figure 3 The red area is more regular and concentrated, the black background area is clearer, and the edges of the red part are relatively distinct, indicating that after processing, the noise in the signal is effectively suppressed and the spectral characteristics of the useful signal are more prominent.

[0041] The spectrum after SN-2 level-L processing is as follows: Figure 5 As shown in the image, there are obvious yellow and red blocky areas against the blue background. Some areas are marked with red boxes. These marked areas may be residual noise or signal distortion that still exists after this processing method. The overall signal clarity is relatively low. Figure 4 It has declined somewhat.

[0042] The spectrum after SNI-2stage-v3 processing is as follows Figure 6 As shown in the figure, there are yellow and red areas on the blue background. The areas marked by the red boxes are smaller and fewer in number compared to Figure 3, indicating that this processing method is better than SN-2 level-L in suppressing noise and preserving useful signals, and the signal quality is closer to that of SN-2 level-L. Figure 4 The clean spectrum diagram.

[0043] The red boxes highlight key differences in the IRM spectra. IRM spectra analysis reveals that, compared to the original audio, both the "SNI-2stage-v3" and "SN-2stage-L" frameworks exhibit superior interference suppression in the non-overlapping spectra. Furthermore, the "SNI-2stage-v3" framework demonstrates better performance in spectral detail recovery. This precise restoration of spectral detail is crucial for improving the correlation metric.

[0044] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0045] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0046] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0047] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for extracting target speech, characterized in that, The method includes: Acquire multi-channel mixed speech signals and lip video of the target speaker; extract the logarithmic power spectrum features and cross-channel phase difference features of the multi-channel mixed speech signals, and the temporal visual features of the lip video; concatenate the above three features along the channel dimension to obtain multimodal fusion features; perform audiovisual alignment processing on the multimodal fusion features to output a speech mask, and derive a noise mask from the speech mask; calculate the covariance matrices of speech and noise based on the speech mask and noise mask, respectively, and solve the optimal weights of the covariance matrices using a generalized eigenvalue decomposition beamformer to obtain speech beam features and noise beam features; The logarithmic power spectrum feature, the temporal visual feature, and the speech beam feature are fused to transform into a high-dimensional speech feature, and the logarithmic power spectrum feature, the temporal visual feature, and the noise beam feature are fused to transform into a high-dimensional noise feature. Based on the cross-attention principle, a frame-level mutually exclusive interaction logic is designed for the beam feature, so that the high-dimensional speech feature and the high-dimensional noise feature are cross-calculated at the frame level. Based on the mutual exclusion between features, residual noise in the speech feature is removed, and leaked speech components in the noise feature are removed, outputting the cleaned speech feature and noise feature. Based on the purified speech and noise features, a final ideal speech mask and noise mask are generated; the final ideal speech mask is applied to the original speech spectrum, and through time-frequency domain masking and reconstruction, a clean target speech is output.

2. The target speech extraction method as described in claim 1, characterized in that, Based on the principle of cross-attention, a frame-level mutually exclusive interaction logic is designed for beam features, enabling cross-calculation of high-dimensional features on the speech side and high-dimensional features on the noise side at the frame level. Based on the mutual exclusion between features, residual noise in the speech features is removed, and leaked speech components in the noise features are removed, outputting cleaned speech features and noise features. The specific steps are as follows: Extract the first decoupling feature from the high-dimensional features on the speech side to calculate the speech side attention; extract the first decoupling feature from the high-dimensional features on the noise side to calculate the speech side attention. Extract the second decoupling feature from the high-dimensional features of the speech side, and use it to calculate the noise-side attention; Extract the second decoupling feature from the high-dimensional features on the noise side, and use it to calculate the noise-side attention; The first decoupled feature on the speech side and the first decoupled feature on the noise side are interacted, and then the SoftMax layer is used to quantify the degree of interference of the noise features on the speech features to obtain the speech side attention. The second decoupled features on the speech side and the second decoupled features on the noise side are interacted, and then the SoftMax layer is used to quantify the discriminative power of the speech features on the noise features to obtain the noise-side attention. A speech-side suppression mask is generated from speech-side attention to accurately label the noise components that need to be suppressed in speech features; Noise-side suppression masks are generated from noise-side attention and used to accurately label the speech components that need to be suppressed in noise features; Subtracting the result of element-wise multiplication of the high-dimensional features on the noise side and the noise suppression mask from the high-dimensional features on the speech side removes residual noise from the speech features and outputs the cleaned speech features; subtracting the result of element-wise multiplication of the high-dimensional features on the speech side and the speech suppression mask from the high-dimensional features on the noise side removes leaked speech components from the noise features and outputs the cleaned noise features.

3. The target speech extraction method as described in claim 1, characterized in that, The audiovisual alignment processing of the multimodal fusion features is performed using a multimodal mask estimation network. The multimodal mask estimation network is based on the DPCRN network structure, with the addition of a cross-modal interaction module to enhance the audiovisual feature alignment.

4. The target speech extraction method as described in claim 1, characterized in that, The temporal visual features of the lip video are extracted using a Video Extractor network structure consisting of a spatiotemporal convolutional module, a ResNet-16 network, and a two-layer gated recurrent unit.

5. The target speech extraction method as described in claim 1, characterized in that, The process of fusing the logarithmic power spectrum features, the temporal visual features, and the speech beam features to transform them into high-dimensional speech features, and fusing the logarithmic power spectrum features, the temporal visual features, and the noise beam features to transform them into high-dimensional noise features, specifically includes: The logarithmic power spectrum features, the temporal visual features, and the speech beam features are concatenated along the channel dimension, and feature extraction and fusion are performed through an encoder and a DPRNN network to transform them into high-dimensional speech features. The logarithmic power spectrum features, the temporal visual features, and the noise beam features are concatenated along the channel dimension, and then feature extraction and fusion are performed through an encoder and a DPRNN network to transform them into high-dimensional noise-side features.

6. The target speech extraction method as described in claim 2, characterized in that, The formulas for calculating speech-side attention and noise-side attention are as follows: ； ； In the formula, For speech-side attention; The first decoupling feature on the speech side; This represents the first-path decoupling characteristic on the noise side. For noise-side attention; This refers to the second decoupling feature on the speech side; This is the second-path decoupling feature on the noise side; This is the feature dimension scaling factor.

7. The target speech extraction method as described in claim 1, characterized in that, The formulas for calculating the purified speech features and noise features are as follows: ； ； In the formula, These are the purified speech features; High-dimensional features on the speech side; For speech-side suppression mask, High-dimensional features on the noise side; For noise suppression mask; The noise characteristics after purification.

8. A computer system, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the target speech extraction method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, is capable of performing the target speech extraction method as defined in any one of claims 1 to 7.

Images