Long-distance speech enhancement method and system based on knowledge distillation, medium and terminal

Abstract

The invention provides a long-distance speech enhancement method and system based on knowledge distillation, a medium and a terminal, and the method comprises the steps: obtaining and preprocessing a target speech signal and an ultrasonic echo signal, and obtaining a noisy speech feature and an ultrasonic lip movement feature; inputting the noisy speech features and the ultrasonic lip movement features to a student model and a pre-trained teacher model; the student model and the teacher model respectively output enhanced voice signals and fusion features; constructing a total loss function according to the student enhanced voice signal, the student fusion feature, the teacher enhanced voice signal and the teacher fusion feature; performing back propagation optimization on the student model based on the total loss function; and repeatedly iteratively training the student model until convergence to obtain a far-field speech enhancement model for generating an enhanced speech signal according to the currently input to-be-enhanced audio. According to the invention, the practicability and robustness of a speech enhancement technology based on ultrasonic lip movement perception in an actual complex environment can be significantly improved.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and in particular to a long-distance voice enhancement method, system, medium, and terminal based on knowledge distillation. Background Technology

[0002] Voice interaction technology has become a key component of human-computer interaction, with applications widely covering remote work, online education, intelligent vehicle systems, and daily communication. In these applications, the quality and clarity of user voice input are fundamental prerequisites for ensuring smooth interaction and reliable system control. However, in real-world environments, voice signals are highly susceptible to various acoustic interferences, including background noise (such as keyboard clicks and vehicle noise) and interference from non-target speakers. This noise pollution leads to a series of problems, including decreased automatic speech recognition rates, deteriorated communication quality, and authentication errors, severely limiting the efficiency and user experience of voice applications. To address these challenges, voice enhancement technology has emerged. Its core objective is to recover the pure target voice component from contaminated mixed voice signals as much as possible, thereby effectively suppressing environmental noise and interfering human voices, and improving the robustness of subsequent voice processing modules.

[0003] In recent years, acoustic perception-based multimodal speech enhancement technology has attracted attention. The core idea of ​​this approach is to use high-frequency ultrasonic signals (typically above 17kHz) to sense lip movements. Specifically, an ultrasonic signal inaudible to the human ear is emitted through a speaker, and a microphone receives the echoes reflected from the lips, jaw, and other vocal organs. By processing the echo signals, features highly sensitive to minute lip movements, such as Doppler shift and phase changes, can be extracted. These features effectively reflect lip movement patterns and are largely unaffected by ambient noise in the audible frequency range. This type of acoustic perception solution requires no additional imaging equipment; the ultrasonic waves operate in the inaudible frequency range, allowing it to function even in low light or when viewed from the side, and offers a natural advantage in terms of privacy protection. Despite the promising prospects of ultrasonic perception-based multimodal speech enhancement, existing technologies still have significant limitations, restricting their application in real-world, complex scenarios.

[0004] Most current research designs and experiments are based on the "close proximity" assumption, typically requiring the distance between the speaker / microphone and the user's lips to be within 20 to 40 centimeters. Within this range, the ultrasonic echo signal-to-noise ratio is high, and lip movement features are significant. However, in many typical real-world applications, such as video conferencing with a laptop, interacting in front of a smart TV, or using a voice assistant in a vehicle environment, the interaction distance between the user and the device often reaches 0.8 meters or even more than 1.2 meters. As the distance increases, the ultrasonic signal experiences severe energy attenuation during propagation and is more susceptible to multipath reflections and environmental interference. This leads to a sharp decline in the quality of the extracted lip movement features, a significant reduction in signal-to-noise ratio and discriminative power, directly causing the enhancement performance of existing technologies to severely degrade under long-distance conditions.

[0005] Therefore, it is necessary to provide a long-distance speech enhancement method, system, medium, and terminal based on knowledge distillation to solve the above-mentioned problems in the prior art. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a long-distance speech enhancement method, system, medium and terminal based on knowledge distillation, to solve the technical problem of speech enhancement in long-distance scenarios that is limited by the prior art.

[0007] To achieve the above and other related objectives, a first aspect of this application provides a long-distance speech enhancement method based on knowledge distillation, comprising:

[0008] The system acquires the target speech signal from a distant scene captured by a microphone and the ultrasonic echo signal reflected back from the user's lips by an ultrasonic wave emitted by a speaker. The target speech signal and the ultrasonic echo signal are preprocessed to obtain noisy speech features and ultrasonic lip movement features, respectively.

[0009] The preprocessed noisy speech features and ultrasonic lip movement features are input into the student model and the pre-trained teacher model; the student model outputs the student-enhanced speech signal and student fusion features; the teacher model outputs the teacher-enhanced speech signal and teacher fusion features.

[0010] The total loss function is constructed based on the student enhanced speech signal and student fusion features output by the student model, and the teacher enhanced speech signal and teacher fusion features output by the teacher model.

[0011] The student model is optimized by backpropagation based on the constructed total loss function to update the parameters of the student model; the student model is iteratively trained until convergence to obtain the far-field speech enhancement model.

[0012] The far-field speech enhancement model is deployed to generate an enhanced speech signal based on the currently input audio to be enhanced.

[0013] In some embodiments of the first aspect of this application, the network architecture of the student model includes: a speech encoder for extracting features from the input noisy speech features to obtain speech features; an ultrasonic encoder for extracting features from the input ultrasonic lip movement features to obtain real ultrasonic features; a multimodal attention fusion module based on a memory enhancement mechanism for reconstructing pseudo-ultrasonic features based on a predefined learning memory bank and the input speech features, and generating cross-modal fusion features based on the reconstructed pseudo-ultrasonic features using a gated cross-attention mechanism; a decoder for generating a time-frequency mask based on the input cross-modal fusion features; and a spectrum enhancement module for multiplying the time-frequency mask generated by the decoder with the corresponding noisy speech features to obtain an enhanced speech signal.

[0014] In some embodiments of the first aspect of this application, the specific process of generating cross-modal fusion features based on the reconstructed pseudo-ultrasound features using the gated cross-attention mechanism includes: fusing the reconstructed pseudo-ultrasound features and the input real ultrasound features, and using the fused pseudo-ultrasound features and the real ultrasound features as keys and values; performing multi-head cross-attention calculation with the speech features as the query, and concatenating and linearly projecting the outputs of all attention to obtain attention features; generating a dynamic gating vector based on the reconstructed pseudo-ultrasound features, and performing cross-modal fusion of the speech features and the attention features based on the gated residual connection and the dynamic gating vector to obtain cross-modal fusion features.

[0015] In some embodiments of the first aspect of this application, the teacher model is constructed by training based on target speech signals and ultrasonic echo signals collected in close-range scenes, and the network architecture of the teacher model includes an encoder, a multimodal attention fusion module, a decoder, and a spectrum enhancement module.

[0016] In some embodiments of the first aspect of this application, the far-field speech enhancement model is fine-tuned based on data collected in mixed near, medium and far distance scenarios, and the distance domain is predicted based on a lightweight distance domain discriminator.

[0017] In some embodiments of the first aspect of this application, the process of preprocessing the target speech signal to obtain noisy speech features includes: filtering out high-frequency ultrasonic components and other high-frequency noise from the target speech signal based on a low-pass filter to obtain a subject audio segment; resampling the subject audio segment to unify the sampling frequency; linearly superimposing the collected noise signal with the resampled subject audio segment according to a preset signal-to-noise ratio to obtain a synthesized noisy speech signal; performing a short-time Fourier transform on the synthesized noisy speech signal to convert it to the time-frequency domain, and extracting the amplitude spectrum to obtain noisy speech features.

[0018] In some embodiments of the first aspect of this application, the process of preprocessing the ultrasonic echo signal to obtain ultrasonic lip movement characteristics includes: separating the mixed signal containing ultrasonic echoes from the mixed signal based on a high-frequency bandpass filter to obtain an ultrasonic echo signal within the frequency modulation continuous wave band; multiplying the ultrasonic echo signal within the band with a local reference transmitted signal copy and passing it through a low-pass filter to obtain a baseband complex signal containing a complex sequence; performing a discrete Fourier transform on the complex sequence within each frequency modulation cycle along the fast time dimension to obtain the distance dimension spectrum corresponding to each frequency modulation cycle; the distance dimension spectrum has multiple distance units; extracting the time series of the complex spectrum value changing with slow time for each selected distance unit, and performing first-order difference or high-pass filtering on the time series of each distance unit; decomposing the processed complex sequence into two channels, amplitude and phase, and combining them to obtain ultrasonic lip movement characteristics.

[0019] To achieve the above and other related objectives, a second aspect of this application provides a long-distance speech enhancement system based on knowledge distillation, comprising:

[0020] The acquisition module is used to acquire the target speech signal in a long-distance scene collected by the microphone and the ultrasonic echo signal reflected back by the user's lips from the ultrasonic waves emitted by the speaker, and to preprocess the target speech signal and the ultrasonic echo signal respectively to obtain noisy speech features and ultrasonic lip movement features.

[0021] The student model and teacher model output module is used to input the preprocessed noisy speech features and ultrasonic lip movement features into the student model and the pre-trained teacher model; the student model outputs the student-enhanced speech signal and student fusion features; the teacher model outputs the teacher-enhanced speech signal and teacher fusion features.

[0022] The total loss function construction module is used to construct the total loss function based on the student enhanced speech signal and student fusion features output by the student model, and the teacher enhanced speech signal and teacher fusion features output by the teacher model.

[0023] The model optimization module is used to optimize the student model by backpropagation based on the constructed total loss function, so as to update the parameters of the student model; the student model is repeatedly trained iteratively until convergence to obtain the far-field speech enhancement model;

[0024] The model deployment module is used to deploy the far-field speech enhancement model and generate an enhanced speech signal based on the currently input audio to be enhanced.

[0025] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0026] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0027] As described above, the knowledge distillation-based long-distance speech enhancement method, system, medium, and terminal of this application have the following beneficial effects:

[0028] By acquiring and preprocessing the target speech signal from a distant scene captured by a microphone and the ultrasonic echo signal reflected back from the user's lips by ultrasonic waves emitted by a speaker, noisy speech features and ultrasonic lip movement features are obtained. Then, the noisy speech features and ultrasonic lip movement features are simultaneously input into a student model and a pre-trained teacher model. The student model outputs the student-enhanced speech signal and student fusion features, while the teacher model outputs the teacher-enhanced speech signal and teacher fusion features. A total loss function is then constructed based on the student-enhanced speech signal and student fusion features, and the teacher-enhanced speech signal and teacher fusion features. Backpropagation is performed based on the constructed total loss function to optimize the student model. The student model is iteratively trained until convergence, thus obtaining a far-field speech enhancement model. Finally, the trained far-field speech enhancement model is deployed and applied to generate an enhanced speech signal based on the input audio to be enhanced. This effectively transfers the high-quality near-field prior knowledge of the teacher model to the far-field scene, significantly improving the practicality and robustness of ultrasonic lip movement perception-based speech enhancement technology in real-world complex environments. Attached Figure Description

[0029] Figure 1 The diagram shown is a flowchart of a long-distance speech enhancement method based on knowledge distillation in one embodiment of this application.

[0030] Figure 2 The diagram shows the workflow of a knowledge distillation-based long-distance speech enhancement method during the training phase in one embodiment of this application.

[0031] Figure 3 The diagram shown is a schematic representation of the network architecture of a student model in one embodiment of this application.

[0032] Figure 4 The diagram shown is a block diagram illustrating a memory enhancement mechanism in one embodiment of this application.

[0033] Figure 5 The diagram shows a performance comparison of a knowledge distillation-based long-distance speech enhancement method and a contrast method on different noise type test sets in one embodiment of this application.

[0034] Figure 6 The diagram shows a performance comparison of a knowledge distillation-based long-distance speech enhancement method and a non-distillation method on different distance test sets in one embodiment of this application.

[0035] Figure 7 The diagram shown is a block diagram of a long-distance speech enhancement system based on knowledge distillation, according to an embodiment of this application.

[0036] Figure 8 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0037] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0038] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, "first XX" and "second XX" are merely used to distinguish different XXs and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0039] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0040] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0041] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0042] <1> The Discrete Fourier Transform (DFT) is a mathematical tool that converts a time-domain signal into a frequency-domain signal. It obtains discrete sampled values ​​in the frequency domain by performing a Fourier transform on a discrete-time signal.

[0043] <2> Fast Fourier Transform (FFT) is an efficient computation method for Discrete Fourier Transform (DFT). DFT is a method to transform a signal from the time domain to the frequency domain, but directly calculating DFT is highly complex. FFT greatly simplifies the computation process by utilizing some mathematical properties of DFT, such as the periodicity and symmetry of the twitch factor, as well as the idea of ​​divide-and-conquer algorithm.

[0044] <3> Short-Time Fourier Transform (STFT) is a time-frequency analysis method used to analyze the frequency characteristics of signals that change over time. STFT obtains the frequency information of the signal at different time points by dividing the signal into small segments and then performing a Fourier transform on each segment.

[0045] <4> Inverse Short-Time Fourier Transform (ISTFT) is a signal processing technique used to convert signals represented in the frequency domain back to their time domain representation.

[0046] <5> KL divergence (KL divergence) is a statistic that measures the difference between two probability distributions. It is widely used in information theory and machine learning. It quantifies the information loss introduced when using an approximate probability distribution Q to describe the true probability distribution P. The smaller the KL divergence value, the better the distribution Q fits the distribution P. It plays an important role in model training, variational inference, and generative models.

[0047] Traditional speech enhancement methods primarily rely on prior assumptions about the statistical properties of the signal. For example, spectral subtraction recovers the speech amplitude spectrum by estimating and subtracting the noise power spectrum, but its performance is poor when the noise estimation is inaccurate; Wiener filtering constructs an optimal filter under the minimum mean square error criterion; and subspace methods attempt to separate speech and noise by projecting them into different feature subspaces. While these methods have the advantage of low computational complexity, their performance heavily depends on assumptions such as noise stationarity, and they often fail to achieve satisfactory results when dealing with complex real-world scenarios such as non-stationary noise and multi-speaker aliasing.

[0048] In recent years, with the rapid development of deep learning technology, data-driven deep neural network methods have become the mainstream research direction in the field of speech enhancement, achieving significant breakthroughs in performance. These methods can be broadly categorized into two types: frequency domain processing and time domain end-to-end processing. Frequency domain processing methods typically perform a short-time Fourier transform on noisy speech, using the resulting time-frequency spectrum as network input. The network outputs a time-frequency mask or an enhanced spectrum. Representative masks include ideal ratio masks, phase-sensitive masks, and complex ideal ratio masks processed in the complex domain. Regarding network structure, researchers have introduced fully connected networks, convolutional neural networks, recurrent neural networks, and attention-based Transformers to better model the time-frequency structure and long-term dependencies of speech. Time domain end-to-end processing methods operate directly on the original waveform, learning the mapping from noisy waveforms to clean waveforms through one-dimensional convolutional or temporal convolutional networks, such as Conv-TasNet models, avoiding explicit time-frequency transformations and phase processing issues. Furthermore, generative adversarial networks have also been introduced to improve the subjective listening quality of enhanced speech. Although deep learning-based single-modal audio methods have demonstrated strong capabilities in non-stationary noise scenarios, their performance still faces bottlenecks in scenarios with extremely low signal-to-noise ratios, uncertain target speaker identities, and strong interference from multiple people, such as "cocktail parties." These limitations make it difficult to fundamentally solve the problems of ambiguous target source assignment and excessive residual noise.

[0049] To overcome the limitations of single audio modalities, researchers began exploring the introduction of other auxiliary modalities that are related to speech content but relatively independent of environmental noise, thus forming a multimodal speech enhancement technology approach. Early mainstream research focused on audio-visual speech enhancement, which utilizes cameras to capture image sequences of the target speaker's lip and facial movements. The lip movement information provided by the visual modality is highly synchronized with the speech content, providing the network with key clues about "who is speaking" and "when is speaking," thereby significantly improving the effectiveness of speech separation and enhancement in low signal-to-noise ratio and multi-person scenarios. A series of studies, such as "Looking to Listen" and "The Conversation," have validated the effectiveness of jointly modeling audio and visual signals. However, vision-based multimodal enhancement solutions have two inherent and unavoidable drawbacks: First, to obtain effective lip movement information, it is necessary to continuously and clearly capture the user's face, which will significantly increase hardware costs and integration complexity on a large number of voice interaction devices that do not have cameras as standard (such as traditional smart speakers and in-vehicle voice terminals); Second, continuous collection of user facial videos involves significant privacy and security risks, making it difficult to be widely accepted and deployed in privacy-sensitive scenarios such as homes, medical facilities, and in-vehicle environments.

[0050] Given the aforementioned limitations of visual modalities, multimodal speech enhancement techniques based on acoustic perception have emerged. For example, UltraSE emits multi-frequency continuous waves and constructs ultrasonic time-frequency features, then combines these features with an audio spectrogram to input a conditional generative adversarial network, achieving ultrasound-assisted speech enhancement. WaveVoice, on the other hand, captures lip movements using phase and amplitude changes in ultrasonic signals and designs a lightweight end-to-end convolutional network. However, these existing methods generally lack systematic modeling and targeted optimization for mid- to long-distance scenarios, making them difficult to apply to speech enhancement acquisition in long-distance scenarios.

[0051] Based on the above analysis, it is evident that existing speech enhancement technologies, whether traditional single-audio methods or emerging multimodal methods, face significant challenges in handling long-distance interaction scenarios. Single-audio methods suffer from performance bottlenecks in complex acoustic environments; audio-visual methods are limited by hardware configuration and privacy concerns; and existing ultrasonic sensing methods are hampered by the near-range assumption and insufficient dynamic robustness. Therefore, this application provides a long-distance speech enhancement method, system, medium, and terminal based on knowledge distillation, which can overcome existing near-range limitations and maintain robust performance even when ultrasonic sensing signals attenuate or degrade, thus improving the robustness of speech enhancement and promoting the wider practical application of privacy-friendly and device-compatible next-generation voice interaction technologies.

[0052] To facilitate understanding of the embodiments of this application, in conjunction with Figure 1 and Figure 2 Detailed explanation. Figure 1A flowchart illustrating a long-distance speech enhancement method based on knowledge distillation, as described in an embodiment of the present invention, is shown. Figure 2 This diagram illustrates the workflow of the knowledge distillation-based long-distance speech enhancement method during the training phase, as described in this embodiment of the invention. The knowledge distillation-based long-distance speech enhancement method in this embodiment includes the following steps:

[0053] Step S11: Acquire the target speech signal in a long-distance scene collected by the microphone and the ultrasonic echo signal emitted by the speaker and reflected back by the user's lips, and preprocess the target speech signal and the ultrasonic echo signal respectively to obtain noisy speech features and ultrasonic lip movement features respectively.

[0054] It should be noted that, in addition to collecting mixed speech signals containing ultrasonic echoes in long-distance scenarios, mixed speech signals containing ultrasonic echoes in close-range and medium-range scenarios were also collected as a backup multimodal speech dataset.

[0055] For example, multiple volunteers were recruited and, in an acoustically controlled indoor environment, used a data acquisition device equipped with speakers and microphones to simultaneously record audio of volunteers reading a specified text at three typical distances: near distance (e.g., 0.3-0.5 meters), medium distance (e.g., 0.7-0.9 meters), and far distance (e.g., 1.1-1.3 meters). Simultaneously, the speaker of the acquisition device continuously emitted a frequency-modulated continuous wave (FMCW) ultrasonic signal with a frequency higher than 18 kHz, and the microphone simultaneously recorded a broadband signal containing ultrasonic echoes. This resulted in mixed speech signals containing ultrasonic echoes for near, medium, and far distance scenarios. The raw bandwidth signals were then processed to construct a multimodal speech dataset covering various interaction distances.

[0056] In some embodiments of this application, the process of preprocessing the target speech signal to obtain noisy speech features includes: filtering out high-frequency ultrasonic components and other high-frequency noise from the target speech signal based on a low-pass filter to obtain a subject audio segment; resampling the subject audio segment to unify the sampling frequency; linearly superimposing the collected noise signal with the resampled subject audio segment according to a preset signal-to-noise ratio to obtain a synthesized noisy speech signal; performing a short-time Fourier transform on the synthesized noisy speech signal to convert it to the time-frequency domain, and extracting the amplitude spectrum to obtain noisy speech features.

[0057] Specifically, the raw time-domain signal (target speech signal) acquired by the microphone is first filtered by a low-pass filter (e.g., cutoff frequency 8kHz) to remove high-frequency ultrasonic components and other high-frequency noise, retaining the main speech audio segments. Then, the signal is resampled to a standard speech processing sampling rate (e.g., 16kHz) to unify the sampling frequency, thus completing the initial processing of the target speech signal. To simulate a real noise environment, various background noises (e.g., white noise, crowd noise), human voice interference, and music clips are selected from publicly available datasets. These noise signals are linearly superimposed on the resampled main speech audio segments according to a preset signal-to-noise ratio (e.g., -5dB to 5dB) to obtain a synthesized noisy speech signal. The synthesized noisy speech signal is processed by Short Time Fourier Transform (STFT) to transform it into the time-frequency domain. The parameters of STFT are usually set as follows: the window function is a cosine-weighted window (such as the Hanning window), the window length is 32 milliseconds, the frame shift is 16 milliseconds, and the number of Fast Fourier Transform (FFT) points is 512. This yields the complex time-frequency spectrum, and its amplitude spectrum is taken as the speech modal input feature of the model - the noisy speech feature. .in, For time frames, The number of frequency units, It is the set of real numbers.

[0058] In some embodiments of this application, the process of preprocessing the ultrasonic echo signal to obtain ultrasonic lip movement characteristics includes: separating the mixed signal containing ultrasonic echoes from the mixed signal based on a high-frequency bandpass filter to obtain ultrasonic echo signals within the frequency modulation continuous wave band; multiplying the ultrasonic echo signals within this band with a local reference transmitted signal copy and then performing a low-pass filter to obtain a baseband complex signal containing a complex sequence; performing a discrete Fourier transform on the complex sequence in each frequency modulation cycle along the fast time dimension to obtain the distance dimension spectrum corresponding to each frequency modulation cycle; the distance dimension spectrum has multiple distance units; extracting the time series of the complex spectrum value changing with slow time for each selected distance unit, and performing first-order difference or high-pass filtering on the time series of each distance unit; decomposing the processed complex sequence into two channels, amplitude and phase, and combining them to obtain ultrasonic lip movement characteristics.

[0059] Specifically, for the received mixed signal containing ultrasonic echoes, firstly, the signal within the frequency modulated continuous wave (FMCW) band is separated using a high-frequency bandpass filter. Then, this signal is multiplied (mixed) with a locally generated copy of the reference transmitted signal, and then low-pass filtered to obtain the baseband complex signal; this process is called down-conversion. The baseband complex signal contains in-phase and quadrature components, forming a complex sequence. Next, a discrete Fourier transform is performed on this complex sequence along the fast time dimension within each chirp to obtain the range-bin spectrum, the amplitude of which reflects the reflection intensity at different distances. Then, for each selected range cell, a time series of its complex spectral values ​​changing with slow time (Chirp index) is extracted. The phase and amplitude changes in this time series encode the changes caused by lip micro-movements. To suppress static background reflection and low-frequency drift, the time series of each distance unit is subjected to first-order difference or high-pass filtering, and optionally smoothing to remove random disturbances. Finally, the data are combined to obtain a two-dimensional ultrasound lip movement feature map. Where T is the number of time frames, B is the selected number of distance units, and the two channels correspond to the processed amplitude and phase information, respectively.

[0060] Step S12: Input the preprocessed noisy speech features and ultrasonic lip movement features into the student model and the pre-trained teacher model; the student model outputs the student enhanced speech signal and student fusion features; the teacher model outputs the teacher enhanced speech signal and teacher fusion features.

[0061] In some embodiments of this application, such as Figure 3 and Figure 4 As shown, the network architecture of the student model includes: a speech encoder for extracting features from the input noisy speech features to obtain speech features; an ultrasonic encoder for extracting features from the input ultrasonic lip movement features to obtain real ultrasonic features; a multimodal attention fusion module based on a memory enhancement mechanism for reconstructing pseudo-ultrasonic features based on a predefined learning memory bank and the input speech features, and generating cross-modal fusion features based on the reconstructed pseudo-ultrasonic features using a gated cross-attention mechanism; a decoder for generating a time-frequency mask based on the input cross-modal fusion features; and a spectrum enhancement module for multiplying the time-frequency mask generated by the decoder with the corresponding noisy speech features to obtain an enhanced speech signal.

[0062] In some embodiments of this application, the specific process of generating cross-modal fusion features based on the reconstructed pseudo-ultrasound features using the gated cross-attention mechanism includes: fusing the reconstructed pseudo-ultrasound features and the input real ultrasound features, and using the fused pseudo-ultrasound features and the real ultrasound features as keys and values; performing multi-head cross-attention calculation with the speech features as the query, and concatenating and linearly projecting the outputs of all attention to obtain attention features; generating a dynamic gating vector based on the reconstructed pseudo-ultrasound features, and performing cross-modal fusion of the speech features and the attention features based on the gated residual connection and the dynamic gating vector to obtain cross-modal fusion features.

[0063] The noisy speech features obtained through the aforementioned preprocessing and ultrasound lip movement characteristics As a bimodal input, it is encoded, fused and decoded respectively, and finally outputs a time-frequency mask. Based on the output time-frequency mask, noise is filtered out from noisy speech to recover clean speech audio.

[0064] Specifically, two independent encoders are designed to perform deep feature extraction on speech modal input data and ultrasonic modal input data, respectively. The speech encoder consists of stacked convolutional layers (ConvBlock), residual blocks (ResBlock), and a bidirectional long-short-term memory (Bi-LSTM) network. The convolutional layers and residual blocks contain two-dimensional convolutions (Conv2D), batch normalization (BN) layers, and non-linear activation functions, such as ReLU. A recurrent neural network layer is then connected at the end to model long-term dependencies. This speech encoder extracts features from noisy speech. Mapped to high-level speech features Where T is the number of time frames and D is the feature dimension. The set is real numbers. Compared to speech encoders, ultrasonic encoders do not incorporate a bidirectional long short-term memory network and are specifically designed to process two-dimensional ultrasonic lip movement feature maps. Output the true ultrasound features of the same dimension Where T is the number of time frames and D is the feature dimension. It is the set of real numbers.

[0065] A multimodal attention fusion module based on memory enhancement mechanism was designed to address the issue of real ultrasonic features in long-distance scenarios. To address the issue of quality degradation, this module comprises a memory enhancement mechanism and an attention-based fusion mechanism. Specifically, a predefined trainable memory matrix pair... ,in, The key (Key, K) matrix, A value (V) matrix is ​​constructed, where N is the number of memory slots and D is the feature dimension. This learning memory learns and stores mapping patterns from speech features to high-quality ultrasound features during training. Then, ultrasound feature reconstruction is performed based on a query, specifically: inputting the current speech features... At this time, the query vector is first generated through a linear projection layer (fully connected layer). ,in, To query the transformation matrix; then, calculate the query... With memory keys in the learning memory bank The similarity is calculated and normalized to attention weights using the Softmax function. The specific formula is as follows:

[0066] ;Formula (1)

[0067] in, is the scaling factor; T is the number of time frames.

[0068] Reuse weights Memory values ​​in the learning memory bank We perform weighted summation to reconstruct pseudo-ultrasound features relevant to the current speech content. The specific formula is as follows:

[0069] ;Formula (2)

[0070] According to formula (2) Approximating high-quality ultrasound features in the feature space to compensate for potentially degraded real ultrasound features. .

[0071] To adaptively fuse information from real observations and memory reconstruction, a gated cross-attention mechanism was designed. Based on this mechanism, the pseudo-ultrasound features obtained from the reconstruction were analyzed. Generate cross-modal fusion features, specifically: first, based on pseudo-ultrasound features... For true ultrasound features Compensation is performed, and the compensated ultrasound context is used as the key (K) and value (V). The specific formula is as follows:

[0072] ;Formula (3)

[0073] Based on speech features As a query, multi-head cross-attention calculation is performed for the first... The specific formula for calculating attention is as follows:

[0074] ;Formula (4)

[0075] in, For the serial number Attention head output; For the first The query transformation matrix for each head; For the first The key transformation matrix of the head; For the first The value transformation matrix of each head; is the scaling factor; T is the number of time frames.

[0076] The outputs of all attention heads calculated according to formula (4) are concatenated and linearly projected to obtain the attention features. Simultaneously, a dynamic gating vector is introduced. This adaptively adjusts the contribution of memory information, where T is the number of time frames and D is the feature dimension. This dynamic gating vector is derived from the pseudo-ultrasonic features reconstructed from memory. The specific formula for generation is as follows:

[0077] ;Formula (5)

[0078] in, For the Sigmoid function, For the gated weight matrix, This is the gated bias vector.

[0079] Ultimately, cross-modal fusion features This is achieved through gated residual connections, which fuse speech features and attention features across modalities to obtain cross-modal fused features. The specific formula is as follows:

[0080] ;Formula (6)

[0081] The design of a multimodal attention fusion module based on memory enhancement mechanisms enables the network to... The quality (i.e. the reliability of the current ultrasound signal) dynamically adjusts the strength of memory compensation, and takes the received speech features and the context formed by the combination of real ultrasound features and pseudo ultrasound features as input to calculate the dynamic attention of speech features to the ultrasound context, thereby obtaining high-quality cross-modal fusion features.

[0082] Then fuse cross-modal features The data is fed into a decoder, which typically consists of alternating depthwise separable convolutional (DSCBlock) layers (transposed convolutions or upsampling layers) and convolutional layers, progressively upsampling the feature map to match the input noisy speech features. The same time-frequency resolution. The depthwise separable convolutional layer includes a one-dimensional depthwise convolution (DepthwiseConv1D), batch normalization (BN), ReLU function, global average pooling layer (GlobalAvgPool1D), fully connected layer (Dense), and sigmoid activation function.

[0083] Finally, the time-frequency mask is predicted through a convolutional layer and a sigmoid activation function. Where T is the number of time frames, This represents the number of frequency units. The enhanced speech amplitude spectrum is... The specific formula is as follows:

[0084] ;Formula (7)

[0085] The enhanced speech amplitude spectrum calculated according to formula (7) is combined with the estimated phase and then reconstructed by inverse short-time Fourier transform (ISTFT) to obtain the enhanced time-domain speech signal, thus obtaining clean speech audio.

[0086] In some embodiments of this application, the teacher model is constructed by training based on the target speech signal and ultrasonic echo signal collected in close-range scenes, and the network architecture of the teacher model includes an encoder, a multimodal attention fusion module, a decoder and a spectrum enhancement module.

[0087] The first stage of knowledge distillation training for long-distance scenarios involves pre-training a teacher model for near-distance scenarios. This is achieved using noisy speech features and ultrasonic lip movement features acquired in high-quality near-distance scenes (e.g., 0.4 meters) to train a base teacher model (T). The network architecture of this teacher model includes an encoder, a multimodal attention fusion module, a decoder, and a spectral enhancement module, but does not include a memory enhancement mechanism. Its goal is to learn the optimal speech enhancement mapping assisted by ideal ultrasonic features. The trained teacher model parameters are then saved. The loss function used, for example, is the scale-invariant signal-to-noise ratio (SI-SDR) loss, which directly minimizes the difference between the enhanced speech output and the clean target speech. The specific formula is as follows:

[0088] ); Formula (8)

[0089] ;Formula (9)

[0090] in, For scale-invariant signal-to-noise ratio loss; For clean target speech; To enhance voice output; T represents the number of time frames.

[0091] The trained teacher model can provide reliable fusion features and attention distribution as knowledge sources. Based on the noisy speech features and ultrasonic lip movement features collected in the input long-distance scene, it outputs the teacher-enhanced speech signal and teacher fusion features as knowledge distillation in the training stage of the student model.

[0092] In the second stage of knowledge distillation training, the complete encoder, the multimodal attention fusion module based on the memory enhancement mechanism, the decoder, and the spectral enhancement module are used to initialize the training of the memory enhancement mechanism module. Training is still performed using noisy speech features and ultrasonic lip movement features acquired in near-field scenarios. The aim is to enable the memory enhancement mechanism module to learn an effective mapping between speech and ultrasonic features. The trained module parameters are then saved. The loss function used is... Enhance the main loss for speech Alignment loss with memory The combination of these factors is shown in the following formula:

[0093] ;Formula (10)

[0094] ;Formula (11)

[0095] in, The main loss for speech enhancement (i.e., SI-SDR loss); For memory alignment loss, pseudo-ultrasound features are aligned to high-quality real ultrasound features to force memory reconstruction. To balance the hyperparameters.

[0096] Step S13: Construct the total loss function based on the student enhanced speech signal and student fusion features output by the student model, and the teacher enhanced speech signal and teacher fusion features output by the teacher model.

[0097] Step S14: Optimize the student model by backpropagation based on the constructed total loss function to update the parameters of the student model; iterate and train the student model repeatedly until convergence to obtain the far-field speech enhancement model.

[0098] The third stage of knowledge distillation training, which is the core of knowledge transfer, involves training a student model (S) based on noisy speech features and ultrasonic lip movement features collected at a distance (e.g., 1.2 meters). At this stage, the parameters of the pre-trained teacher model T are frozen. The student model S utilizes a memory enhancement mechanism module obtained from its second-stage training to compensate for the degraded ultrasonic input in the far field and receives multi-level distillation supervision. In other words, the multimodal attention fusion module based on the student model and its memory enhancement mechanism solves the problem of long-distance ultrasonic cue degradation. The loss function for this stage is designed as follows:

[0099] First, design the output distillation loss function. This allows student output to approximate teacher output (soft objective), and the specific formula is as follows:

[0100] ;Formula (12)

[0101] in, Enhance the student's speech signal output by the student model; Enhanced speech signal of the teacher output for the teacher model.

[0102] Design another characteristic distillation loss function This makes the student integration characteristics approximate the teacher integration characteristics, as shown in the following formula:

[0103] ;Formula (13)

[0104] in, To integrate students' characteristics; This is a characteristic of teacher integration.

[0105] Design another attention distillation loss function KL divergence is used to approximate the teacher's attention distribution in the student model. The specific formula is as follows:

[0106] ;Formula (14)

[0107] in, The attention distribution for the student model; The attention distribution of the teacher model; For temperature parameters; KL divergence is used to measure the difference between two probability distributions.

[0108] Design another memory consistency loss function In far-field scenarios, continue to constrain pseudo-ultrasound features. Compared to the current (albeit degraded) true ultrasound features To maintain a certain level of similarity, the specific formula is as follows:

[0109] ;Formula (15)

[0110] Based on the distillation loss function and the self-supervised enhancement loss of the student model according to formulas (12), (13), (14) and (15) The weighted sum is used to calculate the total loss function of the student model. The specific formula is as follows:

[0111] ;Formula (16)

[0112] in, , , , These are weighted hyperparameters.

[0113] The student model is optimized by backpropagation based on the total loss function constructed according to formula (16), thereby updating the parameters of the student model and minimizing the total loss. After continuous iterative training until convergence, the far-field speech enhancement model is finally obtained.

[0114] In some embodiments of this application, the far-field speech enhancement model is fine-tuned based on data collected in mixed near, medium, and far-distance scenarios, and the distance domain is predicted based on a lightweight distance domain discriminator.

[0115] The fourth stage of knowledge distillation training involves multi-distance mixing fine-tuning and domain adaptation. To further improve the model's generalization ability across different distances, samples collected from near, medium, and far distance scenarios are mixed for final fine-tuning, enabling the student model to generalize to arbitrary distances. A lightweight distance domain discriminator is also introduced at this stage. Attempting to extract features from fusion The algorithm first determines the source distance of the discriminant samples. During the main network training, a gradient inversion layer encourages the encoder to generate features that confuse the discriminant, thereby learning a distance-invariant robust representation. This adversarial loss is represented as follows: At the same time, it can increase the consistency loss across distances. The goal is to constrain the fusion features of the same speech content to be as similar as possible at different distances, and the final overall optimization objective is:

[0116] ;Formula (17)

[0117] in, , , , These are the weighting coefficients for each type of loss.

[0118] Through the fourth stage of training, the model is ultimately optimized into a unified and robust speech enhancement system capable of adaptively handling continuous distance variations from the near field to the far field. The four-stage training framework designed in this application transfers knowledge from the near-field model with superior signal quality to the far-field model that needs to cope with signal attenuation, thereby enabling the model to possess far-field robustness.

[0119] Step S15: Deploy the far-field speech enhancement model to generate an enhanced speech signal based on the currently input audio to be enhanced.

[0120] By deploying and applying the far-field speech enhancement model obtained through four-stage training, a clean speech signal can be generated based on the current input audio to be enhanced. This overcomes the shortcomings of existing technologies where the performance of ultrasonic signals attenuates sharply with distance or is subject to interference in long-distance scenarios. Furthermore, robust speech enhancement is achieved through an end-to-end multimodal network that integrates memory enhancement mechanisms and a knowledge distillation training framework.

[0121] In the inference phase, the deployed far-field speech enhancement model acquires mixed signals through a microphone for the audio to be enhanced, processes the synchronously acquired ultrasonic echo signals to obtain an ultrasonic feature map, and processes the signal to output a speech feature representation based on the encoder. Then, it passes through a pre-trained multimodal attention fusion module based on a memory enhancement mechanism to generate cross-modal fusion features. The model uses speech features and the ultrasonic features generated and combined by memory to perform cross-modal fusion. The fused features are processed by the decoder to generate the final time-frequency mask. The generated mask is multiplied point-by-point with the input noisy amplitude spectrum to obtain the enhanced speech amplitude spectrum. Then, the enhanced time-domain speech signal is reconstructed through inverse short-time Fourier transform (ISTFT) to complete the entire enhancement process.

[0122] To verify the effectiveness of the knowledge distillation-based long-distance speech enhancement method provided in this application, a systematic experimental evaluation was conducted. The test set included noise under various noise conditions (ambient noise, speaker interference noise, and music noise). Figure 5As shown, in terms of objective evaluation metrics, SDR (Signal-to-Distortion Ratio), PESQ (Perceptual Evaluation of Speech Quality), and STOI (Short-Time Objective Intelligibility), the knowledge distillation-based long-distance speech enhancement method provided in this application demonstrates significant and consistent performance improvements over traditional pure audio deep enhancement methods (SEGAN, MetricGAN+, PHASEN, MEVoice, etc.) under all noise types, especially when dealing with human voice interference and mixed noise.

[0123] To verify the robustness of the knowledge distillation-based long-range speech enhancement method provided in this application under real-world interference scenarios for ultrasonic signals, we tested its performance under medium- and long-range conditions. Figure 6 As shown, compared to the No-KD (No-Knowledge Distillation) model that uses non-distilled long-distance data, the long-distance speech enhancement method based on knowledge distillation provided in this application benefits from its long-distance training design mechanism, and its performance remains stable, significantly outperforming the comparison method.

[0124] Furthermore, current research based on acoustic perception typically relies on the idealized assumption that the target user is relatively stationary and facing the device. However, in real-world applications, users may exhibit natural movements such as head rotation and body swaying. These macroscopic movements can cause signal changes in the ultrasound echo that are far greater than subtle lip movements, thus masking crucial articulation cues and disrupting the temporal alignment between perceived features and speech content, severely impacting enhancement effectiveness. Although some studies, such as the MUS model, have introduced an "ultrasound memory" module to generate pseudo-ultrasound features by retrieving historical patterns from audio features when ultrasound signals are missing, these methods often employ relatively simple feature fusion strategies and lack a mechanism for adaptively balancing the contributions of real perceived signals and memory-reconstructed signals at a fine-grained temporal and frequency level. When ultrasound signals degrade to varying degrees, model performance may become unstable or experience a precipitous decline. Another study investigated the USpeech model to reduce its dependence on real ultrasound data through cross-modal synthesis, but the distributional differences between synthesized and real features, as well as the model's adaptability to mixed modal states, still require improvement. To address the insufficient robustness of existing methods in dynamic scenarios, this application provides a knowledge distillation-based long-distance speech enhancement method that can improve robustness in dynamic scenarios, thereby enhancing the performance of speech enhancement.

[0125] The proposed knowledge distillation-based long-distance speech enhancement method effectively addresses the core issue of severe performance degradation in long-distance scenarios by introducing a cross-modal fusion module with a memory enhancement mechanism and a four-stage distillation training framework. Specifically, the memory enhancement mechanism can adaptively reconstruct effective lip movement features based on speech content when the quality of the ultrasonic signal degrades due to distance attenuation or occlusion, significantly improving robustness under signal degradation conditions. Furthermore, combined with the knowledge distillation strategy, the powerful multimodal fusion capability learned by the near-field teacher model is transferred to the far-field student model, enabling it to maintain excellent enhancement performance even under weak ultrasonic cues. The resulting long-field speech enhancement model can still achieve high-quality, highly intelligible target speech recovery at far-field distances of up to 1.2 meters and in complex noisy environments, thus broadening the practical application boundaries of the technology.

[0126] Figure 7 This is a schematic block diagram of a long-distance speech enhancement system based on knowledge distillation provided in an embodiment of this application. Figure 7 As shown, the knowledge distillation-based long-distance speech enhancement system 700 includes:

[0127] The acquisition module 701 is used to acquire the target speech signal in a long-distance scene collected by the microphone and the ultrasonic echo signal reflected back by the user's lips from the ultrasonic wave emitted by the speaker, and to preprocess the target speech signal and the ultrasonic echo signal respectively to obtain noisy speech features and ultrasonic lip movement features respectively.

[0128] The student model and teacher model output module 702 is used to input the preprocessed noisy speech features and ultrasonic lip movement features into the student model and the pre-trained teacher model; the student model outputs the student enhanced speech signal and student fusion features; the teacher model outputs the teacher enhanced speech signal and teacher fusion features;

[0129] The total loss function construction module 703 is used to construct a total loss function based on the student enhanced speech signal and student fusion features output by the student model and the teacher enhanced speech signal and teacher fusion features output by the teacher model.

[0130] The model optimization module 704 is used to optimize the student model by backpropagation based on the constructed total loss function, so as to update the parameters of the student model; the student model is repeatedly trained iteratively until convergence to obtain the far-field speech enhancement model;

[0131] The model deployment module 705 is used to deploy the far-field speech enhancement model and to generate an enhanced speech signal based on the currently input audio to be enhanced.

[0132] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0133] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0134] Figure 8 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 8 As shown, the electronic terminal 800 includes at least one processor 801, a memory 802, at least one network interface 803, and a user interface 805. The various components in the electronic terminal 800 are coupled together via a bus system 804. It is understood that the bus system 804 is used to implement communication between these components. In addition to a data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 8 The general will label all buses as bus systems.

[0135] The user interface 805 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0136] It is understood that memory 802 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0137] In this embodiment of the invention, the memory 802 is used to store various types of data to support the operation of the electronic terminal 800. Examples of this data include: any executable program for operation on the electronic terminal 800, such as the operating system 8021 and application programs 8022; the operating system 8021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 8022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The method provided in this embodiment of the invention can be included in the application program 8022.

[0138] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 801 or by instructions in software form. The processor 801 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 801 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 801 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0139] In an exemplary embodiment, the electronic terminal 800 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0140] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figures 1 to 6 The method of any of the embodiments shown.

[0141] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0142] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0143] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0148] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] In summary, addressing the limitations of existing technologies in speech enhancement for long-distance scenarios, this application provides a knowledge distillation-based method, system, medium, and terminal for long-distance speech enhancement. It acquires and preprocesses target speech signals from a microphone in a long-distance scenario, as well as ultrasonic echo signals reflected back from the user's lips by ultrasonic waves emitted from a speaker, thereby obtaining noisy speech features and ultrasonic lip movement features. Then, the noisy speech features and ultrasonic lip movement features are simultaneously input into a student model and a pre-trained teacher model. The student model outputs the student-enhanced speech signal and student fusion features, while the teacher model outputs the teacher-enhanced speech signal and teacher fusion features. The system first identifies the combined features of the student and teacher enhanced speech signals. Then, it constructs a total loss function based on the student's enhanced speech signal and student fusion features, and the teacher's enhanced speech signal and teacher fusion features. Backpropagation is then performed to optimize the student model based on this total loss function. The student model is iteratively trained until convergence, resulting in a far-field speech enhancement model. Finally, the trained far-field speech enhancement model is deployed and applied to generate enhanced speech signals based on the input audio to be enhanced. This effectively transfers the high-quality near-field prior knowledge of the teacher model to the far-field scenario, significantly improving the practicality and robustness of ultrasonic lip movement sensing-based speech enhancement technology in complex real-world environments. Therefore, this application effectively overcomes the shortcomings of existing technologies and has high industrial application value.

[0150] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A long-distance speech enhancement method based on knowledge distillation, characterized in that, include: The system acquires the target speech signal from a distant scene captured by a microphone and the ultrasonic echo signal reflected back from the user's lips by an ultrasonic wave emitted by a speaker. The target speech signal and the ultrasonic echo signal are preprocessed to obtain noisy speech features and ultrasonic lip movement features, respectively. The preprocessed noisy speech features and ultrasonic lip movement features are input into the student model and the pre-trained teacher model; the student model outputs the student-enhanced speech signal and student fusion features; the teacher model outputs the teacher-enhanced speech signal and teacher fusion features. The total loss function is constructed based on the student enhanced speech signal and student fusion features output by the student model, and the teacher enhanced speech signal and teacher fusion features output by the teacher model. The student model is optimized by backpropagation based on the constructed total loss function to update the parameters of the student model; the student model is iteratively trained until convergence to obtain the far-field speech enhancement model. The far-field speech enhancement model is deployed to generate an enhanced speech signal based on the currently input audio to be enhanced.

2. The long-distance speech enhancement method based on knowledge distillation according to claim 1, characterized in that, The network architecture of the student model includes: A speech encoder is used to extract features from the input noisy speech features to obtain speech features; An ultrasonic encoder is used to extract features from the input ultrasonic lip movement features in order to obtain real ultrasonic features; A multimodal attention fusion module based on a memory enhancement mechanism is used to reconstruct pseudo-ultrasound features based on the input speech features using a predefined learning memory bank, and to generate cross-modal fusion features based on the reconstructed pseudo-ultrasound features using a gated cross-attention mechanism. A decoder is used to generate a time-frequency mask based on the input cross-modal fusion features; The spectrum enhancement module is used to multiply the time-frequency mask generated by the decoder with the corresponding noisy speech features to obtain the enhanced speech signal.

3. The long-distance speech enhancement method based on knowledge distillation according to claim 2, characterized in that, The process of generating cross-modal fusion features based on the gated cross-attention mechanism according to the reconstructed pseudo-ultrasound features includes: The reconstructed pseudo-ultrasound features and the input real ultrasound features are fused together, and the fused pseudo-ultrasound features and real ultrasound features are used as keys and values. Multi-head cross-attention calculation is performed using the aforementioned speech features as the query, and the outputs of all attention are concatenated and linearly projected to obtain the attention features; A dynamic gating vector is generated based on the reconstructed pseudo-ultrasound features, and the speech features and attention features are fused across modally based on the gating residuals and the dynamic gating vector to obtain cross-modal fused features.

4. The long-distance speech enhancement method based on knowledge distillation according to claim 1, characterized in that, The teacher model is constructed by training based on the target speech signal and ultrasonic echo signal collected in close-range scenes, and the network architecture of the teacher model includes an encoder, a multimodal attention fusion module, a decoder and a spectrum enhancement module.

5. The long-distance speech enhancement method based on knowledge distillation according to claim 1, characterized in that, The far-field speech enhancement model is fine-tuned based on data collected in mixed near, medium and far distance scenarios, and the distance domain is predicted based on a lightweight distance domain discriminator.

6. The long-distance speech enhancement method based on knowledge distillation according to claim 1, characterized in that, The process of preprocessing the target speech signal to obtain noisy speech features includes: The target speech signal is filtered by a low-pass filter to remove high-frequency ultrasonic components and other high-frequency noise to obtain the subject audio segment; and the subject audio segment is then resampled to unify the sampling frequency. The collected noise signal is linearly superimposed with the resampled subject audio segment according to the preset signal-to-noise ratio to obtain a synthesized noisy speech signal. The synthesized noisy speech signal is subjected to a short-time Fourier transform to convert it to the time-frequency domain, and the amplitude spectrum is extracted to obtain the noisy speech features.

7. The long-distance speech enhancement method based on knowledge distillation according to claim 1, characterized in that, The process of preprocessing the ultrasonic echo signal to obtain ultrasonic lip movement characteristics includes: The ultrasonic echo signal within the frequency modulated continuous wave band is obtained by separating the mixed signal containing ultrasonic echo based on the high-frequency bandpass filter. The ultrasonic echo signal in this frequency band is multiplied by a local reference transmitted signal copy and then low-pass filtered to obtain a baseband complex signal containing a complex sequence. A discrete Fourier transform is performed along the fast time dimension on the complex sequence within each frequency modulation cycle to obtain the distance-dimensional spectrum corresponding to each frequency modulation cycle; the distance-dimensional spectrum has multiple distance units. For each selected distance cell, extract the time series of complex spectral values ​​as a function of slow time, and perform first-order difference or high-pass filtering on the time series of each distance cell. The processed complex sequence is decomposed into two channels: amplitude and phase, and then combined to obtain the ultrasonic lip movement characteristics.

8. A long-distance speech enhancement system based on knowledge distillation, characterized in that, include: The acquisition module is used to acquire the target speech signal in a long-distance scene collected by the microphone and the ultrasonic echo signal reflected back by the user's lips from the ultrasonic waves emitted by the speaker, and to preprocess the target speech signal and the ultrasonic echo signal respectively to obtain noisy speech features and ultrasonic lip movement features. The student model and teacher model output module is used to input the preprocessed noisy speech features and ultrasonic lip movement features into the student model and the pre-trained teacher model; the student model outputs the student-enhanced speech signal and student fusion features; the teacher model outputs the teacher-enhanced speech signal and teacher fusion features. The total loss function construction module is used to construct the total loss function based on the student enhanced speech signal and student fusion features output by the student model, and the teacher enhanced speech signal and teacher fusion features output by the teacher model. The model optimization module is used to optimize the student model by backpropagation based on the constructed total loss function, so as to update the parameters of the student model; the student model is repeatedly trained iteratively until convergence to obtain the far-field speech enhancement model; The model deployment module is used to deploy the far-field speech enhancement model and generate an enhanced speech signal based on the currently input audio to be enhanced.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1 to 7.

Images