Multi-sound-source anti-noise voice sensing method and system integrating millimeter-wave radar and microphone

Abstract

The invention provides a millimeter-wave radar and microphone fused multi-sound-source anti-noise voice sensing method and system, and the method comprises the steps: S1, obtaining a millimeter-wave radar signal and a monaural microphone signal in a to-be-detected region, and carrying out the preprocessing of the obtained millimeter-wave radar signal, and obtaining a vibration displacement time-history signal meeting a preset requirement; s2, superposing the vibration displacement time history signal meeting a preset requirement with the single-track microphone signal to obtain a fusion signal; s3, constructing a speech enhancement model, and training the constructed speech enhancement model to obtain a trained speech enhancement model; s4, performing voice enhancement on the fusion signal by using the trained voice enhancement model to obtain voice signals of a plurality of target sound sources; the speech enhancement model performs speech enhancement based on a generative neural network.

Description

Technical Field

[0001] The present invention relates to the technical field of acoustic sensing and speech enhancement. Specifically, it relates to a multi-source anti-noise speech perception method and system that integrates millimeter-wave radar and microphones. More specifically, it relates to a speech enhancement method and system through the integration of millimeter-wave radar vibration monitoring technology and microphone technology. Background Art

[0002] Voice user interfaces are becoming increasingly common in daily life. With the further expansion of the current demand for non-contact interaction, voice sensing is particularly important in this application scenario. However, there is currently no miniaturized high-quality voice sensing device that can be applied to multi-target voice aliasing.

[0003] Voice sensing is a classic topic. People invented microphones to record sounds a long time ago and have been using them ever since. How to accurately extract high-quality speech in a noisy environment is an important challenge, which we call the speech enhancement problem. Traditional statistical methods can be used for the speech enhancement problem. Using specific algorithms for stationary noise usually yields good noise reduction effects, but they are usually ineffective for low signal-to-noise ratios and impulsive noise. In recent years, due to the rapid development of computer performance, many neural network-based speech enhancement methods have emerged and achieved excellent results. These methods usually make the neural network learn the mapping between the speech spectrum with noise and the clear speech spectrum, filtering out the frequency components that are not relevant to it. An important drawback of these works is that they cannot extract the one of interest from the speech signals of multiple speakers with similar signal-to-noise ratios.

[0004] The sound source localization method based on a microphone array generally uses a microphone array with a specific structure combination to obtain sound signals at multiple positions, and performs sound source localization and reconstruction through array signal processing algorithms to help separate the voices of multiple speakers and enhance the demodulated sound signals. However, microphone arrays usually require a relatively large spatial size to obtain good resolution, and the signal processing is complex, which does not meet the requirements of the miniaturization of intelligent voice devices.

[0005] In addition to microphone technology, in recent years, there have also been indirect sound perception technologies that use vibrations to restore sound, including laser microphone technology, visual microphone technology, micro-vibration sensor sound perception technology, and millimeter-wave radar sound perception technology. Compared with other technologies, millimeter-wave radar, as a wireless sensing device, can simultaneously perceive sound source targets throughout the field and extract their vibration information. Millimeter-wave radar naturally has the characteristic of source separation and can be used for sound source discrimination and regional sound pickup. In addition, since millimeter waves can penetrate through vacuum and non-conductive sound insulation materials, it has great advantages in eavesdropping scenarios compared to microphones. However, in past work, it has been found that due to the extremely weak vibration amplitude of high-frequency speech, it is difficult to be collected by millimeter-wave radar and will be submerged in background noise. In addition, compared with the mainstream audio sampling frequencies of 16,000 Hz and 22,050 Hz, the sampling rate of existing commercial millimeter-wave radars is also relatively low, which will inevitably lead to the loss of high-frequency band information in the speech collected using millimeter-wave signals. Some work has attempted to use neural networks to recover high-frequency information. The neural network learns the mapping from the downsampled radar signal to clear speech, which is a process of generating information. Even though a large amount of data is used in the training of the network, there is still a lack of prior information in the generation process, so this method may result in information forgery. Although a single millimeter-wave device can pick up the speech information of multiple targets, it is difficult to guarantee the speech quality.

[0006] The most commonly used existing voice sensing technology usually uses a single microphone and relies on deep learning technology for voice enhancement. The training and learning process of the model is usually unsupervised. The time-domain or frequency-domain signal of the microphone and the original clean voice signal are input in pairs, and the neural network learns the mapping between the two and filters out the noise. Technicians do not need to annotate the microphone data. When applied to multiple target voice aliasing or when the target voice is masked by irrelevant voices, this method cannot distinguish and enhance the target voice.

[0007] The existing microphone arrays also have a relatively complex structure, a large volume, and a high computational complexity. In addition, in multi-source sound localization and identification, in order to solve the coherence problem, microphone arrays often require significant differences in the characteristics of sound sources and it is difficult to achieve the localization of low-frequency sound sources.

[0008] The voice perception technology using a single millimeter-wave radar restores sound by sensing the vibration displacement signal of the target sound source and naturally has anti-aliasing and anti-noise performance. However, existing millimeter-wave radars are difficult to sense the weak vibrations of high-frequency voice signals, and at the same time, the sampling rate is low, and the voice signal does not have a high-frequency part, resulting in poor quality.

[0009] The method based on a laser Doppler vibrometer can only directionally measure sound signals when the sound source position is known, and it cannot accurately locate the sound source under unknown circumstances. Due to the single-point detection method, there are still significant limitations in the automatic identification of sound sources and multi-source detection by laser microphones.

[0010] Patent document CN118112557A (application number: 202211510627.2) discloses an acoustic sensing method and system that combines a millimeter-wave radar and a microphone, including: detecting a target sound source using a millimeter-wave radar to obtain an echo signal, and estimating the vibration information of the target sound source based on the echo signal; preprocessing the vibration information of the target sound source and the signal collected by the microphone to obtain respective time-frequency diagrams, and estimating the fundamental tone of the target sound source and the fundamental tone of interfering sound sources based on the time-frequency diagrams; processing the signal collected by the microphone according to the estimated fundamental tone of the target sound source and the fundamental tone of interfering sound sources, and extracting the signal of the target sound source therefrom. Summary of the Invention

[0011] Aiming at the defects in the prior art, the purpose of the present invention is to provide a multi-source noise-resistant speech perception method and system that combines a millimeter-wave radar and a microphone.

[0012] According to a multi-source noise-resistant speech perception method that combines a millimeter-wave radar and a microphone provided by the present invention, it includes:

[0013] Step S1: Obtain millimeter-wave radar signals and monaural microphone signals within the area to be measured, and preprocess the obtained millimeter-wave radar signals to obtain a vibration displacement time history signal that meets the preset requirements;

[0014] Step S2: Superimpose the vibration displacement time history signal that meets the preset requirements and the monaural microphone signal to obtain a fused signal;

[0015] Step S3: Construct a speech enhancement model, and train the constructed speech enhancement model to obtain a trained speech enhancement model;

[0016] Step S4: Use the trained speech enhancement model to enhance the fused signal to obtain speech signals of multiple target sound sources;

[0017] The speech enhancement model performs speech enhancement based on a generative neural network.

[0018] Preferably, the step S1 includes:

[0019] Step S1.1: Use the microphone facing the area to be measured to obtain a monaural microphone signal;

[0020] Step S1.2: Use the radar antenna facing the area to be measured to obtain a linear frequency modulated continuous wave microwave signal, and synchronously collect the radar baseband signals of each channel; perform a two-dimensional fast Fourier transform on the collected radar baseband signals of each channel to obtain a two-dimensional spectrum, and then calculate the amplitude of the two-dimensional spectrum to obtain a range-angle image heat map;

[0021] Step S1.3: Extract the displacement evolution time series of the targets whose amplitudes in the range-angle image heat map meet the preset requirements;

[0022] Step S1.4: Perform filtering processing on the displacement evolution time series of each extracted target to obtain a vibration displacement time history signal that meets the preset requirements.

[0023] Preferably, the step S2 includes:

[0024] Step S2.1: The vibration displacement time history signal that meets the preset requirements passes through a low-pass filter to obtain the low-frequency part of the target voice that meets the preset requirements;

[0025] Step S2.2: The monaural microphone signal passes through a high-pass filter to obtain the high-frequency part of the target voice that meets the preset requirements;

[0026] Step S2.3: Superimpose the obtained low-frequency part of the target voice that meets the preset requirements and the obtained high-frequency part of the target voice that meets the preset requirements to obtain a fusion signal.

[0027] Preferably, the step S3 includes:

[0028] Step S3.1: Construct a voice enhancement model based on the conditional diffusion model;

[0029] Step S3.2: Construct a fusion signal data set and divide it into a training set, a validation set, and a test set according to a preset ratio;

[0030] Step S3.3: Use the training set to train the voice enhancement model; during the training process, it includes: a forward process and a reverse process; in the forward process, noise is gradually added to the data to generate noise samples; in the reverse process, the noise is restored to the data by learning denoising through the conditional diffusion model;

[0031] The forward process includes:

[0032] dx t =f(x t ,y)dt+g(t)dw

[0033] where f(x t ,y) is the drift coefficient, representing the moving direction of the sample at the t-th step; g(t) is the diffusion coefficient, representing the amount of noise added to the sample at the t-th step; w represents a standard Wiener process;

[0034] The reverse process includes:

[0035]

[0036] Wherein, is called a fraction, s θ (x t , y, t) is called a speech enhancement model, represents a standard Wiener process in the reverse process; by inputting the parameters x t , y, t into the speech enhancement model, the speech enhancement model will predict the noise added in the diffusion process from the t-th step, thereby completing the reverse process of the t-th step;

[0037] Step S3.4: Input the validation set into the trained speech enhancement model to obtain the enhanced speech, evaluate the enhanced speech based on a preset evaluation metric, and further adjust the hyperparameters of the trained speech enhancement model based on the evaluation results;

[0038] Step S3.5: Use the test set to verify the adjusted speech enhancement model.

[0039] Preferably, the step S4 includes: inputting the fusion signal into the trained speech enhancement model, and completing the speech enhancement processing of the fusion signal through the reverse process of the trained speech enhancement model.

[0040] According to a multi-source anti-noise speech perception system integrating a millimeter-wave radar and a microphone provided by the present invention, it includes:

[0041] Module M1: Obtain the millimeter-wave radar signal and the monaural microphone signal in the area to be measured, and preprocess the obtained millimeter-wave radar signal to obtain a vibration displacement time history signal that meets the preset requirements;

[0042] Module M2: Superimpose the vibration displacement time history signal that meets the preset requirements and the monaural microphone signal to obtain a fusion signal;

[0043] Module M3: Construct a speech enhancement model, and train the constructed speech enhancement model to obtain a trained speech enhancement model;

[0044] Module M4: Use the trained speech enhancement model to perform speech enhancement on the fusion signal to obtain the speech signals of multiple target sound sources;

[0045] The speech enhancement model performs speech enhancement based on a generative neural network.

[0046] Preferably, the module M1 includes:

[0047] Module M1.1: Obtain a monaural microphone signal using a microphone facing the area to be measured;

[0048] Module M1.2: Obtain a linear frequency modulated continuous wave microwave signal using a radar antenna facing the area to be measured, and synchronously collect radar baseband signals of each channel; perform a two-dimensional fast Fourier transform on the collected radar baseband signals of each channel to obtain a two-dimensional spectrum, and then calculate the amplitude of the two-dimensional spectrum to obtain a range-angle image heat map;

[0049] Module M1.3: Extract the displacement evolution time series of the targets whose amplitudes in the range-angle image heat map meet the preset requirements;

[0050] Module M1.4: Perform filtering processing on the displacement evolution time series of each extracted target to obtain a vibration displacement time history signal that meets the preset requirements.

[0051] Preferably, the module M2 includes:

[0052] Module M2.1: The vibration displacement time history signal that meets the preset requirements passes through a low-pass filter to obtain the low-frequency part of the target voice that meets the preset requirements;

[0053] Module M2.2: The monaural microphone signal passes through a high-pass filter to obtain the high-frequency part of the target voice that meets the preset requirements;

[0054] Module M2.3: Superimpose the obtained low-frequency part of the target voice that meets the preset requirements and the obtained high-frequency part of the target voice that meets the preset requirements to obtain a fusion signal.

[0055] Preferably, the module M3 includes:

[0056] Module M3.1: Construct a voice enhancement model based on a conditional diffusion model;

[0057] Module M3.2: Construct a fusion signal data set and divide it into a training set, a validation set, and a test set according to a preset ratio;

[0058] Module M3.3: Use the training set to train the voice enhancement model; during the training process, it includes: a forward process and a reverse process; in the forward process, noise is gradually added to the data to generate noise samples; in the reverse process, the conditional diffusion model is used to learn denoising and restore the noise to the data;

[0059] The forward process includes:

[0060] dx t =f(x t ,y)dt+g(t)dw

[0061] where, f(x t, y) is the drift coefficient, representing the moving direction of the sample at the t-th step; g(t) is the diffusion coefficient, representing the amount of noise added to the sample at the t-th step; w represents a standard Wiener process;

[0062] The reverse process includes:

[0063]

[0064] Among them, is called a fraction, s θ (x t , y, t) is called the speech enhancement model, represents the standard Wiener process in the reverse process; by inputting the parameters x t , y, t into the speech enhancement model, the speech enhancement model will predict the noise added during the diffusion process from the t-th step, thereby completing the t-th step of the reverse process;

[0065] Module M3.4: Input the validation set into the trained speech enhancement model to obtain the enhanced speech, evaluate the enhanced speech based on preset evaluation metrics, and further adjust the hyperparameters of the trained speech enhancement model based on the evaluation results;

[0066] Module M3.5: Validate the adjusted speech enhancement model using the test set.

[0067] Preferably, the module M4 includes: inputting the fusion signal into the trained speech enhancement model, and completing the speech enhancement processing of the fusion signal through the reverse process of the trained speech enhancement model.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] 1. The present invention proposes a novel speech perception method for multi-target speech aliasing scenarios, that is, using the fusion method of millimeter-wave radar and monaural microphone to complete high-quality speech perception in multi-target aliasing and strong noise scenarios;

[0070] 2. The present invention uses a fusion signal generation method in which the low-frequency part of the target vibration displacement time history signal sensed by the millimeter-wave radar is directly superimposed on the high-frequency part of the microphone signal;

[0071] 3. The present invention uses the monaural microphone signal and the different target vibration displacement time history signals detected by the millimeter-wave radar to generate fusion signals respectively, and then performs speech enhancement based on deep learning on multiple fusion signals respectively to obtain high-quality broadband speech signals of multiple targets, so as to achieve the anti-aliasing multi-target speech perception effect;

[0072] 4. By introducing millimeter-wave radar signals and generating a fused signal with the mono microphone signal, and performing machine learning-based speech enhancement on the fused signal, the present invention can be applied in scenarios where multiple target voices are aliased, the target voice is masked by a voice with higher energy, strong noise, etc., and at the same time can solve the problem that it is difficult to integrate a microphone array into small devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Other features, objects, and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0074] Figure 1 It is a flowchart of a multi-source anti-noise speech perception method based on the fusion of millimeter-wave radar and microphone.

[0075] Figure 2 It is a schematic diagram of the structure of a machine learning diffusion model.

[0076] Figure 3 It is a schematic diagram of the structure of a multi-source anti-noise speech perception system based on the fusion of millimeter-wave radar and microphone.

[0077] Figure 4 It is a test scenario and a radar heat map.

[0078] Figure 5 It is a time-frequency diagram of a test example.

[0079] Figure 6 is a schematic diagram of test metrics and results. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0080] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0081] Embodiment 1

[0082] According to a multi-source anti-noise speech perception method based on the fusion of millimeter-wave radar and microphone provided by the present invention, as Figure 1 、 3 shown, it includes:

[0083] Step 1: The radar antenna and the microphone face the area to be measured. The microphone is turned on and records sound. The radar transmits and receives a chirp continuous wave microwave signal, and synchronously acquires the radar baseband signals of each channel to obtain the range-angle image heatmap of the radar beam radiation area. Perform a two-dimensional fast Fourier transform on the baseband signals of each channel for a certain sweep period, and then calculate the amplitude of the two-dimensional spectrum obtained by the two-dimensional fast Fourier transform to obtain the range-angle image heatmap. Extract the vibration displacement time history signal of the target with a relatively large amplitude in the radar range-angle image heatmap.

[0084] Specifically, Step 1 includes:

[0085] Step 1.1: The microphone faces the area to be measured. The microphone is turned on and records sound.

[0086] Step 1.2: The radar antenna faces the area to be measured, transmits and receives a chirp continuous wave microwave signal, and synchronously acquires the radar baseband signals of each channel to obtain the range-angle image heatmap of the radar beam radiation area.

[0087] Step 1.3: Extract the displacement evolution time series of the target with a relatively large amplitude in the radar range-angle image heatmap.

[0088] Step 1.4: Perform filtering on the displacement evolution time series signal of each extracted target to obtain an effective vibration displacement time history signal. The filtering method is high-pass or band-pass filtering;

[0089] Step 2: Generate a fused signal. Since the amplitude of high-frequency vibration is very weak, the existing commercial millimeter-wave radars cannot sense the high-frequency part of the voice signal. And due to the limitation of its sampling rate, there is a large gap with the sampling rate of standard voice of 16000 Hz or 22050 Hz. The vibration displacement time history signal collected by the millimeter-wave radar can only represent the low-frequency part of the target voice. Therefore, we use this low-frequency part to superimpose with the high-frequency part of the microphone signal, and the fused signal can filter out the noise in the low-frequency part of the microphone signal.

[0090] Specifically, Step 2 includes:

[0091] Step 2.1: The millimeter-wave vibration displacement time history signal passes through a low-pass filter. When the sampling rate of the millimeter-wave radar itself is relatively low, it can also not pass through the low-pass filter to filter out the high-frequency background noise;

[0092] Step 2.2: The microphone signal passes through a high-pass filter to filter out the aliased voice and scene noise unrelated to the target voice in the low-frequency band, and also filter out the target voice in the low-frequency band;

[0093] Step 2.3, directly superimpose the two obtained audio signals after adjusting the energy to obtain a fused signal. The low-frequency part of the fused signal is derived from the millimeter-wave radar signal. The prior information of the target speech exists in the full frequency band of the fused signal. The low-frequency noise comes from the background noise of the millimeter-wave radar vibration displacement time history signal, and the high-frequency noise comes from the aliased non-target speech and scene noise. For different target speeches in the same acquisition process, their low-frequency bands are different, while the high-frequency bands are the same.

[0094] Step 3, use a neural network to enhance the speech of the fused signal.

[0095] First, make a dataset of the fused signal:

[0096] Step 3.1, production of the millimeter-wave vibration displacement time history signal dataset. The speaker plays a clean speech dataset, and the millimeter-wave vibration displacement time history signal is recorded using the method shown in Step 1. Through an automatic program script, repeat several times to obtain a radar signal dataset.

[0097] Step 3.2, production of the microphone dataset. Add other irrelevant speeches and noise data in various scenarios to the clean speech signal to generate a microphone dataset. Through controlling the types of added noise and irrelevant speeches and the signal-to-noise ratio, data augmentation is performed to increase the diversity of training data.

[0098] Step 3.3, production of the fused signal dataset. Use the method in Step 2 to pair-make a dataset of the fused signal from the data recorded in Steps 3.1 and 3.2.

[0099] Then train and validate the neural network;

[0100] Step 3.4, neural network training. Divide the fused signal dataset into a training set, a validation set, and a test set according to experience at 80%, 15%, and 5%, and pair it with the clean speech and input it into the neural network for training. When using this method for speech enhancement in this step, various neural networks can be used, including but not limited to generative neural networks such as diffusion models, generative adversarial networks, and variational autoencoders. The diffusion model is used in this implementation process.

[0101] Step 3.5, neural network validation. After training, separately input the fused signal audio in the test set into the neural network, and then the enhanced clear speech can be obtained via the neural network. In this implementation example, multiple metrics are used to evaluate the generated speech; in this implementation example, 10 metrics in Figure 6(b) are used to evaluate the generated speech. These metrics all adopt fixed algorithms, and scores are obtained by calculating the similarity of the enhanced speech and the original clean speech in different features. The specific scores are shown in Figure 6(a).

[0102] Among them, the selection and design of the neural network. In the present invention, a diffusion model (other machine learning models that can be used for speech enhancement can also be used) is selected as the speech enhancement model for the fused signal. The diffusion model has already achieved good results in the monaural microphone speech enhancement task. The diffusion model is a type of generative model that perturbs data to pure noise by gradually adding noise and then learns the inverse process, that is, removing noise step by step from the noise to generate new data. Its process is divided into two stages: the forward process and the reverse process. The forward process gradually adds noise to the data to generate a series of intermediate states; the reverse process learns to denoise through a neural network and restores the noise to data. The diffusion model is widely used in speech generation tasks because of its high-quality generated samples and stable training. Compared with other generative models, the advantage of the diffusion model is that it can generate samples with high details and diversity. The diffusion model used in the present invention is a conditional diffusion model, and the fused signal obtained in step 2 is used as a condition to guide the direction of the diffusion process. When using this method for speech enhancement in this step, various neural networks can be used, including but not limited to generative neural networks such as diffusion models, generative adversarial networks, and variational autoencoders.

[0103] In this embodiment, as Figure 2 shown, the basic principle of the diffusion model applicable to the speech enhancement of the millimeter-wave and microphone fused signal. For the clean speech sample x0, through T steps (t ∈ [0, T]), the forward process gradually adds noise to the data sample to generate the final noise sample x T and a series of intermediate states x t ; the reverse process learns the added noise through a neural network and removes it step by step. In the figure, y is the fused signal sample generated in step 2, called the condition, which is used to guide the direction of each step of the reverse process. q(x t |x t-1 ) represents the t-th step of the forward process, and p(x t-1 |x t ) represents the t-th step of the reverse process.

[0104] For the forward process, it can be described by the formula:

[0105] dx t = f(x t , y)dt + g(t)dw

[0106] where f(x t , y) is the drift coefficient, representing the moving direction of the sample at the t-th step, g(t) is the diffusion coefficient, representing the amount of noise added to the sample at the t-th step, and w represents a standard Wiener process. By adding noise from the Wiener process, the forward process gradually converts a clean speech sample into a noise sample.

[0107] For the reverse process, it can be described by the formula:

[0108]

[0109] where is called a fraction, and s θ (x t , y, t) is called a fraction model, which is the neural network we need to train. represents the standard Wiener process in the reverse process. By passing the parameters x t , y, t into the neural network, the neural network will predict the noise added in the diffusion process at the t-th step, thus completing the reverse process at the t-th step.

[0110] Step 4, system evaluation. When the neural network training is completed, the system is put into use. Figure 4 、 5 Figures 5 and 6 respectively show materials such as the scene setup site, voice examples, and voice evaluation metrics in the system performance evaluation under the scenario of two target sound sources overlapping.

[0111] The present invention also provides a multi-source anti-noise voice perception system integrating a millimeter-wave radar and a microphone. The multi-source anti-noise voice perception system integrating a millimeter-wave radar and a microphone can be implemented by executing the process steps of the multi-source anti-noise voice perception method integrating a millimeter-wave radar and a microphone. That is, those skilled in the art can understand the multi-source anti-noise voice perception method integrating a millimeter-wave radar and a microphone as a preferred implementation manner of the multi-source anti-noise voice perception system integrating a millimeter-wave radar and a microphone.

[0112] Based on the monaural microphone voice enhancement technology, the present invention proposes a fusion enhancement method by introducing a millimeter-wave radar. Since millimeter-wave signals naturally have anti-aliasing and anti-noise capabilities when performing vibration and sound perception, the vibration displacement time history signals of multiple tested targets are separated from each other. The present invention uses the monaural microphone signal and the vibration displacement time history signals of different targets detected by the millimeter-wave radar to generate fusion signals respectively, and then performs deep learning-based voice enhancement on multiple fusion signals respectively to obtain high-quality broadband voice signals of multiple targets, so as to achieve the anti-aliasing multi-target voice perception effect.

[0113] Those skilled in the art know that in addition to implementing the system, device, and their respective modules provided by the present invention in the form of pure computer-readable program code, it is entirely possible to make the system, device, and their respective modules provided by the present invention be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system, device, and their respective modules provided by the present invention can be regarded as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structures within the hardware component.

[0114] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-source anti-noise speech perception method integrating millimeter-wave radar and microphone, characterized in that, Including: Step S1: Obtain millimeter-wave radar signals and monaural microphone signals within the area to be measured, and preprocess the obtained millimeter-wave radar signals to obtain a vibration displacement time history signal that meets the preset requirements; Step S2: Superimpose the vibration displacement time history signal that meets the preset requirements on the monaural microphone signal to obtain a fused signal; Step S3: Construct a speech enhancement model, and train the constructed speech enhancement model to obtain a trained speech enhancement model; Step S4: Use the trained speech enhancement model to enhance the speech of the fused signal to obtain speech signals of multiple target sound sources; The speech enhancement model performs speech enhancement based on a generative neural network.

2. The multi-source noise-canceling speech perception method by fusing a millimeter-wave radar and a microphone according to claim 1, wherein The said Step S1 includes: Step S1.1: Use a microphone facing the area to be measured to obtain a monaural microphone signal; Step S1.2: Use a radar antenna facing the area to be measured to obtain a chirp continuous wave microwave signal, and synchronously collect radar baseband signals of each channel; perform two-dimensional fast Fourier transform on the collected radar baseband signals of each channel to obtain a two-dimensional spectrum, and then calculate the magnitude of the two-dimensional spectrum to obtain a range-angle image heat map; Step S1.3: Extract the displacement evolution time series of the target whose magnitude in the range-angle image heat map meets the preset requirements; Step S1.4: Perform filtering processing on the displacement evolution time series of each extracted target to obtain a vibration displacement time history signal that meets the preset requirements.

3. The multi-source anti-noise voice perception method by fusing millimeter-wave radar and microphone according to claim 1, characterized in that The said Step S2 includes: Step S2.1: The vibration displacement time history signal that meets the preset requirements passes through a low-pass filter to obtain the low-frequency part of the target speech that meets the preset requirements; Step S2.2: The monaural microphone signal passes through a high-pass filter to obtain the high-frequency part of the target speech that meets the preset requirements; Step S2.3: Superimpose the obtained low-frequency part of the target speech that meets the preset requirements on the obtained high-frequency part of the target speech that meets the preset requirements to obtain a fused signal.

4. The multi-source anti-noise voice perception method by fusing millimeter-wave radar and microphone according to claim 1, characterized in that, The said Step S3 includes: Step S3.1: Construct a speech enhancement model based on a conditional diffusion model; Step S3.2: Construct a fused signal data set, and divide it into a training set, a validation set, and a test set according to a preset ratio; Step S3.3: Use the training set to train the speech enhancement model; during the training process, it includes: a forward process and a reverse process; in the forward process, gradually add noise to the data to generate noise samples; in the reverse process, learn to denoise through the conditional diffusion model and restore the noise to the data; The said forward process includes: dx t = f(x t , y)dt + g(t)dw where f(x t , y) is the drift coefficient, representing the moving direction of the sample at the t-th step; g(t) is the diffusion coefficient, representing the amount of noise added to the sample at the t-th step; w represents a standard Wiener process; The said reverse process includes: Among them, is called a fraction, s θ (x t , y, t) is called a speech enhancement model, represents the standard Wiener process in the reverse process; by passing the parameters x t , y, t into the speech enhancement model, the speech enhancement model will predict the noise added in the diffusion process at the t-th step, thereby completing the t-th reverse process; Step S3.4: Input the validation set into the trained speech enhancement model to obtain enhanced speech, evaluate the enhanced speech based on a preset evaluation index, and further adjust the hyperparameters of the trained speech enhancement model based on the evaluation results; Step S3.5: Use the test set to verify the adjusted speech enhancement model.

5. The multi-source noise-canceling speech perception method by fusing millimeter-wave radar and microphone according to claim 1, characterized in that, The said Step S4 includes: Input the fused signal into the trained speech enhancement model, and complete the speech enhancement processing of the fused signal through the reverse process of the trained speech enhancement model.

6. A multi-source noise-canceling voice perception system integrating millimeter-wave radar and microphone, characterized in that, Including: Module M1: Obtain millimeter-wave radar signals and monaural microphone signals within the area to be measured, and preprocess the obtained millimeter-wave radar signals to obtain a vibration displacement time history signal that meets the preset requirements; Module M2: Superimpose the vibration displacement time history signal that meets the preset requirements on the monaural microphone signal to obtain a fused signal; Module M3: Construct a speech enhancement model, and train the constructed speech enhancement model to obtain a trained speech enhancement model; Module M4: Use the trained speech enhancement model to enhance the speech of the fused signal to obtain speech signals of multiple target sound sources; The speech enhancement model performs speech enhancement based on a generative neural network.

7. The multi-source noise-canceling voice perception system integrating millimeter-wave radar and microphone according to claim 6, characterized in that, The Module M1 includes: Module M1.1: Obtain the monaural microphone signal using the microphone facing the area to be measured; Module M1.2: Obtain a linear frequency modulation continuous wave microwave signal using the radar antenna facing the area to be measured, and synchronously collect the radar baseband signals of each channel; perform two-dimensional fast Fourier transform on the collected radar baseband signals of each channel to obtain a two-dimensional spectrum, and then calculate the amplitude of the two-dimensional spectrum to obtain a range-angle image heat map; Module M1.3: Extract the displacement evolution time series of the target whose amplitude in the range-angle image heat map meets the preset requirements; Module M1.4: Perform filtering processing on the displacement evolution time series of each extracted target to obtain a vibration displacement time history signal that meets the preset requirements.

8. The multi-source anti-noise voice perception system integrating millimeter-wave radar and microphone according to claim 6, characterized in that, The Module M2 includes: Module M2.1: The vibration displacement time history signal that meets the preset requirements passes through a low-pass filter to obtain the low-frequency part of the target speech that meets the preset requirements; Module M2.2: The monaural microphone signal passes through a high-pass filter to obtain the high-frequency part of the target speech that meets the preset requirements; Module M2.3: Superimpose the obtained low-frequency part of the target speech that meets the preset requirements on the obtained high-frequency part of the target speech that meets the preset requirements to obtain a fused signal.

9. The multi-source noise-canceling voice perception system integrating millimeter-wave radar and microphone according to claim 6, wherein The Module M3 includes: Module M3.1: Construct a speech enhancement model based on a conditional diffusion model; Module M3.2: Construct a fused signal data set, and divide it into a training set, a validation set, and a test set according to a preset ratio; Module M3.3: Use the training set to train the speech enhancement model; during the training process, it includes: a forward process and a reverse process; in the forward process, noise is gradually added to the data to generate noise samples; in the reverse process, the conditional diffusion model is used to learn denoising and restore the noise to data; The forward process includes: dx t = f(x t , y)dt + g(t)dw where f(x t , y) is the drift coefficient, representing the moving direction of the sample at the t-th step; g(t) is the diffusion coefficient, representing the amount of noise added to the sample at the t-th step; w represents a standard Wiener process; The reverse process includes: Among them, is called a fraction, s θ (x t , y, t) is called a speech enhancement model, represents the standard Wiener process in the reverse process; by passing the parameters x t , y, t into the speech enhancement model, the speech enhancement model will predict the noise added in the diffusion process from the t-th step, thus completing the t-th reverse process; Module M3.4: Input the validation set into the trained speech enhancement model to obtain enhanced speech, evaluate the enhanced speech based on a preset evaluation index, and further adjust the hyperparameters of the trained speech enhancement model based on the evaluation results; Module M3.5: Use the test set to verify the adjusted speech enhancement model.

10. The multi-source anti-noise speech perception system integrating millimeter-wave radar and microphone according to claim 6, characterized in that, The Module M4 includes: Input the fused signal into the trained speech enhancement model, and complete the speech enhancement processing of the fused signal through the reverse process of the trained speech enhancement model.

Images