Millimeter wave radar voice reconstruction and recognition method based on physical guidance network
By constructing a multimodal speech reconstruction network and combining phase stabilization processing and glottal parameters, the problem of reconstructing millimeter-wave radar speech signals in high-noise environments was solved, achieving high-precision and robust speech reconstruction and recognition, and meeting the needs of non-contact speech acquisition.
Patent Information
- Application Number
- CN202610058181.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-16
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies struggle to effectively acquire and reconstruct speech signals in high-noise, obstructed environments. Static background clutter and systematic phase noise in millimeter-wave radar signals make speech feature extraction difficult, and traditional methods are insufficient to meet the requirements for long-distance, non-contact speech acquisition.
A physical guidance network-based approach is adopted. By constructing a multimodal speech reconstruction network and combining phase stabilization processing, drift removal, syllable structure and glottal parameters, the multimodal features of millimeter-wave radar signals and audio signals are fused to perform non-contact reconstruction and recognition of speech signals.
It achieves high-precision speech reconstruction in high-noise and occlusion environments, improves the robustness and interpretability of speech reconstruction, expands the diversity of training sets, improves the clarity and individual consistency of speech reconstruction, and supports contactless speech communication and recognition.
Smart Images
Figure CN121884828A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of millimeter-wave radar signal processing and speech reconstruction technology, and in particular to a method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network. Background Technology
[0002] Speech signals are the core information carrier in human-computer interaction, speech recognition, and communication systems. Traditional speech acquisition typically relies on microphone arrays to capture speech by receiving sound pressure waves through the air medium. However, in environments with high noise, strong echoes, or obstructed sound sources, acoustic signals are easily interfered with, significantly reducing the signal-to-noise ratio, and may even prevent the acquisition of effective speech information. Furthermore, microphone methods depend on the sound wave propagation path and are significantly affected by distance attenuation, making them unsuitable for long-distance, non-contact speech acquisition.
[0003] Millimeter-wave radar, as an active electromagnetic sensor, offers advantages such as non-contact operation, resistance to obstruction, and strong noise immunity. The short wavelength of millimeter waves allows for the detection of sub-millimeter displacements, thus enabling the capture of minute vibration signals from the human throat caused by vocal cord vibration and changes in oral airflow. By observing changes in the phase of the radar echo, the physical characteristics of vocal activity can be indirectly characterized, providing a new approach for non-contact acquisition of speech signals. This characteristic holds significant application potential in scenarios such as privacy protection, medical monitoring, and voice communication in noisy environments.
[0004] However, millimeter-wave radar signals contain a large amount of static background clutter and systematic phase noise, and the amplitude of speech-related vibration signals is much lower than that of the noise background, making direct extraction of speech features quite difficult. The micro-displacement signals during the vocalization process exhibit multi-scale, non-stationary characteristics in the time-frequency domain, containing not only the fundamental frequency vibration of the vocal cords but also low-frequency components such as body and oral resonances, making it difficult for traditional filtering and component decomposition methods to accurately separate effective speech information. Existing research mainly focuses on the detection of low-frequency physiological signals such as respiration and heartbeat monitoring in millimeter-wave radar. These signals have simple patterns and low frequencies, making it impossible to form complex time-frequency patterns of speech or reconstruct high-fidelity speech. Therefore, there is an urgent need for an algorithmic system capable of effectively preprocessing, suppressing clutter, and reconstructing speech features from millimeter-wave radar signals, achieving accurate mapping of radar signals to audio signals and speech reconstruction. Summary of the Invention
[0005] In view of the above-mentioned deficiencies in the prior art, this invention provides a millimeter-wave radar speech reconstruction and recognition method based on a physical guidance network. This method achieves non-contact speech recovery and reconstruction by analyzing micro-vibration signals in the human throat area, offering advantages such as high precision, strong robustness, and privacy protection.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A method for speech reconstruction and recognition using millimeter-wave radar based on a physical guidance network includes the following steps:
[0008] S1. Use millimeter-wave radar to transmit radio frequency signals to the target under test and receive echo signals, while simultaneously acquiring reference audio signals; S2. Extract the steady-state signal Mel spectrum based on the echo signals;
[0009] S3. Generate a simulated radar Mel spectrum by simulating the reference audio signal and the public speech dataset; synchronize and standardize the steady-state signal Mel spectrum and the simulated radar Mel spectrum to construct a speech signal dataset; S4. Construct a multimodal speech reconstruction network model; train the multimodal speech reconstruction network model based on the speech signal dataset; input the newly acquired real millimeter-wave radar signal into the trained multimodal speech reconstruction network model to reconstruct the speech and output a non-contact speech Mel spectrum;
[0010] S5. Input the speech Mel spectrogram into the constructed lightweight convolutional neural network classifier and output the speaker's identity category label.
[0011] Preferably, S2 includes:
[0012] S21. After preprocessing the echo signal, perform a fast Fourier transform in the distance dimension and perform background subtraction.
[0013] S22. Perform slow-time background modeling and removal on the echo signal after background subtraction;
[0014] S23. Construct a spatial covariance matrix based on the multi-channel received signal after slow-time background removal, and use a beamforming algorithm for spatial filtering to generate a beamforming output signal.
[0015] S24. Perform phase demodulation on the beamforming output signal. Remove system drift to obtain a stable signal.
[0016] Preferably, S2 also includes detecting speech segment boundaries based on periodic changes of steady-state signals, performing syllable segmentation through adaptive thresholding or energy envelope analysis, and performing periodic enhancement fitting on each signal segment to obtain syllable structure and time boundaries.
[0017] Preferably, S3 includes:
[0018] S31. Obtain a public speech dataset; perform resampling and frame-segmentation processing on the reference audio signal and the public speech dataset to generate a resampled signal;
[0019] S32. Perform envelope extraction and bandpass filtering on the resampled signal to generate a simulated vibration signal;
[0020] S33. Superimpose Gaussian white noise onto the simulated vibration signal to generate a noise-added simulated radar vibration signal;
[0021] S34. Perform a short-time Fourier transform and map the noise-added simulated radar vibration signal to a Mel filter bank to generate a simulated radar Mel spectrum.
[0022] As a preferred option, S3 also includes:
[0023] S35. Perform linear prediction analysis on the reference audio signal and obtain the glottal residual signal through inverse filtering; based on the spectral analysis of the glottal residual signal, extract the fundamental frequency... The glottal physical parameters include the first harmonic amplitude, the second harmonic amplitude, glottal parameters, and airflow parameters.
[0024] As a preferred option, S3 also includes:
[0025] S36. Calculate the energy envelope of the steady-state signal Mel spectrum and the simulated radar Mel spectrum and perform correlation matching to calculate the optimal time shift; align the steady-state signal Mel spectrum and the simulated radar Mel spectrum according to the optimal time shift to achieve time synchronization between the reference audio signal and the echo signal.
[0026] S37. Based on the time-aligned steady-state signal Mel spectrum and the simulated radar Mel spectrum, construct a speech signal dataset containing radar modes and audio modes.
[0027] Preferably, S36 includes: performing energy convergence on the Mel spectrum of the steady-state signal and the Mel spectrum of the simulated radar along the frequency dimension to obtain the corresponding time energy sequences; calculating the normalized correlation coefficient of the time energy sequences at both ends under time shift; constructing a cost function based on the normalized correlation coefficient; and obtaining the optimal time shift between the audio signal and the radar signal by minimizing the cost function.
[0028] Preferably, S4 includes
[0029] S41. Construct a physically guided multimodal reconstruction network model including an encoder, a cross-modal attention module, a decoder, and a discriminator;
[0030] The steady-state signal Mel spectrum is input into the encoder to extract multi-scale time-frequency features of the millimeter-wave radar signal; the glottal physical parameters are used as conditional input to physically guide the speech generation process; the steady-state signal Mel spectrum and the glottal physical parameters are fused using a cross-modal attention module to generate conditional features; the decoder restores the conditional features to the speech Mel spectrum; the discriminator uses a convolution stacked structure to jointly discriminate the steady-state signal Mel spectrum and the conditional features.
[0031] As a preferred option, S4 also includes:
[0032] S42. By minimizing the difference between the speech Mel spectrogram and the true speech spectrum of the reference audio signal, the joint loss function is constructed as follows:
[0033]
[0034] in, To counteract the loss, For feature matching loss term, For the total variational regularization term , and These represent the weight coefficients of the corresponding loss terms; an R1 gradient penalty term is introduced in each training batch to constrain the gradient magnitude of the discriminator.
[0035] Preferably, S4 further includes: the speech Mel spectrogram is reconstructed by inverse Mel transform and inverse short-time Fourier transform to generate the final speech waveform; in S5, the real speech waveform under the reference audio signal and the final speech waveform are respectively input into a lightweight convolutional neural network classifier, and the identity preservation ability of the multimodal reconstruction network model is evaluated by comparing the recognition results of the two.
[0036] Compared with the prior art, the beneficial effects of the present invention are reflected in:
[0037] (1) Unlike traditional schemes that rely on deep models, this invention introduces physical modeling such as phase stabilization, drift removal, syllable structure and glottal parameters as network conditions to guide the generative model to stably learn the speech mechanism, so that speech reconstruction under weak radar echo has stronger robustness, interpretability and generalization ability.
[0038] (2) Unlike the traditional approach of directly mapping radar signals with deep networks, which makes it difficult to guarantee time-frequency consistency, this invention adopts a combination of phase stabilization processing, drift removal and syllable segmentation, and introduces a cross-modal attention fusion technique, which makes the mapping of radar to speech spectrum more accurate and significantly improves the clarity and intelligibility of reconstructed speech.
[0039] (3) Unlike training methods that rely solely on limited real millimeter-wave data, this invention constructs simulated samples based on acoustic models and radar propagation characteristics, forming vibration characteristics consistent with real data, significantly expanding the diversity of the training set, and improving the generalization ability and stability of the model under different environments and speaker conditions.
[0040] (4) Unlike schemes that only perform spectrogram reconstruction, this invention introduces a multi-task lightweight classification network that simultaneously performs speech semantics and speaker identification, and imposes semantic consistency constraints on the generator, thereby significantly improving the reconstructed speech in terms of content recognizability, individual consistency and application reliability. Attached Figure Description
[0041] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;
[0042] Figure 2 This is a schematic diagram of the overall system structure of Embodiment 1 of the present invention;
[0043] Figure 3 This is a comparison diagram showing the effect of preprocessing radar data in S2 of Embodiment 1 of the present invention;
[0044] Figure 4 This is the result of the radar spectrum diagram corresponding to the statement "The building's on fire" in this invention after syllable segmentation in S3;
[0045] Figure 5 This is a simulated effect diagram generated for the word "bleed" in S3 of the present invention;
[0046] Figure 6 This is a schematic diagram of the structure of the multimodal speech reconstruction network model of Embodiment 1 of the present invention;
[0047] Figure 7 This is a spectral result obtained by a multimodal speech reconstruction network model in Embodiment 1 of the present invention;
[0048] Figure 8 A schematic diagram showing the performance comparison of the model before and after adding simulated data in the speech reconstruction task of this invention;
[0049] Figure 9 This diagram illustrates the performance comparison of the model in the recognition task before and after adding simulated data, as presented in this invention. Detailed Implementation
[0050] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.
[0051] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.
[0052] Example 1:
[0053] like Figure 1 The method for speech reconstruction and recognition based on a physical guidance network for millimeter-wave radar, as shown, includes the following steps:
[0054] S1. Use millimeter-wave radar to transmit radio frequency signals to the target under test and receive echo signals, while simultaneously acquiring reference audio signals;
[0055] The millimeter-wave radar transmits radio frequency signals to the target under test according to preset transmission parameters and obtains the echo signals through a receiving antenna; at the same time, an audio sensor is used to collect synchronous reference voice signals, and the two are aligned with a unified time reference as input data for subsequent processing.
[0056] S2. Preprocess the echo signal and extract the stable phase signal;
[0057] The echo signal is processed to extract the steady-phase signal containing micro-displacements of vocal cord and body resonance, and then subjected to syllable segmentation and polynomial fitting to enhance the periodic structure, such as... Figure 3 The image shows a comparison between the radar spectrum of the pre-processed echo signal and the original radar spectrum without denoising, fitting, or other processing.
[0058] S2 is achieved through the following sub-steps:
[0059] S21. Preprocess the echo signal, including denoising, zero-mean normalization, amplitude normalization, and static or near-body clutter suppression, to enhance the weak reflection characteristics of dynamic targets.
[0060] In practice, the radar sampling rate is set to approximately 14.284 kHz, with each frame containing 256 points. A sliding window background subtraction is used to remove static energy. Subsequently, a Fast Fourier Transform (Range-FFT) is performed in the range dimension to extract the amplitude and phase distribution of each range cell. For each frame of the I / Q cube, the range-direction transform for background subtraction can be expressed as:
[0061]
[0062] in, This is the original radar echo complex signal; The radar echo signal after background subtraction. For slow time frame indexing, For distance units, For receiving channel, The time window length is used for background estimation. This process effectively suppresses static background energy and highlights dynamic speech reflection components.
[0063] S22. Construct a background model in the slow time dimension and remove the intra-frame mean. Suppress residual noise by using a sliding time window and high-pass filtering, thereby highlighting the micro-motion energy components related to vocal cord vibration.
[0064] In this embodiment, background modeling employs a 64-frame moving average to remove slowly varying drift terms. This step enhances the periodic characteristics of the time-domain signal, providing a high signal-to-noise ratio input for steady-state extraction.
[0065] S23. Construct a spatial covariance matrix (R) based on the multi-channel received signal, and employ the Capon (MVDR) beamforming algorithm to enhance the speech reflection signal in the target direction. Its directional power spectrum is expressed as:
[0066]
[0067] in Direction angle Beam power, For array guiding vector, Let covariance matrix be the variance matrix. This indicates the conjugate transpose.
[0068] In the experiment, the array scanning angle range was... To suppress energy leakage in the direction of interference, adaptive focusing is achieved by minimizing interference power, which significantly improves the spatial resolution of the speech direction signal.
[0069] S24. Perform arctangent demodulation and phase unwrapping on the beamforming output signal to remove system drift and obtain a continuous and stable phase signal:
[0070]
[0071] in It is a complex baseband signal. This is for estimating the phase mean.
[0072] Then, slow-drift terms were removed using polynomial fitting:
[0073]
[0074] in , For the basis function vector, This is the corresponding polynomial fitting coefficient vector.
[0075] In practical processing, the polynomial order is generally set to 3 to balance trend removal and signal smoothing. Finally, based on the periodic changes of the steady-state signal, speech segment boundaries are detected. Syllable segmentation is achieved through adaptive thresholding or energy envelope analysis, and periodic enhancement fitting is performed on each signal segment to obtain clear syllable structure and temporal boundaries. The segmentation effect is as follows: Figure 4 As shown.
[0076] S3. Simulated radar Mel spectra are generated by simulating audio signals from reference audio signals and public speech datasets; the stable phase signal Mel spectra and simulated radar Mel spectra are synchronized and standardized to construct a speech signal dataset, with the following results: Figure 5 The images shown are, respectively, the simulated radar Mel spectrum, the real radar spectrum, and the reference audio signal. S3 is implemented through the following sub-steps:
[0077] S31. Obtain a public speech dataset; perform resampling and frame-segmentation processing on the reference audio signal and the public speech dataset to generate a resampled signal;
[0078] The reference audio signal and the public speech dataset are resampled and framed to ensure the sampling rate matches the radar signal, resulting in a resampled signal. The resampling expression is as follows:
[0079]
[0080] in, For reference audio signals and public speech datasets, and These are the audio and radar sampling rates, respectively. This is a resampled signal.
[0081] S32. Perform envelope extraction and bandpass filtering on the resampled signal to generate a simulated vibration signal;
[0082] To simulate the sensing characteristics of millimeter-wave radar for changes in the amplitude of vocal cord micro-vibrations, envelope extraction and bandpass filtering were performed on the resampled audio time-domain signal to obtain a simulated vibration signal consistent with the frequency band characteristics of the radar system. Specifically, an analytic signal was constructed using Hilbert transform:
[0083]
[0084] in, This is a resampled signal. For Hilbert transform operators, To analyze the signal, To maintain frequency band characteristics consistent with millimeter-wave radar systems, a bandpass filter is applied to the envelope sequence. Its frequency domain response is With the bandwidth range set to 100 Hz to 5 kHz, the bandpass envelope signal is obtained:
[0085]
[0086] in, This represents the discrete convolution operation. The simulated vibration signal, which is matched to the radar bandwidth, will be used as the input for subsequent noise addition and Mel spectrum feature construction.
[0087] S33. Superimpose Gaussian white noise onto the simulated vibration signal to generate a noise-added simulated radar vibration signal;
[0088] Based on the set signal-to-noise ratio For simulated vibration signals Gaussian white noise is superimposed to simulate the noise interference characteristics in a real millimeter-wave radar channel. The relationship between its noise power and signal power is as follows:
[0089]
[0090] in To simulate vibration signals average power, The power of Gaussian white noise, the signal-to-noise ratio Set to 15–25 dB to simulate different radar reflection intensities and noise environment conditions.
[0091] Based on the above noise power, a zero-mean Gaussian white noise sequence is generated. This signal is then superimposed on the simulated vibration signal to obtain the noise-added simulated radar vibration signal.
[0092]
[0093] The noisy signal is then linearly normalized to match its amplitude distribution with the statistical characteristics of the actual millimeter-wave radar output signal; This serves as the input signal for the construction of Mel spectrum features and time synchronization in subsequent steps.
[0094] S34. The simulated radar vibration signal after noise addition. Perform a short-time Fourier transform and map the Mel filter bank to construct the simulated radar Mel spectrum for model training, expressed as follows:
[0095]
[0096] in, Represents the short-time Fourier transform. This represents the Mel filter bank mapping function. The simulated radar-Mel spectral features are used as input features for the physically guided speech reconstruction network in subsequent steps and participate in the time synchronization and alignment of audio-radar features.
[0097] S35. Perform linear predictive coding on the reference audio signal and obtain the glottal residual signal through inverse filtering:
[0098]
[0099] in, As a reference, the sampled values of the audio signal in discrete time, This refers to the glottal residual signal obtained after applying a linear prediction inverse filter to the reference audio signal. The prediction filter is obtained from linear prediction analysis. For the first Linear prediction coefficients of order 1 To represent signal delay Unit delay operator for each sampling point; For linear prediction order, in this embodiment, we take... .
[0100] Based on the spectral analysis of the glottal residual signal, the fundamental frequency is extracted. and the amplitude of the first harmonic ( ), second harmonic amplitude ( )( and They represent the residual spectra at... and (Amplitude value at the specified point). Further based on Liljencrants–Fant (LF).
[0101] Glottal model for estimating glottal parameters With airflow parameters :
[0102]
[0103] These physical parameters will be fused with the time-frequency characteristics of the steady-state signal in subsequent models to enhance the physical interpretability of speech reconstruction.
[0104] S36. To achieve time synchronization between the reference audio signal and the radar signal, calculate the energy envelope of the Mel spectrum at both ends and perform correlation matching to estimate the optimal time shift.
[0105] First, the Mel spectra from the radar and audio ends are converged along the frequency dimension to obtain the corresponding time-energy sequences:
[0106]
[0107] in, and These represent the Mel energy spectra at the radar end and the audio end, respectively. For time frame indexing, For frequency indexing.
[0108] Based on this, the time energy sequences at both ends are calculated during time shift. Normalized correlation coefficient:
[0109]
[0110] in, and These are the mean values of the energy sequences between the radar and audio ends, respectively.
[0111] To further improve the robustness of time synchronization, the following cost function is constructed based on the correlation constraint:
[0112]
[0113] in, For time translation The effective summation interval below, and These are the balancing weighting coefficients.
[0114] By minimizing the cost function The optimal time shift between the audio signal and the radar signal is obtained:
[0115]
[0116] The optimal time shift This is used to align audio features with radar features in time, serving as the basis for time synchronization during subsequent model training and feature fusion.
[0117] S37. Standardize and time-align the steady-state signal and the reference audio signal to construct a training set and a validation set containing radar-audio dual modes for model training and performance evaluation.
[0118] S4. Construct a multimodal speech reconstruction network model; train the multimodal speech reconstruction network model based on the speech signal dataset; input the newly acquired real millimeter-wave radar signal into the trained multimodal speech reconstruction network model to reconstruct the speech and output a non-contact speech Mel spectrum.
[0119] A physically guided multimodal speech reconstruction network model was constructed and trained. During the training phase, the Mel spectrum features of radar signals were used as network input. The training samples consisted of a hybrid dataset composed of the Mel spectrum features of the real millimeter-wave radar steady-state signal obtained in step S2 and the Mel spectrum features of the simulated radar signal generated in step S3. The speech time-frequency features corresponding to the reference audio signal were used as the supervision target, and the glottal physical parameters extracted from the reference audio signal were introduced as conditional constraints to learn the mapping relationship from radar features to speech time-frequency features. After the model training was completed, speech reconstruction could be performed on newly acquired real millimeter-wave radar signals, outputting a non-contact speech Mel spectrum.
[0120] S4 is achieved through the following sub-steps.
[0121] S41. In this embodiment, a physically guided multimodal reconstruction network model is constructed. The network input is the steady-state Mel spectrum of the radar signal, used to characterize the sound-related vibration information sensed by the millimeter-wave radar. Simultaneously, glottal physical parameters extracted from a reference audio signal are introduced as conditional input to physically guide the speech generation process. For example... Figure 6 As shown, the model consists of an encoder, a cross-modal attention module, a decoder, and a discriminator.
[0122] The encoder section is used to extract the multi-scale time-frequency features of the Mel spectrum of millimeter-wave radar signals. The input spectrum is obtained by a 1024-point short-time Fourier transform, with a frame shift of 100 points. The frequency range is compressed and clipped to the range of −80 dB to 0 dB using a logarithmic scale.
[0123] The cross-modal attention module is used to fuse radar spectra and physical parameters. The attention embedding dimension is 128, and four attention heads are used to achieve weighted association of cross-modal features.
[0124] The decoder restores the fused conditional features into a speech spectrogram, and the discriminator uses a convolutional stacking structure to jointly discriminate the input spectrogram and conditional features, thus forming an adversarial learning framework.
[0125] The conditional input vector is:
[0126]
[0127] Where C is the fusion condition vector, derived from the radar mell spectrum. With the fundamental frequency extracted from the audio end Harmonic amplitude , Glottal parameters and airflow parameters Together, they form a set of physical prior constraints used to provide multimodal physical constraints in generative models.
[0128] S42. During the training phase, a hybrid dataset consisting of the Mel spectrum features of the real millimeter-wave radar phase-steady signal obtained in step S2 and the Mel spectrum features of the simulated radar signal generated in step S3 is used to jointly train the generator and discriminator. The speech Mel spectrum features corresponding to the reference audio signal are used as the supervision target. By minimizing the difference between the generated speech spectrum and the real speech spectrum, the mapping relationship from radar features to speech time-frequency features is learned. The joint loss function of the model is defined as:
[0129]
[0130] in, To counteract the loss, For feature matching loss term, For the total variational regularization term , and These represent the weight coefficients of the corresponding loss terms. To stabilize the discriminator training process, an R1 gradient penalty term is introduced in each training batch:
[0131]
[0132] The gradient magnitude is used to constrain the discriminator. The gradient penalty coefficient is... This indicates the discriminator's response to the speech Mel spectrogram. The gradient is used to constrain the smoothness and stability of the discriminator gradient.
[0133] The optimizer uses the Adam algorithm, and the learning rate and weight decay parameters are set according to the configuration file. Training employs an alternating update strategy for the generator and discriminator, and uses cosine annealing of the learning rate to improve convergence stability. Before training, the input data is augmented by temporal random shifting (maximum 16 frames) and slight jitter to improve the model's generalization ability.
[0134] S43. After the model training is complete, the Mel spectrum of the preprocessed steady-state signal is input into the generator, and the predicted speech Mel spectrum is output. The result is as follows: Figure 7 The images show (a) the radar spectrogram, i.e., the newly acquired real millimeter-wave radar signal; (b) the speech spectrogram, i.e., the reference audio signal; and (c) the reconstructed radar spectrogram, i.e., the predicted speech Mel spectrum). The speech waveform is then reconstructed using inverse Mel transform and inverse short-time Fourier transform (window 1024, step size 310).
[0135]
[0136] in, The generated speech Mel spectrogram. For the Mel inverse transform operator, It is the inverse short-time Fourier transform, used to recover the predicted time-frequency graph into a time-domain speech signal.
[0137] To evaluate the model's performance in speech reconstruction tasks, this paper uses Normalized Mean Absolute Error (NMAE) as the evaluation metric to measure the relative error between the reconstructed speech and the reference speech. The parameter configuration during the evaluation phase is consistent with that during the training phase to ensure the comparability of experimental results. Figure 8The figure shows the NMAE results of the model in the speech reconstruction task. When trained using only real millimeter-wave radar data, the model's NMAE is 0.161, which is already able to achieve a relatively stable speech reconstruction effect. After introducing simulated data, the reconstruction error was further reduced to 0.134.
[0138] S44. During the system deployment phase, the model is embedded into the real-time reconstruction process. Newly acquired millimeter-wave radar signals are preprocessed to obtain the Mel spectrum and physical parameters of the stable-phase signal, which form the conditional input. The generator outputs a predicted speech spectrogram in real time. After inverse transformation, a speech signal with a sampling rate of 44.1 kHz is obtained, enabling non-contact speech reconstruction by millimeter-wave radar and providing high-fidelity input for subsequent speech recognition and identity classification.
[0139] S5. Input the speech Mel spectrogram into the constructed lightweight convolutional neural network classifier and output the speaker's identity category label;
[0140] A lightweight convolutional neural network classifier is constructed to classify speaker identities based on Mel spectrograms, and to evaluate the speaker consistency and sample discriminability of the reconstructed speech. The classifier takes the speech Mel spectrogram as input and outputs the corresponding speaker identity category label. In the verification phase, real speech samples and speech samples reconstructed from radar signals are input into the same classifier, and the identity preservation capability of the speech reconstruction model is evaluated by comparing the recognition results of the two. Step S5 is implemented through the following sub-steps:
[0141] A lightweight convolutional neural network classifier is constructed for speaker identification. Its input is the Mel spectrogram corresponding to a real speech sample, and its output is the speaker identification category label. The network consists of three convolutional layers and two fully connected layers. Each layer contains convolution, batch normalization, and ReLU activation structures; the convolutional kernel size is [size missing]. Pooling window is The time-frequency representation of speech is obtained through layer-by-layer feature extraction.
[0142] Using the Mel spectrum of real speech samples as input, category labels As a monitoring signal, the cross-entropy loss function is used:
[0143]
[0144] in The output probability is set for Softmax. The optimizer uses the Adam algorithm combined with a learning rate annealing strategy to improve convergence stability. During the training phase, only real samples are used, and no generated data is introduced to ensure the realism and robustness of the classification boundary.
[0145] During the validation phase, real and reconstructed samples are input into the classifier separately, and the classification accuracy is calculated:
[0146]
[0147] in For the predicted results, The data is labeled as real samples. By comparing the recognition accuracy of reconstructed samples with that of real samples, the effectiveness and discriminative power of the speech reconstruction model in preserving speaker identity features are evaluated. Figure 9 The results show the accuracy and recall of the model in the speech recognition task. With the introduction of simulated data, the accuracy of speech recognition based on reconstructed speech has reached 90%, and the recall has reached 87%.
Claims
1. A method for speech reconstruction and recognition using millimeter-wave radar based on a physical guidance network, characterized in that, Includes the following steps: S1. Use millimeter-wave radar to transmit radio frequency signals to the target under test and receive echo signals, while simultaneously acquiring reference audio signals; S2. Extract the steady-state signal Mel spectrum based on the echo signals; S3. Generate a simulated radar Mel spectrum by simulating the reference audio signal and the public speech dataset; synchronize and standardize the steady-state signal Mel spectrum and the simulated radar Mel spectrum to construct a speech signal dataset; S4. Construct a multimodal speech reconstruction network model; train the multimodal speech reconstruction network model based on the speech signal dataset; input the newly acquired real millimeter-wave radar signal into the trained multimodal speech reconstruction network model to reconstruct the speech and output a non-contact speech Mel spectrum; S5. Input the speech Mel spectrogram into the constructed lightweight convolutional neural network classifier and output the speaker's identity category label.
2. The method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 1, characterized in that, S2 include: S21. After preprocessing the echo signal, perform a fast Fourier transform in the distance dimension and perform background subtraction. S22. Perform slow-time background modeling and removal on the echo signal after background subtraction; S23. Construct a spatial covariance matrix based on the multi-channel received signal after slow-time background removal, and use a beamforming algorithm for spatial filtering to generate a beamforming output signal. S24. Perform phase demodulation on the beamforming output signal. Remove system drift to obtain a stable signal.
3. The method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 2, characterized in that, S2 also includes detecting speech segment boundaries based on periodic changes in steady-state signals, performing syllable segmentation through adaptive thresholding or energy envelope analysis, and performing periodic enhancement fitting on each signal segment to obtain syllable structure and time boundaries.
4. The method for speech reconstruction and recognition of millimeter-wave radar based on a physical guidance network according to claim 1, characterized in that, S3 include: S31. Obtain a public speech dataset; perform resampling and frame-segmentation processing on the reference audio signal and the public speech dataset to generate a resampled signal; S32. Perform envelope extraction and bandpass filtering on the resampled signal to generate a simulated vibration signal; S33. Superimpose Gaussian white noise onto the simulated vibration signal to generate a noise-added simulated radar vibration signal; S34. Perform a short-time Fourier transform and map the noise-added simulated radar vibration signal to a Mel filter bank to generate a simulated radar Mel spectrum.
5. The method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 1, characterized in that, S3 also includes: S35. Perform linear prediction analysis on the reference audio signal and obtain the glottal residual signal through inverse filtering; based on the spectral analysis of the glottal residual signal, extract the fundamental frequency... The glottal physical parameters include the first harmonic amplitude, the second harmonic amplitude, glottal parameters, and airflow parameters.
6. The method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 5, characterized in that, S3 also includes: S36. Calculate the energy envelope of the steady-state signal Mel spectrum and the simulated radar Mel spectrum and perform correlation matching to calculate the optimal time shift; align the steady-state signal Mel spectrum and the simulated radar Mel spectrum according to the optimal time shift to achieve time synchronization between the reference audio signal and the echo signal. S37. Based on the time-aligned steady-state signal Mel spectrum and the simulated radar Mel spectrum, construct a speech signal dataset containing radar modes and audio modes.
7. The method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 6, characterized in that, S36 includes: performing energy convergence on the Mel spectrum of the steady-phase signal and the Mel spectrum of the simulated radar along the frequency dimension to obtain the corresponding time energy sequences; calculating the normalized correlation coefficient of the time energy sequences at both ends under time shift; constructing a cost function based on the normalized correlation coefficient; and obtaining the optimal time shift between the audio signal and the radar signal by minimizing the cost function.
8. The method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 1, characterized in that, S4 includes S41. Construct a physically guided multimodal reconstruction network model including an encoder, a cross-modal attention module, a decoder, and a discriminator; The Mel spectrum of the steady-phase signal is input into the encoder to extract the multi-scale time-frequency features of the millimeter-wave radar signal; Using glottal physical parameters as input conditions, the speech generation process is physically guided. Conditional features are generated by fusing the Mel spectrum of the steady-state signal and the glottal physical parameters using a cross-modal attention module. The decoder restores the conditional features to a speech Mel spectrogram; The discriminator uses a convolution stacking structure to jointly discriminate the Mel spectrum and conditional features of the steady-phase signal.
9. A method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 8, characterized in that, S4 also includes: S42. By minimizing the difference between the speech Mel spectrogram and the true speech spectrum of the reference audio signal, the joint loss function is constructed as follows: ， in, To counteract the loss, For feature matching loss term, For the total variational regularization term, , and These represent the weight coefficients of the corresponding loss terms; an R1 gradient penalty term is introduced in each training batch to constrain the gradient magnitude of the discriminator.
10. A method for millimeter-wave radar speech reconstruction and recognition based on a physical guidance network according to claim 9, characterized in that, S4 also includes: the speech Mel spectrogram is reconstructed into the final speech waveform through inverse Mel transform and inverse short-time Fourier transform; in S5, the real speech waveform under the reference audio signal and the final speech waveform are respectively input into a lightweight convolutional neural network classifier, and the identity preservation ability of the multimodal reconstruction network model is evaluated by comparing the recognition results of the two.