An audio reconstruction method based on millimeter wave signals

Abstract

The present invention discloses an audio reconstruction method based on millimeter wave signals, comprising the following steps: spectrum generation: millimeter wave radar emits chirp signals, which are reflected by a vibrating target and returned to the millimeter wave radar to identify the position and distance of the vibrating target; the millimeter wave radar is used to perform a slow time sampling operation on the vibrating target to obtain a phase change, and the vibration displacement is obtained by using the phase change, and then merged into a vibration waveform; the time spectrum is obtained by using the vibration waveform, and converted into a millimeter wave Mel spectrum; audio reconstruction: the millimeter wave Mel spectrum and the corresponding real audio Mel spectrum are input into an artificial intelligence model cGAN, and the model is trained. After the training is completed, the model is used to generate an audio Mel spectrum, and waveform synthesis is performed to achieve audio reconstruction. The method disclosed by the present invention has higher resolution and an unrestricted vocabulary set, and can accurately reconstruct audio at different distances and angles, and through the barriers of different insulating materials.

Description

Technical Field

[0001] The present invention relates to the technical field of audio reconstruction, and in particular to an audio reconstruction method based on millimeter wave signals. Background Art

[0002] Today's information development has entered the era of "Internet of Everything". There are a large number of audio signal receiving and sending devices in personal digital assistants, smart speakers, smart microphones, smart audio and video systems, online conference systems, smart office systems and smart home systems. Voice communication systems are becoming more popular than ever.

[0003] Although a variety of voice collection technologies are used in smart interconnection scenarios, these methods all have deficiencies of one kind or another. Some are applicable to a small spatial range and can only be used to collect voice signals in smaller spatial scenarios. When the usage scenario is in a larger spatial range, it cannot achieve a good sound reception effect, such as the general microphones used in common video office conferences. Some are not applicable to a sufficient frequency domain and can only collect sound signals within a limited human voice frequency domain, but cannot collect lower-frequency and higher-frequency human voice signals or instrument sounds and environmental sounds, resulting in limited applicable scenarios. Some have weak anti-interference capabilities. When the usage scenario has many sound sources, large environmental noise, and environmental echoes, it is impossible to achieve high-quality collection of the target sound, and even a howling phenomenon may occur.

[0004] In summary, the current conventional sound recording methods cannot well meet the needs of collecting high-quality sound signals for wide-band sound signals in a larger space. Summary of the invention

[0005] To solve the above technical problems, the present invention provides an audio reconstruction method based on millimeter wave signals, which has higher resolution and an unrestricted vocabulary set, and can accurately reconstruct audio at different distances and angles and through the barriers of different insulating materials.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] An audio reconstruction method based on millimeter wave signals comprises the following steps:

[0008] Step 1: Spectrum graph generation:

[0009] (1) The millimeter-wave radar emits chirp signals at fixed time intervals. The chirp signals are reflected by the vibrating target and then returned to the millimeter-wave radar. Combined with the echo signal received by the millimeter-wave radar, the fast-time sampling signal is transformed by fast Fourier transform to identify the position and distance of the vibrating target.

[0010] (2) According to the position and distance of the identified vibration target, the millimeter-wave radar is used to perform a slow-time sampling operation on the vibration target, and then the slow-time sampling signal is subjected to a Doppler fast Fourier transform to obtain a phase change, and the vibration displacement is obtained by using the phase change, and then all the vibration displacements are combined into a vibration waveform using a timestamp;

[0011] (3) performing a short-time Fourier transform on the vibration waveform to obtain a time spectrum, and then converting the time spectrum into a millimeter-wave Mel spectrum;

[0012] Step 2: Audio reconstruction:

[0013] The millimeter wave Mel spectrum graph and the corresponding real audio Mel spectrum graph are input into the artificial intelligence model cGAN, and the model is trained. The generator in the model uses the millimeter wave Mel spectrum graph to generate a simulated audio Mel spectrum graph, and the discriminator in the model uses the real audio Mel spectrum graph to identify the authenticity of the simulated audio Mel spectrum graph. The generator and the discriminator are trained alternately, so that the simulated audio Mel spectrum graph generated by the generator is close to the real audio Mel spectrum graph;

[0014] After the training is completed, the generator in the trained model is used to generate an audio Mel spectrogram from the collected millimeter-wave Mel spectrogram, and the Griffin-Lim algorithm is used to perform waveform synthesis on the generated audio Mel spectrogram to restore the audio waveform and achieve audio reconstruction.

[0015] In the above scheme, when identifying the position of the vibration target, a sliding window is applied to the fast time sample within a chirp to generate more precise sub-chirps, and then a range fast Fourier transform is performed on each precise sub-chirp to obtain multiple slow time samples.

[0016] In the above scheme, the calculation formula of the vibration displacement is as follows:

[0017]

[0018] Among them, Δd is the vibration displacement, Δφ is the phase change, and λ is the wavelength of the millimeter wave signal.

[0019] In the above scheme, the formula for short-time Fourier transform is as follows:

[0020]

[0021] Among them, h(τ-t) is the window function, τ is half of the window size at time t, x is the waveform of the millimeter wave signal, f is the frequency of the millimeter wave signal, and j is the complex base unit.

[0022] In the above scheme, the formula for converting the time spectrum graph into the Mel spectrum graph is as follows:

[0023]

[0024] Where mel(f) is the Mel spectrum and f is the frequency of the millimeter wave signal.

[0025] In the above scheme, the generator adopts the UNET architecture, and each convolution block in the generator and the discriminator is composed of a square kernel of size 4×4 with a step size of 2; the discriminator includes three convolution blocks and true and false patch predictions, and the patch size is 30×30.

[0026] In the above scheme, during the model training process, the generator uses the millimeter-wave Mel-spectrogram as conditional input and enhances it; the enhanced simulated Mel-spectrogram is connected to the millimeter-wave Mel-spectrogram and input into the discriminator, which classifies it as fake audio; when the combination of the real audio Mel-spectrogram and the millimeter-wave Mel-spectrogram is input, the discriminator classifies it as real audio; as the training progresses, the generator learns to enhance the input, making it difficult for the discriminator to distinguish between the simulated Mel-spectrogram enhanced by the generator and the real Mel-spectrogram.

[0027] Through the above technical solution, the audio reconstruction method based on millimeter wave signals provided by the present invention has the following beneficial effects:

[0028] (1) The audio reconstruction method implemented by the present invention uses a commercial millimeter wave FMCW radar to accurately capture the vibration of a vibrating target, and then enhances the captured speaker vibration signal by generating a machine learning model to reconstruct a high-quality audio signal with an infinite vocabulary space.

[0029] (2) The present invention uses millimeter wave radar to emit millimeter waves, which have good penetration into conventional sound insulation materials such as glass, polyethylene foam plastic, plywood, etc., and can break through conventional interference measures to a certain extent.

[0030] (3) The present invention detects acoustic vibration signals based on millimeter wave radio signals, has a long effective distance, high accuracy, and relatively low cost.

[0031] (4) The method of the present invention transmits chirp signals at fixed time intervals and combines a series of chirp signals into a frame for range-Doppler processing. A sliding window is introduced into the fast time sampling within a chirp to generate more accurate sub-chirps. The cross-correlation of multiple sub-chirp signals improves the accuracy of the distance position judgment of the sound target.

[0032] (5) The present invention integrates chirp signals for slow time sampling, performs Doppler-fast Fourier transform, and introduces timestamps, thereby improving the speed of extracting the vibration displacement of the sound target and preparing data for the subsequent introduction of artificial intelligence technology.

[0033] (6) The present invention aims at the cGAN artificial intelligence model and innovatively adopts a two-dimensional image expression method of audio signals with a timestamp, namely the Mel-spectrogram, which effectively reduces the information loss in the process of converting millimeter wave signals to audio signals.

[0034] (7) The present invention innovatively utilizes the artificial intelligence technology in the field of image processing - cGAN, to enhance the converted millimeter wave signal, broadens the artificial intelligence enhancement technology route for audio signals, and realizes the reconstruction technology of audio signals with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0036] Figure 1 A schematic diagram of an audio reconstruction method based on millimeter wave signals disclosed in an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of vibration extraction from FMCW;

[0038] Figure 3 Schematic diagram of phase extraction from FMCW;

[0039] Figure 4 The original sound spectrum and the millimeter wave reflected signal spectrum of the loudspeaker; (a) is the original sound spectrum, (b) is the millimeter wave reflected signal spectrum of the loudspeaker;

[0040] Figure 5 This is the architecture diagram of the anti-generation network cGAN;

[0041] Figure 6 Reconstruction system diagram for millimeter wave based sound signal;

[0042] Figure 7 The experimental scene layout diagram; (a) conference room scene with wooden door interference; (b) conference room scene with double-glazed glass interference;

[0043] Figure 8 Comparison of audio reconstruction performance; (a) Spectrum of real speech, (b) Spectrum of original data of unenhanced millimeter-wave sampling, (c) Spectrum of sound reconstructed from millimeter-wave sampling data using cGAN;

[0044] Fig. 9 To give an objective MCD score for the sound;

[0045] Fig.10The subjective rating of the sound; (a) audio clip 1 (60s), (b) audio clip 2 (58s);

[0046] Fig.11 Vibration extraction performance diagrams at different distances and angles; (a) at different distances, (b) at different angles;

[0047] Fig.12 The audio reconstruction performance diagram at different distances and angles; (a) at different distances, (b) at different angles;

[0048] Fig.13 The effect of different sound insulation materials on audio reconstruction performance;

[0049] Fig.14 recreate performance for audio from different speakers;

[0050] Fig.15 For model generalization performance: Compare the MCD scores of cGAN under different voice samples: (a) The MCD scores when the voices of User 2, User 3, and User 4 are tested using models trained with their own data and when tested using the model trained with User 1 data, (b) The MCD scores when the model is trained using data from User 1 to User 4 and tested on User 5 to User 7. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0052] The present invention designs and implements a sound signal reconstruction system called "Milli-Ear". It has a millimeter-wave radar that can capture sounds caused by tiny vibrations. The millimeter-wave radar first transmits a frequency-modulated signal to the speaker (vibration target). The signal reaching the speaker is reflected back to the millimeter-wave radar. By processing and enhancing the received signal, the "Milli-Ear" can extract the vibration of the speaker. The vibration data will then be input into the generative adversarial network cGAN for enhancement and noise reduction, ultimately achieving high-quality audio reconstruction. Figure 1 As shown, Milli-Ear consists of two modules:

[0053] (1) Spectrum Generation (SG): Spectrum generation consists of two stages: target (speaker) localization and spectrum extraction. In order to locate the position of the speaker, the original samples are extracted from the millimeter wave radar as input. The raw data is subjected to a range fast Fourier transform to measure the distance to the target. Then, a Doppler fast Fourier transform is performed based on the result of the range fast Fourier transform to find the candidate distance segments and identify the distance segments containing the desired vibration. In order to improve the resolution of the fast Fourier transform, each chirp of the frame is decomposed into multiple chirps, providing multiple observations while extracting the sound vibration displacement. Each linear frequency modulation is then subjected to a short time Fourier transform (STFT) to obtain a time domain spectrogram.

[0054] (2) Audio Reconstruction (AR): AR uses cGAN and is trained with two spectrogram images, one from mmWave radar sampling data and the other from raw audio. Using the training data, cGAN learns how to enhance mmWave spectrograms by enhancing typical frequency and amplitude components and reducing noise. The trained cGAN model is then used to reconstruct audio directly from the captured mmWave spectrograms.

[0055] Based on the above-mentioned sound signal reconstruction system, the present invention provides an audio reconstruction method based on millimeter wave signals, comprising the following steps:

[0056] Step 1: Spectrum graph generation:

[0057] (1) The millimeter-wave FMCW radar transmits chirp signals at fixed time intervals. The chirp signals are reflected by the vibrating target and then returned to the millimeter-wave radar. Combined with the echo signal received by the millimeter-wave radar, the fast-time sampling signal is transformed by fast Fourier transform to identify the position and distance of the vibrating target.

[0058] Specifically, the millimeter-wave FMCW radar transmits chirp signals at fixed time intervals and combines a series of chirp signals into a frame for range fast Fourier transform. The range fast Fourier transform (Range-FFT) generally takes all fast-time samples of a chirp as input and produces a slow-time sample. However, low-cost commercial millimeter-wave radars cannot guarantee accurate phase extraction based on a single chirp under low signal-to-noise ratio (SNR) conditions. In order to improve phase extraction, a sliding window is applied to the fast-time samples within a chirp to generate more accurate sub-chirps, such as Figure 2As shown. These sub-chirps are used to reference each other. Then, a range fast Fourier transform (Range-FFT) is performed on each sub-linear frequency modulation signal to obtain multiple slow-time samples. Since the duration of a slow-time sample (a frame) is much longer than the duration of a fast-time sample (a chirp), the time variance of a group of chirps within a chirp can be ignored, that is, these sub-chirps can be considered to be transmitted simultaneously. Figure 2 As shown, the position of the voice mailbox detected by sub-chirp2 (dark chirp) is different from the position of other sub-chirps (light chirps). Since the present invention has multiple cross-validated observations, the abnormal part (dark color) can be identified and eliminated. In this way, the present invention can accurately identify the correct vibration target-voice mailbox position.

[0059] (2) According to the position and distance of the identified vibration target, the millimeter-wave radar is used to perform a slow-time sampling operation on the vibration target, and then the slow-time sampling signal is subjected to a Doppler fast Fourier transform to obtain a phase change, and the vibration displacement is obtained using the phase change. Then, a timestamp is used to merge all the vibration displacements into a vibration waveform.

[0060] By accurately extracting the vibration target - the voice mailbox, the slow time sampling is subjected to Doppler fast Fourier transform to obtain the phase change. The vibration displacement can be calculated according to formula (1).

[0061]

[0062] Among them, Δd is the vibration displacement, Δφ is the phase change, and λ is the wavelength of the millimeter wave signal.

[0063] Based on the above formula, the present invention uses a 77GHz frequency modulated continuous wave radar to detect a displacement of about 10 microns. The vibration amplitude of a speaker is usually in the hundreds of microns. Therefore, the millimeter wave radar can capture the tiny vibration of the speaker.

[0064] Based on the above formula, the vibration displacement is directly related to the phase change. The present invention can extract the precise phase change from the millimeter wave signal, and then obtain the vibration displacement.

[0065] S Tx (t) and S Rx (t) is the FMCW signal sent and received, then the following equation is obtained:

[0066] S Tx (t) = A Tx ·cos[2π·f Tx (t)·t+φ Tx ] (2)

[0067] S Rx (t) = A Rx ·cos[2π·f Rx (t)·t+φ Rx ] (3)

[0068] Among them, f Tx (t) and f Rx (t) represent the frequencies of the transmitted and received signals at time t, respectively, Tx and φ Rx Represent the phase of the transmitted signal and the received signal, A Tx and A Rx Represents the amplitude of the transmitted and received signals.

[0069] After applying the mixer to the transmit and receive signals, the following beat frequency signal can be obtained:

[0070]

[0071] Among them, f b (t) = f Tx (t)-f Rx (t) is the frequency variation function of the beat signal, φ b =φ Tx -φ Rx Since the beat frequency (MHz level) is much lower than the carrier frequency (GHz level), the present invention can use a low-pass filter to exclude the carrier, and then the beat frequency signal can be expressed as follows:

[0072] S b =A b ·cos[2π·f b (t)·t+φ b ] (4)

[0073] in, This is the combined amplitude of the transmitter and receiver.

[0074] Therefore, the final beat frequency signal is only related to f b Band φ b In fact, since there are reflected signals from objects at different distances in the original data, S b The frequency components of (t) are different. Figure 3As shown, in linear frequency modulation, a range fast Fourier transform (Range-FFT) is performed on fast time sampling. It maps the time domain signal to the frequency domain. Objects at different distances will have a peak in the frequency domain. For vibration source positioning, the present invention only focuses on the corresponding range of the corresponding distance. Further, a Doppler fast Fourier transform (Doppler-FFT) is performed on the result of deriving the phase change φ1-φ2. For two chirps in the same range, a Doppler fast Fourier transform operation (Doppler-FFT) is performed to extract the phase of the corresponding position. In this way, the time-varying phase caused by the vibration of the speaker is derived.

[0075] To verify the feasibility of the solution, the present invention compared the correlation between the received millimeter wave signal and the audio played through the speaker. In the experiment, the present invention played a speech through the speaker, and the millimeter wave radar was placed one meter in front of the speaker for sampling, with no obstruction between the two. Figure 4 (a) shows the spectrogram of the playback audio captured by the FMCW radar. Figure 4 (b) shows the corresponding millimeter-wave radar signal spectrum. It can be observed that the millimeter-wave signal has a high similarity with the audio signal. Due to the low sampling rate of the FMCW radar, the radar signal has a poor similarity with the audio signal in the high frequency band. In addition, the FMCW radar is also affected by low-frequency noise. To address these two problems, a generative machine learning model is introduced to reconstruct the audio information.

[0076] (3) Perform a short-time Fourier transform on the vibration waveform to obtain a time spectrum diagram, which is then converted into a millimeter-wave Mel spectrum diagram.

[0077] The vibration waveform in this experimental environment is a one-dimensional signal. However, the conditional generative adversarial network (cGAN) in audio reconstruction requires Figure 1 The present invention first converts the vibration waveform into a Mel-spectrogram. Mel-spectrogram is a common representation method for audio signals and is widely used in speech synthesis, audio denoising and other fields. This image-like spectrogram can be directly input into cGAN for enhancement. The enhanced spectrogram is converted back to audio with almost no information loss.

[0078] In this work, the present invention selects the short-time Fourier transform (STFT) to obtain the time spectrum. STFT is essentially a windowed Fourier transform, which is defined as follows:

[0079]

[0080] Among them, h(τ-t) is the window function, τ is half of the window size at time t, x is the waveform of the millimeter wave signal, f is the frequency of the millimeter wave signal, and j is the complex basic unit (same as i).

[0081] Since the amplitude of the generated spectrogram is relatively large, in order to obtain the sound features of appropriate size, it is usually necessary to generate a mel spectrum through a mel-scale filter bank. Studies have shown that humans cannot perceive frequency changes linearly. Humans can more easily detect differences at low frequencies than at high frequencies. For example, the present invention can easily distinguish the difference between 500Hz and 1000Hz, but it is difficult for the present invention to distinguish the difference between 10000Hz and 10500Hz. In order to capture this feature, the present invention converts the spectrogram generated by STFT into a mel spectrum. The conversion process for calculating the mel spectrum conforms to the following equation:

[0082]

[0083] Where mel(f) is the Mel spectrum and f is the frequency of the millimeter wave signal.

[0084] The above process realizes the transformation of the vibration signal and the corresponding audio waveform during the test.

[0085] Step 2: Audio reconstruction:

[0086] The millimeter-wave Mel-spectrogram and the corresponding real audio Mel-spectrogram are input into the artificial intelligence model cGAN to train the model. The generator in the model uses the millimeter-wave Mel-spectrogram to generate a simulated audio Mel-spectrogram, and the discriminator in the model uses the real audio Mel-spectrogram to identify the authenticity of the simulated audio Mel-spectrogram. The generator and the discriminator are trained alternately so that the simulated audio Mel-spectrogram generated by the generator is close to the real audio Mel-spectrogram.

[0087] A.cGAN Structure

[0088] The present invention uses an image-to-image conversion method to enhance the millimeter wave mel spectrum map. The present invention uses a conditional adversarial generative network called cGAN. Unlike other GANs that generate data from a random noise vector, cGAN additionally obtains a conditional variable to control the generated data. The objectives of the generator and discriminator are modified to include conditional inputs. The generator and discriminator formulas are expressed as log(1-D(y,G(z,y))) and log(D(y,x)), respectively.

[0089] Generative Adversarial Networks (cGANs) belong to a class of generative models. The goal of cGANs is to learn a function that can map two distributions: source and target. The source is a random noise distribution (p z (z)), the target is data (p data). Once this mapping is completed, cGAN can take a sample z∈p z , and maps it to x∈p data The cGAN model is trained by simulating a min-max game between two networks, one is the generator (G) and the other is the discriminator (D). The goal of the generator is to obtain the z (z)) to generate samples to deceive the discriminator, (p z (z)) are similar to those p data The job of the discriminator is to correctly label the data coming from the generator as fake data and the data coming from p data The data is marked as true data. The objective function V(G,D) of the min-max game between the two networks can be written as

[0090]

[0091] The goal of the generator is to minimize log(1-D(G(z))), and the goal of the discriminator is to minimize log D(x). When the model is doing this training task, it reaches an equilibrium when the generator has successfully approximated the data and the discriminator can no longer distinguish between real and fake data.

[0092] Figure 5 The cGAN structure of the present invention is shown. During the training process, the generator uses the millimeter-wave Mel spectrogram as a conditional input and enhances it. The enhanced simulated audio Mel spectrogram is connected to the millimeter-wave Mel spectrogram and input into the discriminator, which can identify that it is fake. In addition, when the combination of the real audio Mel spectrogram and the millimeter-wave Mel spectrogram is input, the discriminator classifies it as real audio. As the training progresses, the generator learns to enhance the input so that it is difficult for the discriminator to distinguish between the simulated Mel spectrogram enhanced by the generator and the real audio Mel spectrogram obtained from real audio. After this, for testing, the generator is used independently to enhance the millimeter-wave Mel spectrogram without the presence of the discriminator. It can be observed that the discriminator essentially helps the generator learn by pointing out errors in the generated data.

[0093] For the generator network, the present invention utilizes the UNET architecture. UNET is an encoder / decoder-based structure for biomedical image segmentation. Each convolutional block in the generator and discriminator consists of a square kernel of size 4×4 with a step size of 2, followed by batch normalization and rectified linear units (ReLU) to handle nonlinearity. Batch normalization normalizes the activations of different units and accelerates network convergence. A dropout value of 0.5 is used in the intermediate layers, and the number of filters is set to multiples of 64, and in subsequent layers, the filter size is linearly reduced as suggested by experience. For the discriminator, the present invention uses three convolutional blocks, followed by true and false patch predictions, with a patch size of 30×30. Compared with pixel-by-pixel or image-by-image predictions, block-by-block predictions exploit the independence of distant regions. In addition, since the captured millimeter wave data does not include high-frequency components of audio, the network's predictions for these parts can be improved independently. The generator network and the discriminator network can be trained alternately. The present invention uses a binary cross entropy loss between the predicted and true value labels on the generator network and an L1 norm as the loss function. The L1 norm provides regularization without the blurring artifacts of the L2 norm. We empirically observed that a learning rate of 0.0002 leads to faster convergence. The Adam optimizer is used to optimize the network. The network is trained for 200 epochs, and the performance on a validation set is used to select the best training epoch.

[0094] B. Reconstruction from enhanced spectrogram

[0095] After the training is completed, the generator in the trained model is used to generate an audio Mel spectrogram from the collected millimeter-wave Mel spectrogram, and the Griffin-Lim algorithm is used to perform waveform synthesis on the generated audio Mel spectrogram to restore the audio waveform and achieve audio reconstruction.

[0096] cGAN enhances the millimeter wave Mel-spectrogram to make it have richer acoustic features, and the present invention can convert acoustic parameters into speech waveforms using a vocoder. The present invention uses the Griffin-Lim algorithm to perform waveform synthesis on the generated spectrogram. Griffin-Lim uses phase constraints between frames to achieve iterative convergence, and uses the spectrogram to reconstruct the speech signal based on the lack of original phase information. It solves the problem of how to find the approximate phase without destroying the adjacent amplitude spectrum and the amplitude spectrum itself. Considering that there is a large difference between the worst-case and best-case phases, a more accurate phase can be obtained through iteration. In this way, even without the original phase information, the present invention can use the Griffin-Lim algorithm to restore the audio waveform to a large extent.

[0097] Performance evaluation:

[0098] In real-life scenarios, there are often some obstacles in the use environment that affect the propagation of sound, and some are even some sound insulation materials, such as glass, polyethylene foam plastics, plywood, etc. The sound signal reconstruction system of the present invention can better collect and reconstruct sound signals even in the presence of certain sound insulation material interference. Figure 6 As shown, the present invention can envision that the sound receiving system is in the following situation:

[0099] (i) There is sound isolation between the sound receiving system and the sound source, that is, the sound from the sound source cannot penetrate the sound insulation device; the sound receiving system cannot deploy any equipment or sensors in the room where the sound source is located;

[0100] (ii) The sound recording system has no prior knowledge of the sound information emitted by the source speaker. The sound recording system is not only able to classify a small number of audio signals (e.g., words or numbers), but can also recreate any audio from an entire vocabulary (including complete sentences).

[0101] (iii) The sound receiving system is portable and relatively cheap. In this case, the sound receiving system can use low-cost commercial millimeter-wave radar to collect sound signals outside the soundproof space.

[0102] The sampling rate of all audio samples used in the experiments of the present invention is 44.1KHz. The present invention uses a typical conference room setting, with the speaker volume set to 70dB and the background noise of about 45dB (typical indoor office background noise). Figure 7 (a) shows a typical conference room scene used in the experiment of the present invention, with a wooden door blocking the middle. Figure 7 (b) shows a typical conference room scene used in the experiment of the present invention, with a glass partition in the middle. Under different settings, the present invention evaluates the impact of the "milli-ear" capture sensing distance and angle, isolation materials, etc. on each setting.

[0103] Specifically, the effectiveness of the system was evaluated from the following aspects:

[0104] Mel spectrum distortion:

[0105] Mel-spectral distortion (MCD) is an objective method for evaluating speech quality. It has been widely used to compare the quality of synthesized speech with the original and natural speech. Generally speaking, a smaller MCD score indicates a better similarity between the reconstructed audio and the original audio.

[0106] Likert Rating:

[0107] In order to conduct a subjective evaluation of the reconstructed audio, the present invention recruited 20 volunteers to listen to the restored audio. These participants included native and non-native English speakers aged between 20 and 30. The present invention asked them to listen to the reconstructed audio and the original audio one by one, and then rate the quality of the restored audio on a scale of 0 to 10. Here, the higher the Likert score, the better the quality of the reconstructed audio. 0 points means that the reconstructed audio is incomprehensible, while 10 points means that there is almost no difference between the reconstructed audio and the original audio.

[0108] In terms of numerical results, the present invention analyzes the experimental results of the present invention in two parts:

[0109] (i) Overall audio reconstruction performance of the “milli-ear”;

[0110] (ii) Robustness of milli-ears in different scenarios and settings.

[0111] Audio reconstruction performance:

[0112] The present invention first evaluates the ability of the human ear to reconstruct speech signals in a conference room environment, such as Figure 7 (a) and Figure 7 (b) Here, the mmWave sensor and speaker are separated by a double-panel glass wall. Figure 8 (a) shows the spectrogram of the original audio used by the user. Figure 8 (b) shows the spectrum generated directly from mmWave sampling without any enhancement. Figure 8 (c) shows the spectrogram of the audio reconstructed from the mmWave sample data using the cGAN model of the present invention. We observe that the spectrograms of the original sound and the reconstructed sound have a high similarity. This is because the cGAN model of the present invention is able to learn how to enhance the spectrogram of the mmWave sample data by reducing the noise in the mmWave sample data and adding specific acoustic data at different frequencies and amplitudes.

[0113] Fig. 9 The MCD scores for User1's voice to User4's voice are shown. Here, the cGAN model is trained and tested for each User's voice separately. It can be observed that the average MCD of all voices is less than 4. This means that the reconstructed audio is not only human-discernible, but also has a strong similarity to the original voice. Furthermore, the present invention also performs a subjective similarity assessment on the voice. Fig.10 (a) and Fig.10 (b) shows the median Likert ratings of 20 volunteers for four audio samples (including original and reconstructed). Fig.10 (a) and Fig.10As shown in (b), the median score for each user on both audio sample clips is above 6, which indicates that the milli-ear is capable of reconstructing clearly intelligible sounds.

[0114] In real situations, the sound receiving system may need to adjust the position of the millimeter wave sensor in order to collect sound signals. However, adjusting the position will change the distance and direction between the sound source device and the millimeter wave radar. Therefore, it is necessary to evaluate the stability of the "milli-ear" to different distances and directions. In the experiment of the present invention, the distance between the millimeter wave sensor and the speaker was changed from 1 meter to 5 meters, and the angle was changed from 0° to 45°.

[0115] Fig.11 (a) shows the performance of the vibration extraction of the proposed system for sound targets at different distances. Fig.11 (b) shows the vibration extraction performance of the present system for sound targets at different angles. The present invention uses the relative error er to evaluate the accuracy of extracting vibration signals from millimeter wave signals (without enhancement). Since the amplitudes are at different scales, the amplitudes are normalized before calculating the relative errors at different distances and angles. The relative error relative to the original audio is based on The derived results show that the normalized amplitude of the vibration waveform and the normalized amplitude of the original audio signal have achieved an average relative error of 8.9% in distance and 9.6% in angle, respectively. The comparison results show that the relative error between 0° and 45° is 8.8%, and the relative error between 0° and 45° is 10.2%. This shows that in the experiment of the present invention, the vibration extraction of the system has achieved a high accuracy.

[0116] Fig.12 (a) shows the MCD of 4 sound samples (User1 to User4), with the test distance ranging from 1 meter to 5 meters. The present invention observes that the MCD score increases accordingly, indicating that the reconstruction quality gradually decreases. However, within the detection range of the radar, the overall change is not obvious. Fig.12 (b) shows that angle has a greater impact on the quality of reconstructed audio than distance. This can be attributed to the fact that when there is a certain angle between the speaker and the millimeter wave radar, the vibration (i.e., reciprocating motion) of the speaker surface is increasingly difficult to capture by the radar. The present invention finds that the "milli-ear" system can reasonably and accurately reconstruct audio within 45°. This shows that the system proposed by the present invention can be used to collect sound at a reasonable distance and direction.

[0117] The spatial characteristics in the real environment are complex, and there will be a large number of sound insulation materials. The present invention conducts experiments to test the stability of the resistance of "milliars" to different insulation materials. The present invention selects 5 popular sound insulation boards, which are composed of dense wood, polyester fiber, cotton, glass and sound insulation gypsum. Fig.13As shown, except for glass, the performance of the “milli-ear” does not change significantly when the observed MCD is less than 4. Since glass is the strongest reflector of millimeter-wave signals among the materials studied, the quality of the sound reconstruction is somewhat degraded, but still within an acceptable range.

[0118] Considering that speakers from different manufacturers have different characteristics (background, materials, etc.), the present invention evaluates four different types of speakers. They are Philips SPA33, Philips SPA311, Edifier R12U, and Tmall IN. The present invention noticed that there is no cover on the speaker diaphragm of Philips SPA311 and Edifier R12U, while the diaphragm in the speaker box of Philips SPA33 and Tmall IN is covered. Fig.14 The results show that the audio reconstruction performance of Philips SPA311 and Edifier R12U is better than that of Philips SPA33 and Tmall IN because the vibrating surfaces of the former two are directly exposed to the mmWave sensor.

[0119] Model generalization of cross-user training. To demonstrate the generalizability of the model, we train and test the cGAN model for different users (cross-user training and testing). First, we train the model using the data of User 1 and test it using the data of User 2, User 3, and User 4. Fig.15 (a) shows the MCD reduction when the speech of User 2, User 3, and User 4 is tested with their own individually trained models and the model trained using the data of User 1. The present invention finds that although the audio reconstruction performance is significantly reduced, the overall performance is still within a reasonable range. This reduction can be attributed to the fact that different people's speech characteristics have different dominant frequency components, which are not always accurately reconstructed when training across users.

[0120] To evaluate whether adding more user data to training would further improve cross-user performance, the present invention trains the model using data from User 1 to User 4 and tests it on User 5 to User 7. Fig.15 (b) shows the MCD. We found that when more users’ data were considered in training, the model generalized better by learning to capture a more diverse set of acoustic features. These cross-user training results show that we can train the model offline using the audio data of a large number of users, and then sample and reconstruct the audio data of unknown users, making the audio reconstruction method proposed by the present invention more effective.

[0121] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio reconstruction method based on millimeter wave signals, characterized in that: The steps include: Step 1: Spectrum graph generation: (1) The millimeter-wave radar emits chirp signals at fixed time intervals. The chirp signals are reflected by the vibrating target and then returned to the millimeter-wave radar. Combined with the echo signal received by the millimeter-wave radar, the fast-time sampling signal is transformed by fast Fourier transform to identify the position and distance of the vibrating target. (2) According to the position and distance of the identified vibration target, the millimeter-wave radar is used to perform a slow-time sampling operation on the vibration target, and then the slow-time sampling signal is subjected to a Doppler fast Fourier transform to obtain a phase change, and the vibration displacement is obtained by using the phase change, and then all the vibration displacements are combined into a vibration waveform using a timestamp; (3) performing a short-time Fourier transform on the vibration waveform to obtain a time spectrum, and then converting the time spectrum into a millimeter-wave Mel spectrum; Step 2: Audio reconstruction: The millimeter wave Mel spectrum graph and the corresponding real audio Mel spectrum graph are input into the artificial intelligence model cGAN, and the model is trained. The generator in the model uses the millimeter wave Mel spectrum graph to generate a simulated audio Mel spectrum graph, and the discriminator in the model uses the real audio Mel spectrum graph to identify the authenticity of the simulated audio Mel spectrum graph. The generator and the discriminator are trained alternately, so that the simulated audio Mel spectrum graph generated by the generator is close to the real audio Mel spectrum graph; After the training is completed, the generator in the trained model is used to generate an audio Mel spectrogram from the collected millimeter-wave Mel spectrogram, and the Griffin-Lim algorithm is used to perform waveform synthesis on the generated audio Mel spectrogram to restore the audio waveform and achieve audio reconstruction.

2. The method for audio reconstruction based on millimeter wave signals according to claim 1, characterized in that: When identifying the position of a vibrating target, a sliding window is applied to the fast-time samples within a chirp to generate more precise sub-chirps, and then a range fast Fourier transform is performed on each precise sub-chirp to obtain multiple slow-time samples.

3. The method for audio reconstruction based on millimeter wave signals according to claim 1, characterized in that: The calculation formula of the vibration displacement is as follows: Among them, Δd is the vibration displacement, Δφ is the phase change, and λ is the wavelength of the millimeter wave signal.

4. The method for audio reconstruction based on millimeter wave signals according to claim 1, characterized in that: The formula for the short-time Fourier transform is as follows: Among them, h(τ-t) is the window function, τ is half of the window size at time t, x is the waveform of the millimeter wave signal, f is the frequency of the millimeter wave signal, and j is the complex base unit.

5. The method for audio reconstruction based on millimeter wave signals according to claim 1, characterized in that: The formula for converting a time spectrum graph into a Mel spectrum graph is as follows: Where mel(f) is the Mel spectrum and f is the frequency of the millimeter wave signal.

6. The method for audio reconstruction based on millimeter wave signals according to claim 1, characterized in that: The generator adopts the UNET architecture, and each convolution block in the generator and the discriminator consists of a 4×4 square kernel with a step size of 2; the discriminator includes three convolution blocks and true and false patch predictions, and the patch size is 30×30.

7. The method for audio reconstruction based on millimeter wave signals according to claim 1, characterized in that: During the model training process, the generator uses the millimeter-wave Mel-spectrogram as a conditional input and enhances it. The enhanced simulated Mel-spectrogram is concatenated with the millimeter-wave Mel-spectrogram and input into the discriminator, which classifies it as fake audio. When the combination of the real audio Mel-spectrogram and the millimeter-wave Mel-spectrogram is input, the discriminator classifies it as real audio. As training progresses, the generator learns to enhance the input, making it difficult for the discriminator to distinguish between the simulated mel-spectrograms enhanced by the generator and the real mel-spectrograms.

Images