Voice detection method based on millimeter wave radar

Abstract

A voice detection method based on millimeter wave radar, through millimeter wave radar, millimeter wave raw data are collected and pretreated, after extracting the object surface vibration signal, the vibration signal is decomposed into a plurality of target corresponding vibration signals through the voice separation network; the vibration signal is further identified through the convolution network to obtain the voice content of the target and detect the sensitive information therein. The object surface vibration in the multi-person speaking scene is collected, the vibration from different targets is separated by using the voice separation network, and the voice recognition module is sequentially fed, and finally the sensitive information in the voice content of each target is detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and particularly relates to a speech detection method based on millimeter wave radar. BACKGROUND

[0002] Speech detection technology can be classified according to the type of sensor used, including acoustic sensor-based, vision sensor-based, laser technology-based, and wireless signal-based methods. Acoustic sensor-based speech detection is easily hindered by anti-monitoring devices; vision sensor-based speech detection usually uses a high-speed camera to capture object vibration in the scene to detect speech, but is easily disturbed by light and shielding; laser-based speech detection technology is costly and is also easily affected by shielding; wireless signal-based speech detection technology either requires the sound source to be within the coverage range of the signal and directly detects the sound content through the vibration of the sound source, which has a high requirement for the position of the sound source, or indirectly detects through the vibration of surrounding objects, but cannot simultaneously detect the speech of multiple targets. SUMMARY

[0003] The present application aims to solve the problem that the prior art cannot effectively perceive speech when the sound source is not within the radar detection range, and provides a speech detection method based on millimeter wave radar, which collects object surface vibration in a multi-person speaking scene, separates the vibration from different targets using a speech separation network, sequentially inputs the vibration into a speech recognition module, and finally detects sensitive information in the speech content of each target.

[0004] The present application is achieved by the following technical solutions:

[0005] The present application relates to a speech detection method based on millimeter wave radar, which collects millimeter wave raw data using a millimeter wave radar, pre-processes the data, extracts object surface vibration signals, separates the vibration signals into vibration signals corresponding to multiple targets using a speech separation network, and further identifies the vibration signals using a convolution network to obtain the speech content of the targets and detect sensitive information therein.

[0006] The speech separation network includes an encoder, a corresponding decoder and a mask generator, wherein: the encoder converts the object surface vibration signal into a feature map, the decoder converts the feature map into a vector with a length of L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets.

[0007] The convolution network includes a convolution layer with a ReLU function, a normalization layer, a full connection layer and a Softmax layer as an output layer.

[0008] Technical effects

[0009] The present application is based on the technology of vibrating objects in the millimeter wave radar perception and positioning scene; the speech separation technology of separating the vibration signals of multiple targets from the object surface vibration signals; the technology of identifying the speech content from the vibration signals. Compared with the existing millimeter wave radar-based speech perception technology, which requires the sound source to be located within the detection range of the radar and can only detect the speech of one target, the present application can use the millimeter wave radar to capture the vibration of the objects around the sound source to restore the sound information, can detect even if the sound source is located outside the line of sight, and can identify the speech content of multiple targets at the same time by combining the speech separation technology. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The flowchart of the inventive system is shown in Figure 1.

[0011] Figure 2 The speech separation network structure designed for the present application is shown in Figure 2.

[0012] Figure 3 The flowchart of the speech recognition algorithm designed for the present application is shown in Figure 3.

[0013] Figure 4 The experimental scene diagram of the present application is shown in Figure 4.

[0014] Figure 5 The experimental results of the present application are shown in Figure 5. DETAILED DESCRIPTION

[0015] As shown in Figure 1, the present embodiment relates to a millimeter wave radar-based speech detection method, which includes the following steps: Figure 1

[0016] Step 1, signal acquisition: deploy the millimeter wave radar in the scene, use the 2-transmit 4-receive antenna configuration for data acquisition, and obtain the original radar data.

[0017] Step 2, pre-process the collected signals, which specifically includes:

[0018] 2.1 Perform one-dimensional FFT transformation on each frame of the original radar data to obtain an N*nFFT spectrum, where N is the number of frames and nFFT is the number of FFT transformation points.

[0019] In the present embodiment, nFFT is 1024.

[0020] 2.2 Remove the static clutter from the spectrum obtained in step 2.1 to distinguish the static background and dynamic targets, so as to accurately locate the vibrating objects and obtain a spectrum that only retains the dynamic target signals: use the average phase cancellation method to filter the static clutter, and use the difference in phase change between the echo signals generated by the static objects and the vibrating targets to suppress the clutter, which is specifically: ​Wherein: R[n,m] is a point in the spectrum, n is the coordinate in the time dimension, that is, the serial number of the frame, and m is the coordinate in the distance dimension.

[0021] 2.3 Calculate the variance of each distance point in the time dimension, and the distance point m with the maximum variance object As the object position, specifically:

[0022] 2.4 Calculate the phase of each frame spectrum at m object , obtain a phase sequence with a length of N, and then use phase unwrapping to restore the original phase sequence S, which contains the vibration information of the object surface.

[0023] Step three, input the phase sequence S obtained in step two into the speech separation network containing the encoder, the corresponding decoder and the mask generator, to obtain the speech signal, wherein: the encoder converts the object surface vibration signal into a feature map, the decoder converts the feature map into a vector with a length of L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets.

[0024] The input is preferably segmented into several overlapping 1xL segments before being input into the network, wherein: L is 128, and there is an overlapping segment with a size of 64 between adjacent segments.

[0025] As Figure 2 shown, the encoder includes: Pointwise (PW) convolution layers connected in sequence, corresponding ReLU activation functions and their normalization layers, and Depthwise (DW) convolution layers, corresponding ReLU activation functions and their normalization layers, wherein: the input vector of 1xL is first subjected to the PW convolution layer to obtain a vector with a size of 1xL, and the DW convolution layer is used to obtain a feature map with a size of MxL, and each convolution layer uses ReLU activation function and normalization layer.

[0026] The mask generator includes: 8 convolution blocks with the same structure, the outputs of the 8 convolution blocks are fused into a feature map with a size of MxL, and then subjected to ReLU activation, DW convolution layer and Sigmoid activation to obtain a mask with a size of CxMxL, wherein: C is the number of speakers, each number in the mask is between 0 and 1, the mask is multiplied by the feature map generated by the encoder to obtain C feature maps with a size of MxL, which are used as the input of the decoder.

[0027] The convolution block includes: a sequentially connected PW convolution layer and its corresponding ReLU activation function and normalization layer, a DW convolution layer and its corresponding ReLU activation function and normalization layer, and two parallel output PW convolution layers, wherein: both normalization layers output an M×L feature map, a branch of the output PW convolution layer is connected to the next convolution block for deeper feature extraction, and the branch of the other output PW convolution layer is connected to an addition operator. The features extracted by convolution blocks at different levels are directly summed, so that low-level signal features and high-level semantic features can be fully integrated to generate a more accurate mask representation.

[0028] The DW convolution layer is a dilated convolution layer, and the interval between its input elements is the dilation factor d, which takes values ​​of 1, 2, 4, 8…, 128 in the 8 convolution blocks.

[0029] The structure of the decoder is opposite to that of the encoder. The decoder outputs C vectors of length L, representing the separated signals S1, S2, ...Sc corresponding to the C targets.

[0030] Step 4: Figure 3 As shown, the C signals obtained in step 3 are respectively subjected to speech recognition, specifically including:

[0031] 4.1 Divide the spectrum of the C signals obtained in step 3 into the spectrum of several words: For the i-th signal Si, first perform short-time Fourier transform (STFT) on Si to generate a time spectrum, and calculate the energy of each frame in the time spectrum Where nFFT is the number of Fourier transform points, and Spec(t,i) represents the i-th value of the spectrum in the t-th frame. Set the spectrum energy threshold Et. M consecutive frames with energy below Et are used as the demarcation points between words. Based on all demarcation points, the spectrum is split into several spectrograms containing only single words.

[0032] In this embodiment, Et=1000, M=100.

[0033] 4.2 Classify the word spectrogram obtained in step 4.1 through a lightweight convolutional neural network, and select the category with the highest probability as the category of the current word spectrogram to identify sensitive information in the speech.

[0034] like Figure 3As shown, the convolutional neural network comprises a convolution layer with a ReLU function, a normalization layer, a full connection layer, and a Softmax layer as an output layer, wherein: the input spectrum is scaled to 64x64 size, and a 3x64x64 feature map is obtained through the first convolution layer. Then, two convolution blocks with the same structure are followed, each block consisting of 4 3x3 convolution layers and 1 normalization layer. The feature maps obtained by the two convolution blocks are flattened into one-dimensional vectors for input of the full connection layer. The output of the last full connection layer is activated by Softmax to obtain a probability vector.

[0035] The meaning of the category usually depends on the definition of sensitive information, for example, if a number is taken as sensitive information, each number corresponds to a category, and there are a total of 12 categories of zero, one, two, three, …, ten and non-number.

[0036] 4.3 The classification results of all word spectrum graphs are integrated to detect the sensitive content in the target voice.

[0037] Through specific actual experiments, as shown in Figure 4 As shown, multiple speakers are used to simulate multiple speaking targets in the scene, and the voice containing digital information is played at the same time. The vibrating object is a tin foil, and the Texas Instruments IWR1443 millimeter wave radar is used to collect data on the tin foil. The radar uses a 2-transmit and 4-receive antenna configuration, and the 2 transmitting antennas alternately transmit frequency-modulated signals, a total of 510 times per frame. The transmission period of each transmitting antenna is 53.6 microseconds, which can achieve a sampling rate of 18kHz. The experimental data obtained are as shown in Figure 5 As shown, the recognition accuracy of the 11 categories of numbers from zero to ten in the voice is 0.72, 0.68, 0.84, 0.79, 0.80, 0.81, 0.86, 0.79, 0.82, 0.84 and 0.88, and the recognition recall rate is 0.92, 0.89, 0.91, 0.86, 0.96, 0.88, 0.94, 0.85, 0.94, 0.94 and 0.96. After repeated testing, the accuracy of the system in recognizing numbers in the voice is 84.85%, and the recall rate is 89.16%.

[0038] Compared with the prior art which cannot simultaneously detect the voice content of multiple targets, the present application combines a voice separation technology, can effectively identify digital information from multiple targets speaking at the same time, and achieve high recognition accuracy and recall rate, so as to detect the key content in the voice. Through the millimeter wave technology, the vibration of the object surface in the scene is perceived, rather than directly perceiving the sound source. This indirect detection method does not require the millimeter wave to directly cover the sound source position. The present application combines a voice separation algorithm to separate the vibration information affected by multiple sound sources, so as to obtain the voice information of multiple targets, which can better meet the needs of real scenes.

[0039] The above specific embodiments can be modified in various ways by those skilled in the art without departing from the principles and spirit of the present application, the scope of protection of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation within the scope thereof is subject to the present application.

Claims

1. A voice detection method based on millimeter wave radar, characterized in that: After collecting and preprocessing raw millimeter-wave data through millimeter-wave radar, the surface vibration signal is extracted and decomposed into vibration signals corresponding to multiple targets through a speech separation network. The vibration signal is further identified through a convolutional network to obtain the target's speech content and detect sensitive information. The speech separation network includes: an encoder and its corresponding decoder and a mask generator, wherein: the encoder converts the surface vibration signal of the object into a feature map, the decoder converts the feature map into a vector of length L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets; The convolutional network includes: a convolutional layer with a ReLU function, a normalization layer, a fully connected layer, and a Softmax layer as an output layer; The voice detection specifically includes: Step 1: Signal acquisition: Deploy the millimeter-wave radar in the scene to collect data and obtain raw radar data; Step 2: pre-process the collected raw radar data to obtain a phase sequence; Step 3: Use the phase sequence obtained in step 2 to input a speech separation network including an encoder and its corresponding decoder and a mask generator to obtain a speech signal, wherein: the encoder converts the surface vibration signal of the object into a feature map, the decoder converts the feature map into a vector of length L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets; Step 4: Perform speech recognition on the C signals obtained in step 3, specifically including: 4.1 Divide the spectrum of the C signals obtained in step 3 into the spectrum of several words: For the i-th signal Si, first perform short-time Fourier transform on Si to generate a time spectrum, and calculate the energy of each frame in the time spectrum , where nFFT is the number of Fourier transform points, Spec(t,i) represents the i-th value of the spectrum of the t-th frame; set the spectrum energy threshold Et, and when there are M consecutive frames with energy lower than Et, they are used as the dividing points between words; according to all the dividing points, the spectrum is divided into several spectrograms containing only single words; 4.2 Classify the word spectrogram obtained in step 4.1 using a lightweight convolutional neural network, and select the category with the highest probability as the category of the current word spectrogram to identify sensitive information in the speech; 4.3 By integrating the classification results of all word spectrograms, the sensitive content in the target speech can be detected.

2. The method for voice detection based on millimeter wave radar according to claim 1, wherein: The step 2 is specifically include: 2.1 Perform a one-dimensional FFT transform on each frame of the original radar data to obtain an N*nFFT spectrum, where N is the number of frames and nFFT is the number of FFT transform points; 2.2 Remove static clutter from the spectrum obtained in step 2.1, distinguish the static background from the dynamic target, accurately locate the vibrating object, and obtain a spectrum that only retains the dynamic target signal: Use the average phase cancellation method to filter static clutter and use the difference in phase change between the echo signals generated by the static object and the vibrating target to achieve clutter suppression. Specifically: , where: R[n,m] is a point in the spectrum, n is the coordinate in the time dimension, that is, the frame number, and m is the coordinate in the distance dimension; 2.3 Calculate the variance of each distance point in the time dimension, and the distance point with the largest variance As the object position, specifically: ; 2.4 Calculate the spectrum of each frame The phase at , a phase sequence of length N is obtained, and then phase unwrapping is performed on it to restore it to the original phase sequence S containing the vibration information of the object surface.

3. The method for voice detection based on millimeter wave radar according to claim 1, wherein: The input is to divide the phase sequence S into several 1×L overlapping segments and then input them into the network, where: L is 128, and there are overlapping segments of size 64 between adjacent segments.

4. The method for voice detection based on millimeter wave radar according to claim 1, wherein: The encoder includes: a Pointwise (PW) convolution layer, a corresponding ReLU activation function and a normalization layer, and a Depthwise (DW) convolution layer, a corresponding ReLU activation function and a normalization layer, which are connected in sequence. The 1×L input vector first passes through the PW convolution layer to obtain a 1×L vector, and then passes through the DW convolution layer to obtain an M×L feature map. Each convolution layer uses a ReLU activation function and a normalization layer.

5. The method for voice detection based on millimeter wave radar according to claim 4, wherein: The mask generator includes: 8 convolution blocks with the same structure. The outputs of the 8 convolution blocks are fused into an M×L feature map. Then, after ReLU activation, DW convolution layer and Sigmoid activation, a C×M×L mask is obtained, where: C is the number of speakers, and each number in the mask is between 0 and 1. The mask is multiplied with the feature map generated by the encoder to obtain C M×L feature maps as the input of the decoder.

6. The method for voice detection based on millimeter wave radar according to claim 5, wherein: The convolution block includes: a sequentially connected PW convolution layer and its corresponding ReLU activation function and normalization layer, a DW convolution layer and its corresponding ReLU activation function and normalization layer, and two parallel output PW convolution layers, wherein: both normalization layers output an M×L feature map, a branch of the output PW convolution layer is connected to the next convolution block for deeper feature extraction, and the branch of the other output PW convolution layer is connected to an addition operator. The features extracted by convolution blocks at different levels are directly summed, so that low-level signal features and high-level semantic features can be fully integrated to generate a more accurate mask representation.

7. The method for voice detection based on millimeter wave radar according to claim 1, wherein: The structure of the decoder is opposite to that of the encoder. The decoder outputs C vectors of length L, representing the separated signals S1, S2, ...Sc corresponding to the C targets.

Images