Voice detection method based on millimeter wave radar

Abstract

The invention discloses a voice detection method based on a millimeter wave radar, and the method comprises the steps: collecting millimeter wave original data through the millimeter wave radar, carrying out the preprocessing, extracting an object surface vibration signal, and decomposing the vibration signal into vibration signals corresponding to a plurality of targets through a voice separation network; and further identifying the vibration signal through the convolutional network to obtain the voice content of the target, and detecting the sensitive information in the voice content. According to the invention, object surface vibration in a multi-person speaking scene is collected, vibration from different targets is separated by using a voice separation network, the vibration is sequentially sent to a voice recognition module, and finally sensitive information in the voice content of each target is detected.

Description

Technical Field

[0001] The present invention relates to a technology in the field of speech processing, in particular to a speech detection method based on millimeter wave radar. Background Art

[0002] Voice detection technology can be classified according to the type of sensor used, including acoustic sensor-based, visual sensor-based, laser technology-based, and wireless signal-based. Voice detection based on acoustic sensors is easily hindered by anti-eavesdropping devices; voice detection based on visual sensors usually uses high-speed cameras to capture the vibration of objects in the scene to detect voice, but is easily interfered by light and occlusion; laser-based voice detection technology is expensive and is also easily affected by occlusion; voice detection technology based on wireless signals either requires the sound source to be within the coverage range of the signal and directly detects the sound content through the vibration of the sound source, which has high requirements for the location of the sound source, or detects indirectly through the vibration of surrounding objects, but cannot detect the voices of multiple targets at the same time. Summary of the invention

[0003] In view of the deficiency of the prior art that the sound source cannot achieve effective speech perception when the sound source is not within the radar detection range, the present invention proposes a speech detection method based on millimeter wave radar, which collects the surface vibrations of objects in a scene where multiple people are talking, and uses a speech separation network to separate the vibrations from different targets, which are sent to the speech recognition module in turn, and finally the sensitive information in the speech content of each target is detected.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a speech detection method based on millimeter wave radar. After collecting and preprocessing millimeter wave raw data through the millimeter wave radar, after extracting the vibration signal of the object surface, the vibration signal is decomposed into vibration signals corresponding to multiple targets through a speech separation network; the vibration signal is further identified through a convolutional network to obtain the speech content of the target and detect sensitive information therein.

[0006] The speech separation network includes: an encoder and its corresponding decoder and a mask generator, wherein: the encoder converts the surface vibration signal of the object into a feature map, the decoder converts the feature map into a vector with a length of L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets.

[0007] The convolutional network includes: a convolutional layer with a ReLU function, a normalization layer, a fully connected layer, and a Softmax layer as an output layer. Technical Effects

[0008] The present invention is based on the technology of sensing and locating vibrating objects in the scene by millimeter-wave radar; the speech separation technology of separating the vibration signals of multiple targets from the vibration signals on the surface of the object; and the technology of identifying the speech content from the vibration signal. Compared with the existing technology of speech perception based on millimeter-wave radar, which requires the sound source to be within the detection range of the radar and can only detect the speech of one target, the present invention can use millimeter-wave radar to capture the vibration of objects around the sound source and restore the sound information. It can detect even if the sound source is out of sight, and combined with the speech separation technology, it can simultaneously identify the speech content of multiple targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 To invent a system flow chart;

[0010] Figure 2 The speech separation network structure designed for the present invention;

[0011] Figure 3 Flowchart of the speech recognition algorithm designed for the present invention;

[0012] Figure 4 It is an illustration of the experimental scene of the present invention;

[0013] Figure 5 This is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0014] like Figure 1 As shown, this embodiment relates to a voice detection method based on millimeter wave radar, including:

[0015] Step 1: Signal collection: Deploy the millimeter-wave radar in the scene and use a 2-transmit 4-receive antenna configuration to collect data to obtain raw radar data.

[0016] Step 2: pre-process the collected signal, including:

[0017] 2.1 Perform a one-dimensional FFT transform on each frame of the original radar data to obtain an N*nFFT spectrum, where N is the number of frames and nFFT is the number of points of the FFT transform.

[0018] In this embodiment, the value of nFFT is 1024.

[0019] 2.2 Remove static clutter from the spectrum obtained in step 2.1, distinguish static background from dynamic targets, accurately locate vibrating objects, and obtain a spectrum that only retains dynamic target signals: Use the average phase cancellation method to filter static clutter, and use the difference in phase change between the echo signals generated by static objects and vibrating targets to suppress clutter, specifically: Among them: R[n,m] is a point in the spectrum, n is the coordinate in the time dimension, that is, the frame number, and m is the coordinate in the distance dimension.

[0020] 2.3 Calculate the variance of each distance point in the time dimension, and the distance point with the largest variance is m object As the object position, specifically:

[0021] 2.4 Calculate the spectrum of each frame in m object The phase at , we get a phase sequence of length N, and then use phase unwrapping to restore it to the original phase sequence S, S contains the vibration information of the object surface.

[0022] Step three, use the phase sequence S obtained in step two to input the speech separation network including the encoder and its corresponding decoder and mask generator to obtain the speech signal, wherein: the encoder converts the surface vibration signal of the object into a feature map, the decoder converts the feature map into a vector of length L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets.

[0023] The input is preferably input into the network after the phase sequence S is divided into a number of 1×L overlapping segments, wherein: L is 128, and there are overlapping segments of size 64 between adjacent segments.

[0024] like Figure 2 As shown, the encoder includes: Pointwise (PW) convolutional layers, corresponding ReLU activation functions and normalization layers, and Depthwise (DW) convolutional layers, corresponding ReLU activation functions and normalization layers connected in sequence, wherein: a 1×L input vector first passes through the PW convolutional layer to obtain a 1×L vector, and then passes through the DW convolutional layer to obtain a feature map of M×L size, and each convolutional layer uses a ReLU activation function and a normalization layer.

[0025] The mask generator includes: 8 convolution blocks with the same structure, the outputs of the 8 convolution blocks are fused into an M×L feature map, and then through ReLU activation, DW convolution layer and Sigmoid activation, a C×M×L mask is obtained, where: C is the number of speakers, each number in the mask is between 0 and 1, the mask is multiplied with the feature map generated by the encoder, and C M×L feature maps are obtained as the input of the decoder.

[0026] The convolution block includes: a PW convolution layer and its corresponding ReLU activation function and normalization layer, a DW convolution layer and its corresponding ReLU activation function and normalization layer, and two parallel output PW convolution layers, wherein: the two normalization layers each output an M×L feature map, a branch of an output PW convolution layer is connected to the next convolution block for deeper feature extraction, and another branch of the output PW convolution layer is connected to an addition operator. The features extracted by convolution blocks at different levels are directly summed, so that low-level signal features and high-level semantic features can be fully integrated to generate a more accurate mask representation.

[0027] The DW convolution layer is a dilated convolution layer, and the interval of its input elements is the dilation factor d, and its values ​​in the 8 convolution blocks are 1, 2, 4, 8..., 128 respectively.

[0028] The structure of the decoder is opposite to that of the encoder. The decoder outputs C vectors of length L, representing the separated signals S1, S2, ...Sc corresponding to the C targets.

[0029] Step 4: Figure 3 As shown, the C signals obtained in step 3 are respectively subjected to speech recognition, specifically including:

[0030] 4.1 Divide the spectrum of the C signals obtained in step 3 into the spectrum of several words: For the i-th signal Si, first perform short-time Fourier transform (STFT) on Si to generate a time spectrum, and calculate the energy of each frame in the time spectrum Where: nFFT is the number of Fourier transform points, Spec(t,i) represents the i-th value of the spectrum of the t-th frame. Set the energy threshold Et of the spectrum. When there are M consecutive frames with energy lower than Et, it is used as the dividing point between words. According to all the dividing points, the spectrum is divided into several spectrograms containing only a single word.

[0031] In this embodiment, Et=1000, M=100.

[0032] 4.2 Classify the word spectrogram obtained in step 4.1 through a lightweight convolutional neural network, and select the category with the highest probability as the category of the current word spectrogram to identify sensitive information in the speech.

[0033] like Figure 3As shown, the convolutional neural network includes: a convolution layer with a ReLU function, a normalization layer, a fully connected layer, and a Softmax layer as an output layer, wherein: the input spectrogram is scaled to a size of 64×64, and a 3×64×64 feature map is obtained after the first convolution layer. Next are two convolution blocks with the same structure, each block consisting of 4 3×3 convolution layers and 1 normalization layer. The feature maps obtained by the two convolution blocks are converted into a one-dimensional vector for the input of the fully connected layer after the flattening layer. The output of the last fully connected layer is activated with Softmax to obtain a probability vector.

[0034] The meaning of the categories usually depends on the definition of sensitive information. For example, if numbers are used as sensitive information, each number corresponds to a category, and there are a total of 12 categories, including zero, one, two, three..., ten and non-number.

[0035] 4.3 By integrating the classification results of all word spectrograms, the sensitive content in the target speech can be detected.

[0036] After specific practical experiments, such as Figure 4 As shown, multiple speakers are used to simulate multiple speaking targets in the scene, and voices containing digital information are played at the same time. The vibrating object is tinfoil, and the IWR1443 millimeter-wave radar of Texas Instruments is aimed at the tinfoil for data collection. The radar adopts a 2-transmit 4-receive antenna configuration. The two transmitting antennas transmit FM signals alternately, and each frame transmits 510 times in total. The transmission period of each transmitting antenna is 53.6 microseconds, which can achieve a sampling rate of 18kHz. The experimental data that can be obtained are: Figure 5 As shown in the figure, the recognition accuracy rates of the 11 digit categories from zero to ten in speech are 0.72, 0.68, 0.84, 0.79, 0.80, 0.81, 0.86, 0.79, 0.82, 0.84 and 0.88 respectively, and the recognition recall rates are 0.92, 0.89, 0.91, 0.86, 0.96, 0.88, 0.94, 0.85, 0.94, 0.94 and 0.96 respectively. After repeated tests, the accuracy rate of the system in recognizing digits in speech is 84.85%, and the recall rate is 89.16%.

[0037] Compared with the existing technology that cannot detect the voice content of multiple targets at the same time, the present invention combines speech separation technology, which can effectively identify digital information from multiple targets speaking at the same time, achieving a high recognition accuracy and recall rate, thereby detecting the key content in the speech. This indirect detection method uses millimeter wave technology to perceive the surface vibration of objects in the scene rather than directly sensing the sound source. It does not require millimeter waves to directly cover the sound source position. The present invention also combines a speech separation algorithm to separate the vibration information affected by multiple sound sources, thereby obtaining the voice information of multiple targets, which can better meet the needs of real-life scenarios.

[0038] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A voice detection method based on millimeter wave radar, characterized in that: After the millimeter wave raw data is collected and preprocessed by the millimeter wave radar, the vibration signal on the surface of the object is extracted and then decomposed into vibration signals corresponding to multiple targets through the speech separation network; the vibration signal is further identified through the convolutional network to obtain the target's speech content and detect sensitive information in it; The speech separation network includes: an encoder and its corresponding decoder and a mask generator, wherein: the encoder converts the surface vibration signal of the object into a feature map, the decoder converts the feature map into a vector of length L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets; The convolutional network includes: a convolutional layer with a ReLU function, a normalization layer, a fully connected layer, and a Softmax layer as an output layer.

2. The method for voice detection based on millimeter wave radar according to claim 1, wherein: include: Step 1: Signal collection: Deploy the millimeter-wave radar in the scene to collect data and obtain raw radar data; Step 2: preprocess the collected raw radar data to obtain a phase sequence; Step 3, using the phase sequence obtained in step 2 to input a speech separation network including an encoder and its corresponding decoder and a mask generator to obtain a speech signal, wherein: the encoder converts the surface vibration signal of the object into a feature map, the decoder converts the feature map into a vector of length L, and the mask generator extracts features from the feature map to generate masks corresponding to different targets; Step 4: Perform speech recognition on the C signals obtained in step 3 respectively, which specifically includes: 4.1 Divide the spectrum of the C signals obtained in step 3 into the spectrum of several words: For the i-th signal Si, first perform short-time Fourier transform on Si to generate a time spectrum, and calculate the energy of each frame in the time spectrum Where: nFFT is the number of Fourier transform points, Spec(t,i) represents the i-th value of the spectrum of the t-th frame; the energy threshold Et of the spectrum is set, and when there are M consecutive frames with energy lower than Et, they are used as the dividing points between words; the spectrum is divided into several spectrograms containing only a single word according to all the dividing points; 4.2 Classify the word spectrogram obtained in step 4.1 through a lightweight convolutional neural network, and select the category with the highest probability as the category of the current word spectrogram to identify sensitive information in the speech; 4.3 By integrating the classification results of all word spectrograms, the sensitive content in the target speech can be detected.

3. The method for voice detection based on millimeter wave radar according to claim 2, characterized in that: The step 2 is as follows: include: 2.1 Perform a one-dimensional FFT transformation on each frame of the original radar data to obtain an N*nFFT spectrum, where N is the number of frames and nFFT is the number of points of the FFT transformation; 2.2 Remove static clutter from the spectrum obtained in step 2.1, distinguish static background from dynamic targets, accurately locate vibrating objects, and obtain a spectrum that only retains dynamic target signals: Use the average phase cancellation method to filter static clutter, and use the difference in phase change between the echo signals generated by static objects and vibrating targets to suppress clutter, specifically: Where: R[n,m] is a point in the spectrum, n is the coordinate in the time dimension, that is, the frame number, and m is the coordinate in the distance dimension; 2.3 Calculate the variance of each distance point in the time dimension, and the distance point with the largest variance is m object As the object position, specifically: 2.4 Calculate the spectrum of each frame in m object The phase at the position is obtained to obtain a phase sequence of length N, which is then phase unwrapped to restore it to the original phase sequence S containing the vibration information of the object surface.

4. The method for voice detection based on millimeter wave radar according to claim 2, characterized in that: The input is to divide the phase sequence S into a number of 1×L overlapping segments and then input them into the network, where: L is 128, and there are overlapping segments of size 64 between adjacent segments.

5. The method for voice detection based on millimeter wave radar according to claim 2, characterized in that: The encoder includes: a Pointwise (PW) convolution layer, a corresponding ReLU activation function and a normalization layer, and a Depthwise (DW) convolution layer, a corresponding ReLU activation function and a normalization layer, which are connected in sequence. The 1×L input vector first passes through the PW convolution layer to obtain a 1×L vector, and then passes through the DW convolution layer to obtain a feature map of M×L size. Each convolution layer uses a ReLU activation function and a normalization layer.

6. The method for voice detection based on millimeter wave radar according to claim 5, characterized in that: The mask generator includes: 8 convolution blocks with the same structure, the outputs of the 8 convolution blocks are fused into an M×L feature map, and then through ReLU activation, DW convolution layer and Sigmoid activation, a C×M×L mask is obtained, where: C is the number of speakers, each number in the mask is between 0 and 1, the mask is multiplied with the feature map generated by the encoder, and C M×L feature maps are obtained as the input of the decoder.

7. The method for voice detection based on millimeter wave radar according to claim 6, characterized in that: The convolution block includes: a PW convolution layer and its corresponding ReLU activation function and normalization layer, a DW convolution layer and its corresponding ReLU activation function and normalization layer, and two parallel output PW convolution layers, wherein: the two normalization layers each output an M×L feature map, a branch of an output PW convolution layer is connected to the next convolution block for deeper feature extraction, and another branch of the output PW convolution layer is connected to an addition operator. The features extracted by convolution blocks at different levels are directly summed, so that low-level signal features and high-level semantic features can be fully integrated to generate a more accurate mask representation.

8. The method for voice detection based on millimeter wave radar according to claim 2, characterized in that: The structure of the decoder is opposite to that of the encoder. The decoder outputs C vectors of length L, representing the separated signals S1, S2, ...Sc corresponding to the C targets.

Images