In-vehicle gesture recognition method and system based on directional loudspeaker

Abstract

The invention relates to an in-vehicle gesture recognition method and system based on a directional loudspeaker, and the method comprises the following steps: collecting gesture data through the directional loudspeaker, and extracting an audio clip containing a gesture, thereby obtaining a gesture waveform; preprocessing the gesture waveform; sTFT transformation is carried out on the preprocessed gesture waveform, the gesture waveform is converted into a spectrogram, and the spectrogram is evenly cut and then serves as input of a gesture recognition model; and gesture recognition is carried out by using a gesture recognition model constructed based on the ResNet network. Compared with the prior art, the method has the advantages of high recognition accuracy, high recognition speed and the like.

Description

Technical Field

[0001] The present invention relates to the field of automobile control technology, and in particular to an in-vehicle gesture recognition method and system based on a directional loudspeaker. Background Art

[0002] With the vigorous development of the automobile industry, especially new energy vehicles, vehicle systems are becoming more and more intelligent, and the interactions between users and vehicles are becoming more and more numerous and complex. However, in the existing interaction methods, using the car screen or the user's smartphone will have a greater impact on the user's driving behavior; although the use of physical buttons has less impact on driving after proficiency, it is relatively rigid and difficult to change the preset operations. And as the car screen becomes larger and larger, the space left for physical buttons is becoming more and more limited. At the same time, acoustic sensors are increasingly commonly used in driving environments. Some studies have also shown that gesture interaction is what consumers want, and it can effectively avoid accidents caused by distraction during interaction.

[0003] Current acoustic-based gesture recognition systems are challenging to use in a vehicle environment due to the complex interference of the in-vehicle environment. Although there have been studies using beamforming to sense gestures, the large size of existing beamforming through speaker arrays makes it difficult to deploy in vehicles.

[0004] Current directional speakers can provide a directional sound field, which is available in a vehicle environment. Although different people in a vehicle usually have different listening needs, directional sound waves can be adjusted to meet different users according to different needs. However, current products do not deeply explore the sensing potential of directional speakers. The challenges faced in the process of using directional speakers for gesture recognition are as follows: 1. The vehicle environment while driving is a complex environment with multipath reflections brought by different objects, and it is difficult to accurately identify the specific source of the received signal. 2. Conflict between transmitting and receiving frequencies. In order to receive sound without affecting the sound sent to the user, the frequency bands of the two sound waves must be carefully designed to avoid crosstalk. 3. The user's gesture changes are usually fast and subtle, which makes it difficult to ensure recognition accuracy. Summary of the invention

[0005] The purpose of the present invention is to provide an in-vehicle gesture recognition method and system based on directional speakers, which utilizes the highly directional sound waves generated by directional speakers, filters out the frequencies generated by user gestures from the received frequencies through frequency analysis, and then uses ResNet-based deep learning to classify different spectrum graphs, thereby achieving high-precision gesture detection in the car.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for in-vehicle gesture recognition based on a directional speaker comprises the following steps:

[0008] The gesture data is collected by using a directional speaker, and the audio clip containing the gesture is extracted to obtain the gesture waveform;

[0009] Preprocess the gesture waveform;

[0010] Perform STFT transformation on the preprocessed gesture waveform and convert it into a spectrum graph, and evenly crop the spectrum graph as the input of the gesture recognition model;

[0011] Gesture recognition is performed using a gesture recognition model built based on the ResNet network.

[0012] The method for extracting the audio segment containing the gesture is:

[0013] Mixing a transmission signal emitted by the directional loudspeaker and a reflection signal received by the microphone after the transmission signal is reflected by the gesture;

[0014] The mixed signal is processed by low-pass filtering and fast Fourier transform in sequence to obtain spectrum data;

[0015] Peak detection is performed on the spectrum data. When the third peak is detected and the peak value of the third peak exceeds a preset threshold, a time window is recorded. The time window starts at the moment when the third peak is detected and ends at the moment when the third peak drops below the preset threshold. The mixed signal corresponding to the time window is used as the gesture waveform.

[0016] The preprocessing specifically includes: taking into account the noise existing in the current, using spectral subtraction to perform noise reduction processing on the gesture audio segment.

[0017] The uniformly cropped spectrogram is input into the gesture recognition model. The spectrogram is preprocessed into a 3×224×224 input, and then processed through the convolution layer, channel attention layer, spatial attention layer and maximum pooling layer to obtain the first feature map with a dimension of 64×64×56.

[0018] The first feature map is input into the residual module of ResNet18, and after residual operations including convolution, regularization, and identity mapping, a 512×7×7 second feature map is obtained;

[0019] The second feature map is further processed through channel attention, spatial attention and average pooling layers to obtain 512 1×1 third feature maps;

[0020] The third feature map is used as the input of the fully connected layer, and the gesture classification result is obtained and output after being processed by three layers of fully connected layers.

[0021] The kernel size of the convolutional layer and the pooling layer is 3×3, the padding is 1, and the stride is 2.

[0022] An in-vehicle gesture recognition system based on a directional speaker, comprising:

[0023] A directional speaker, used to send a transmission signal to a position in the vehicle where gesture recognition is required, wherein the transmission signal is a frequency modulated continuous wave signal with a frequency range of 70KHz-80KHz and a chirp of 0.1s;

[0024] A microphone is used to receive a reflected signal of a transmitted signal after being reflected by a gesture;

[0025] A signal processing module is used to extract an audio segment containing a gesture based on a transmitted signal and a reflected signal to obtain a gesture waveform, preprocess the gesture waveform, and perform STFT transformation on the preprocessed gesture waveform to convert it into a frequency spectrum;

[0026] The gesture recognition module uses the uniformly cropped frequency spectrum as input and performs gesture recognition using a gesture recognition model built based on a ResNet network.

[0027] The method for extracting the audio segment containing the gesture is:

[0028] Mixing a transmission signal emitted by the directional loudspeaker and a reflection signal received by the microphone after the transmission signal is reflected by the gesture;

[0029] The mixed signal is processed by low-pass filtering and fast Fourier transform in sequence to obtain spectrum data;

[0030] Peak detection is performed on the spectrum data. When the third peak is detected and the peak value of the third peak exceeds a preset threshold, a time window is recorded. The time window starts at the moment when the third peak is detected and ends at the moment when the third peak drops below the preset threshold. The mixed signal corresponding to the time window is used as the gesture waveform.

[0031] The preprocessing specifically includes: taking into account the noise existing in the current, using spectral subtraction to perform noise reduction processing on the gesture audio segment.

[0032] The uniformly cropped spectrogram is input into the gesture recognition model. The spectrogram is preprocessed into a 3×224×224 input, and then processed through the convolution layer, channel attention layer, spatial attention layer and maximum pooling layer to obtain the first feature map with a dimension of 64×64×56.

[0033] The first feature map is input into the residual module of ResNet18, and after residual operations including convolution, regularization, and identity mapping, a 512×7×7 second feature map is obtained;

[0034] The second feature map is further processed through channel attention, spatial attention and average pooling layers to obtain 512 1×1 third feature maps;

[0035] The third feature map is used as the input of the fully connected layer, and the gesture classification result is obtained and output after being processed by three layers of fully connected layers.

[0036] The kernel size of the convolutional layer and the pooling layer is 3×3, the padding is 1, and the stride is 2.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention realizes the directional detection of the driver's gestures by using a directional speaker, which can effectively improve the driver's driving experience and driving safety.

[0039] (2) The present invention introduces an attention layer into the ResNet neural network, converts the measured audio signal into a spectrogram, and realizes gesture recognition through deep learning, thereby improving the recognition speed while ensuring the recognition accuracy.

[0040] (3) It is verified that the present invention can still maintain good task accuracy under conditions of different clothing and people of different genders and ages. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 A gesture and its corresponding spectrum diagram in an embodiment;

[0043] Figure 3 A schematic diagram of a network structure of a gesture recognition model in an embodiment;

[0044] Figure 4 A schematic diagram of the accuracy of gesture recognition results in an embodiment. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] This embodiment provides a method for in-vehicle gesture recognition based on a directional speaker. Figure 1As shown, the following steps are included:

[0047] S1, using a directional speaker to collect gesture data and extracting audio clips containing gestures to obtain gesture waveforms.

[0048] Specifically, the following steps are included:

[0049] S11, the directional speaker emits a FMCW (Frequency-Modulated Continuous Wave) signal with a frequency range of 70KHz-80KHz and a chirp of 0.1s. The transmission signal is received by the microphone after being reflected by the gesture to obtain a reflected signal, and the transmission signal and the reflected signal are mixed.

[0050] S12, performing low-pass filtering and fast Fourier transform processing on the mixed signal in sequence to obtain spectrum data.

[0051] S13, peak detection is performed on the spectrum data. The highest peak and the second highest peak in the spectrum are respectively the reflected signals generated by the transmission signal and the human body. Therefore, the detected third peak is the gesture signal. When the third peak is detected and the peak value of the third peak exceeds the preset threshold value S, a time window t is recorded. The time window starts at the time when the third peak is detected and ends at the time when the third peak drops below the preset threshold value S. The mixed signal corresponding to the time window t is used as the gesture waveform.

[0052] S2, preprocessing the gesture waveform: considering the noise in the current, spectral subtraction is used to reduce the noise of the gesture audio clip to optimize the signal-to-noise ratio.

[0053] S3, performs STFT transformation on the preprocessed gesture waveform and converts it into a spectrum diagram, such as Figure 2 As shown in the figure, the spectrum graph is evenly cropped and used as the input of the gesture recognition model.

[0054] S4, gesture recognition is performed using a gesture recognition model built based on the ResNet network.

[0055] like Figure 3 As shown in the figure, the uniformly cropped spectrogram is input into the gesture recognition model. The spectrogram is preprocessed into a 3×224×224 input, and then processed through the convolution layer, the channel attention layer, the spatial attention layer and the maximum pooling layer to obtain the first feature map with a dimension of 64×64×56.

[0056] The first feature map is input into the residual module of ResNet18, and after residual operations including convolution, regularization, and identity mapping, a 512×7×7 second feature map is obtained;

[0057] The second feature map is further processed through channel attention, spatial attention and average pooling layers to obtain 512 1×1 third feature maps;

[0058] The third feature map is used as the input of the fully connected layer, and the gesture classification result is obtained and output after being processed by three layers of fully connected layers.

[0059] In this embodiment, the kernel size of the convolution layer and the pooling layer is 3×3, the padding is 1, and the stride is 2.

[0060] This embodiment collects 6000 sets of gesture data, divides them into 70% training set, 20% validation set and 10% test set, and trains the neural network. On the test set, the overall gesture recognition F1 score is 92.3%, and the average accuracy is 91.7%, indicating the feasibility and applicability of the present invention in realizing gesture interaction in a vehicle environment. The influence of different gloves on the results is tested, and the test results show that in the worst case, there is still an 82.5% recognition accuracy rate.

[0061] In addition to the ResNet network, this embodiment uses VGGNet, LeNet5 and other neural networks for the same experimental verification, but their performance is not as good as the ResNet neural network. After introducing the attention module, the present invention has both good recognition speed and accuracy. Figure 4 The result accuracy of the method of the present invention is shown, wherein the darker the color, the higher the accuracy.

[0062] This embodiment also provides an in-vehicle gesture recognition system based on a directional speaker, including:

[0063] A directional speaker, used to send a transmission signal to a position in the vehicle where gesture recognition is required, wherein the transmission signal is a frequency modulated continuous wave signal with a frequency range of 70KHz-80KHz and a chirp of 0.1s;

[0064] A microphone is used to receive a reflected signal of a transmitted signal after being reflected by a gesture;

[0065] A signal processing module is used to extract an audio segment containing a gesture based on a transmitted signal and a reflected signal to obtain a gesture waveform, preprocess the gesture waveform, and perform STFT transformation on the preprocessed gesture waveform to convert it into a frequency spectrum;

[0066] The gesture recognition module uses the uniformly cropped frequency spectrum as input and performs gesture recognition using a gesture recognition model built based on a ResNet network.

[0067] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0068] In summary, a large number of experiments have verified the feasibility of the present invention in extracting user gestures through sound waves in a vehicle environment. The present invention proposes an acoustic perception system using an array composed of highly directional transducers. Compared with traditional acoustic systems, the present invention can achieve directional gesture detection without interfering with the audible frequency band. In addition, the present invention also proposes a gesture recognition method, which can filter out the main sound path and human body reflection from the received waveform, and then extract specific gesture information through a deep learning algorithm based on ResNet. Since gesture recognition requires a faster image processing speed, the present invention introduces an attention mechanism into ResNet. Through a large number of experimental tests, it was concluded that the detection accuracy of the system was 91.7%, which proved the feasibility and applicability of the present invention in realizing gesture interaction in a vehicle environment.

[0069] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for in-vehicle gesture recognition based on a directional speaker, characterized in that: The following steps are involved: The gesture data is collected by using a directional speaker, and the audio clip containing the gesture is extracted to obtain the gesture waveform; Preprocess the gesture waveform; Perform STFT transformation on the preprocessed gesture waveform and convert it into a spectrum graph, and evenly crop the spectrum graph as the input of the gesture recognition model; Gesture recognition is performed using a gesture recognition model built based on the ResNet network.

2. The in-vehicle gesture recognition method based on a directional speaker according to claim 1, characterized in that: The method for extracting the audio segment containing the gesture is: Mixing a transmission signal emitted by the directional loudspeaker and a reflection signal received by the microphone after the transmission signal is reflected by the gesture; The mixed signal is processed by low-pass filtering and fast Fourier transform in sequence to obtain spectrum data; Peak detection is performed on the spectrum data. When the third peak is detected and the peak value of the third peak exceeds a preset threshold, a time window is recorded. The time window starts at the moment when the third peak is detected and ends at the moment when the third peak drops below the preset threshold. The mixed signal corresponding to the time window is used as the gesture waveform.

3. The in-vehicle gesture recognition method based on a directional speaker according to claim 1, characterized in that: The preprocessing specifically includes: taking into account the noise existing in the current, using spectral subtraction to perform noise reduction processing on the gesture audio segment.

4. The in-vehicle gesture recognition method based on a directional speaker according to claim 1, characterized in that: The uniformly cropped spectrogram is input into the gesture recognition model. The spectrogram is preprocessed into a 3×224×224 input, and then processed through the convolution layer, channel attention layer, spatial attention layer and maximum pooling layer to obtain the first feature map with a dimension of 64×64×56. The first feature map is input into the residual module of ResNet18, and after residual operations including convolution, regularization, and identity mapping, a 512×7×7 second feature map is obtained; The second feature map is further processed through channel attention, spatial attention and average pooling layers to obtain 512 1×1 third feature maps; The third feature map is used as the input of the fully connected layer, and the gesture classification result is obtained and output after being processed by three layers of fully connected layers.

5. The in-vehicle gesture recognition method based on a directional speaker according to claim 4, characterized in that: The kernel size of the convolutional layer and the pooling layer is 3×3, the padding is 1, and the stride is 2.

6. An in-vehicle gesture recognition system based on a directional speaker, characterized in that: include: A directional speaker, used to send a transmission signal to a position in the vehicle where gesture recognition is required, wherein the transmission signal is a frequency modulated continuous wave signal with a frequency range of 70KHz-80KHz and a chirp of 0.1s; A microphone is used to receive a reflected signal of a transmitted signal after being reflected by a gesture; A signal processing module is used to extract an audio segment containing a gesture based on a transmitted signal and a reflected signal to obtain a gesture waveform, preprocess the gesture waveform, and perform STFT transformation on the preprocessed gesture waveform to convert it into a frequency spectrum; The gesture recognition module uses the uniformly cropped frequency spectrum as input and performs gesture recognition using a gesture recognition model built based on a ResNet network.

7. The in-vehicle gesture recognition system based on a directional speaker according to claim 6, characterized in that: The method for extracting the audio segment containing the gesture is: Mixing a transmission signal emitted by the directional loudspeaker and a reflection signal received by the microphone after the transmission signal is reflected by the gesture; The mixed signal is processed by low-pass filtering and fast Fourier transform in sequence to obtain spectrum data; Peak detection is performed on the spectrum data. When the third peak is detected and the peak value of the third peak exceeds a preset threshold, a time window is recorded. The time window starts at the moment when the third peak is detected and ends at the moment when the third peak drops below the preset threshold. The mixed signal corresponding to the time window is used as the gesture waveform.

8. The in-vehicle gesture recognition system based on directional speakers according to claim 6, characterized in that: The preprocessing specifically includes: taking into account the noise existing in the current, using spectral subtraction to perform noise reduction processing on the gesture audio segment.

9. The in-vehicle gesture recognition system based on directional speakers according to claim 6, characterized in that: The uniformly cropped spectrogram is input into the gesture recognition model. The spectrogram is preprocessed into a 3×224×224 input, and then processed through the convolution layer, channel attention layer, spatial attention layer and maximum pooling layer to obtain the first feature map with a dimension of 64×64×56. The first feature map is input into the residual module of ResNet18, and after residual operations including convolution, regularization, and identity mapping, a 512×7×7 second feature map is obtained; The second feature map is further processed through channel attention, spatial attention and average pooling layers to obtain 512 1×1 third feature maps; The third feature map is used as the input of the fully connected layer, and the gesture classification result is obtained and output after being processed by three layers of fully connected layers.

10. The in-vehicle gesture recognition system based on directional speakers according to claim 9, characterized in that: The kernel size of the convolutional layer and the pooling layer is 3×3, the padding is 1, and the stride is 2.

Images