Multi-person gesture recognition method and system based on acoustic signals

Abstract

The invention discloses a multi-person gesture recognition method and system based on acoustic signals. The method comprises the following steps: determining a target sequence length of a target transmission signal and a first sequence length of an original signal; acquiring an original signal, and converting the original signal into frequency domain data with a first sequence length by using fast Fourier transform; performing zero-symmetry filling outside the frequency domain data so as to enable the sequence length of the filled data to be the target sequence length; converting the filled data into a target transmission signal by using inverse fast Fourier transform, and then sending the target transmission signal; receiving a reflection signal reflected by the gesture action, and extracting gesture features in the reflection signal; and based on the gesture features, determining gesture actions corresponding to the gesture features by using a gesture recognition model. According to the scheme provided by the invention, gestures of a plurality of users can be sensed and recognized in a large range at the same time, and any hardware modification and upgrading on the existing loudspeaker and microphone are not needed.

Description

Technical Field

[0001] The present invention relates to the field of information science and technology, and in particular to a method and system for multi-person gesture recognition based on acoustic signals. Background Art

[0002] As an emerging human-computer interaction method, gesture recognition is characterized by contactless interaction, naturalness and convenience. Therefore, it has a large number of application demands and broad application prospects in the fields of smart home, virtual reality technology, smart medical care, smart driving, etc.

[0003] Among the existing non-contact gesture recognition technologies, gesture recognition technology based on computer vision is more sensitive to lighting and environmental conditions. Different light and background may greatly affect the performance of the system. The presence of cameras may also bring the risk of user privacy leakage. Although gesture recognition technology based on radio frequency signals (such as Wi-Fi signals) is not affected by light and environment, the wavelengths of Wi-Fi signals in the 5GHz and 2.4GHz bands are approximately 6 cm and 12.5 cm respectively. The signal wavelengths are relatively long, so it is difficult to perceive complex finger movements in a fine-grained manner. Gesture recognition technology based on millimeter wave radar, although the distance, angle and other information provided by millimeter wave radar technology has certain advantages in distinguishing gestures of multiple people, the millimeter wave radar signal is easy to attenuate and the perception distance is limited. Therefore, gesture recognition technology based on acoustic signals can achieve fine-grained perception, low hardware cost and rich application scenarios (no requirements for light conditions), and has multiple advantages. However, most existing multi-person gesture recognition solutions perceive gestures at a distance of about 1 meter. In the application scenarios of multi-person gestures, multiple users usually require a larger perception space to achieve a better human-computer interaction experience. For example, in large-scale AR / VR games, multiple players need to constantly move their positions and use gestures to control game characters and trigger special effects. Summary of the invention

[0004] In order to solve the technical problem of the limited sensing range of the existing multi-person gesture recognition solutions, the embodiments of the present invention provide a multi-person gesture recognition method and system based on acoustic signals.

[0005] The technical solution of the embodiment of the present invention is achieved as follows: An embodiment of the present invention provides a multi-person gesture recognition method based on acoustic signals, the method comprising: determining a target transmission distance of a target transmission signal, and determining a target sequence length of the target transmission signal according to the target transmission distance; determining a target bandwidth of the target transmission signal, and determining a first sequence length of an original signal according to the target bandwidth; acquiring an original signal, and converting the original signal into frequency domain data of a first sequence length by using a fast Fourier transform; symmetrically padding the frequency domain data with zeros so that the sequence length of the padded data is the target sequence length; converting the padded data into a target transmission signal by using an inverse fast Fourier transform and then sending the data; receiving a reflected signal of the target transmission signal after being reflected by a gesture action, and extracting gesture features from the reflected signal; based on the gesture features, determining the gesture action corresponding to the gesture features by using a gesture recognition model; the gesture recognition model is a deep neural network model.

[0006] In one embodiment, determining the target sequence length of the target transmission signal according to the target transmission distance includes: calculating the target sequence length according to the target transmission distance using the following calculation formula: Calculation formula (1) in, d is the target transmission distance; L is the target sequence length, c is the propagation speed of sound signal, is the signal sampling rate.

[0007] In one embodiment, determining the first sequence length of the original signal according to the target bandwidth includes: determining the first sequence length of the original signal according to the target bandwidth using the following calculation formula: Calculation formula (2) in, B is the target bandwidth, is the signal sampling rate, is the length of the first sequence, is the target sequence length.

[0008] In one embodiment, receiving a reflected signal of the target transmission signal after being reflected by a gesture action includes: determining a minimum resolution for beam scanning based on a target transmission distance of the target transmission signal; dividing a receiving area of ​​the reflected signal into a plurality of first areas, performing a first beam scan on each of the first areas, and determining a first area where the reflected signal after being reflected by the gesture action is located; for the first areas, performing a second beam scan at the minimum resolution, and determining a second area where the reflected signal after being reflected by the gesture action is located; and collecting signals in the second areas, and using the collected signals in the second areas as reflected signals.

[0009] In one embodiment, based on the target transmission distance of the target transmission signal, the minimum resolution for beam scanning is determined, including: establishing a first model based on the target transmission distance of the target transmission signal; the first model is fan-shaped; the angle corresponding to the arc length of the fan is the minimum resolution of the beam scanning.

[0010] In one embodiment, a first model is established based on the target transmission distance of the target transmission signal, including: taking the receiving point of the reflected signal as the center point of the sector, taking the target transmission distance as the side length of the sector, and taking the range of motion of the gesture action as the straight-line distance between the farthest two points on the arc of the sector.

[0011] In one embodiment, extracting gesture features from the reflection signal includes: calculating a channel impulse response based on the reflection signal; removing interference components from the channel impulse response to obtain an effective channel impulse response; displaying the effective channel impulse response in the form of a heat map; and performing feature extraction on the displayed heat map image to obtain gesture features.

[0012] In one embodiment, calculating the channel impulse response based on the reflected signal includes: calculating the channel impulse response based on the reflected signal using the following calculation formula: Calculation formula (3) in, is the channel impulse response, Transmitting signals to the target, is the reflected signal received at the receiver The conjugate of is the multipath delay, yes The target sequence length.

[0013] In one embodiment, removing interference components in the channel impulse response includes: eliminating dynamic signal interference outside a preset range in the channel impulse response; calculating a difference in the channel impulse responses of two adjacent signal frames; and using the difference to eliminate visible path transmission interference and static object reflection signal interference in the channel impulse response.

[0014] An embodiment of the present invention also provides a multi-person gesture recognition system based on acoustic signals, comprising: a signal sending device, a signal receiving device and a processor; the signal sending device is used to send a target transmission signal; the signal receiving device is used to receive a reflected signal; the processor is used to execute the steps of the above-mentioned method.

[0015] The multi-person gesture recognition method and system based on acoustic signals provided in the embodiment of the present invention determines the target transmission distance of the target transmission signal, and determines the target sequence length of the target transmission signal according to the target transmission distance; determines the target bandwidth of the target transmission signal, and determines the first sequence length of the original signal according to the target bandwidth; obtains the original signal, and converts the original signal into frequency domain data of the first sequence length using fast Fourier transform; symmetrically fills the frequency domain data with zeros so that the sequence length of the filled data is the target sequence length; converts the filled data into the target transmission signal using fast Fourier inverse transform and then sends it; receives the reflected signal of the target transmission signal after the gesture action is reflected, and extracts the gesture features in the reflected signal; based on the gesture features, determines the gesture action corresponding to the gesture features using a gesture recognition model; the gesture recognition model is a deep neural network model. The solution provided by the present invention can perceive and recognize the gestures of multiple users at the same time in a large range, without any hardware modification or upgrade of the existing speakers and microphones. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of a flow chart of a method for multi-person gesture recognition based on acoustic signals according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the system framework of an embodiment of the present invention; Figure 3 This is a schematic diagram of a ZC sequence according to an embodiment of the present invention; Figure 4 A schematic diagram of a frequency domain interpolation solution according to an embodiment of the present invention; Figure 5 Schematic diagram of bandwidth of three passband ZC sequences according to an embodiment of the present invention; Figure 6 Schematic diagram of CIR measurement results of double-click gestures in an embodiment of the present invention; wherein, Figure 6 (a) is a schematic diagram of the gesture; Figure 6(b) is the channel measurement result of two double-click gesture diagrams; Figure 7 This is a schematic diagram of the beam scanning resolution of an embodiment of the present invention; Figure 8 A schematic diagram of a two-stage progressive beam scanning solution according to an embodiment of the present invention; Fig. 9 This is a schematic diagram of setting up a multi-person gesture experiment according to an embodiment of the present invention; Fig.10 This is a schematic diagram of the results of a multi-person gesture experiment according to an embodiment of the present invention; wherein: Fig.10 (a) CIR measurement results of 45° in the first round of experiments; Fig.10 (b) is the CIR measurement result of 90° in the first round of experiments; Fig.10 (c) is the CIR measurement result of 135° in the first round of experiments; Fig.10 (d) is the CIR measurement result of 45° in the second round of experiments; Fig.10 (e) is the CIR measurement result of 90° in the second round of experiments; Fig.10 (f) is the CIR measurement result of 135° in the second round of experiments; Fig.10 (g) is the CIR measurement result of 45° in the third round of experiments; Fig.10 (h) is the CIR measurement result of 90° in the third round of experiments; Fig.10 (i) CIR measurement results of 135° in the third round of experiments; Fig.11 This is a schematic diagram of gesture recognition performance in a multi-person scenario according to an embodiment of the present invention; Fig.12 Schematic diagram of nine gestures that can be recognized by the embodiment of the present invention; among them, Fig.12 (a) Definition of reference coordinate system for 9 gesture directions; Fig.12 (b) Schematic diagram of nine gestures; Fig.13 A schematic diagram of gesture recognition performance at different positions according to an embodiment of the present invention; Fig.14 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0018] The embodiment of the present invention provides a method for multi-person gesture recognition based on acoustic signals, such as Figure 1 As shown, the method includes: Step 101: determining a target transmission distance of a target transmission signal, and determining a target sequence length of the target transmission signal according to the target transmission distance; Step 102: determining a target bandwidth of the target transmission signal, and determining a first sequence length of the original signal according to the target bandwidth; Step 103: Acquire an original signal, and convert the original signal into frequency domain data of a first sequence length by using a fast Fourier transform; Step 104: symmetrically padding the frequency domain data with zeros so that the sequence length of the padded data is the target sequence length; Step 105: convert the padded data into a target transmission signal by using an inverse fast Fourier transform and then send the signal; Step 106: receiving a reflected signal of the target transmission signal after being reflected by the gesture action, and extracting gesture features from the reflected signal; Step 107: Based on the gesture feature, determine the gesture action corresponding to the gesture feature using a gesture recognition model; the gesture recognition model is a deep neural network model.

[0019] The method of this embodiment can sense and recognize the gestures of multiple users at the same time within a larger range (>2 meters) without any hardware modification or upgrade of the existing speakers and microphones.

[0020] This embodiment designs a unique transmission signal frame to achieve long-distance perception, and combines a channel measurement method based on channel impulse response and a two-stage progressive beam scanning scheme to distinguish gestures of multiple users at different distances and angles. This embodiment enriches the application of gesture recognition technology based on acoustic signals, and is expected to achieve real-time interaction of multi-person gestures in a variety of human-computer interaction scenarios.

[0021] First, see Figure 2 , Figure 2 The figure is a schematic diagram of the system framework of an embodiment of the present invention. First, the designed ultrasonic signal (i.e., the target transmission signal) is transmitted by the speaker, and the microphone array receives the reflected signal reflected by the gesture action. Then, a progressive beam scan is performed on the area in front of the transceiver to detect possible multi-person gestures. For the signal in the direction where the gesture is detected, the direction signal is directionally enhanced by beamforming. Then, the channel impulse response is measured and the feature image related to the gesture is obtained. The feature image is used as the input of the gesture classification model to classify and recognize the gesture.

[0022] (1) Ultrasonic transmission signal design This embodiment uses the ZC (Zadoff-Chu) sequence as the baseband transmission signal. The sequence has good autocorrelation. The ZC sequence is a complex sequence, such as Figure 3 As shown, its mathematical expression is: Calculation formula (1) in, is the length of the ZC sequence, , is a constant, a variable u is an integer, and , c is the speed of sound signal propagation, is the signal sampling rate.

[0023] Given the propagation speed of a sound signal c and signal sampling rate In the case of a sampling interval Signal transmission distance for: Calculation formula (2) The highest sampling rate usually supported by common commercial acoustic equipment is =48KHz, the speed of sound propagation in air at 1 standard atmospheric pressure and 15℃ c Approximately 340 , we can calculate =0.007 From the meaning of channel taps (i.e. a sequence), this means that each channel tap covers 7 The difference between any two propagation distances is less than 7 The signals of will be collected in the same tap. Given the number of taps , then the difference in the maximum path length of two sound signals propagating in the environment is for: Calculation formula (3) Number of taps It can be set by determining the length of the transmission signal frame, so by setting different lengths The signal perception distance can be flexibly extended. Considering that the signal propagates back and forth between the transceiver and the reflective object, the signal perception distance is In this embodiment, the length of the ZC sequence is set to 2048, that is, the target sequence length of the baseband transmission signal is 2048, and the sensing range of the signal coverage is 7.25m (that is, the target transmission distance).

[0024] The frequency components above 18KHz are inaudible to most people, while the highest frequency of the transmission signal supported by most commercial acoustic equipment is 24KHz. The above two practical factors limit the system's transmission signal to the frequency band of 18KHz-24KHz, which means that the maximum bandwidth of the baseband ZC sequence is 6KHz. However, the bandwidth of the baseband ZC sequence occupies the entire frequency band and cannot meet the system's requirement that the transmission signal bandwidth is at most 6KHz. In order to compress the transmission signal bandwidth to meet the requirement of the inaudible frequency band for the human ear, this embodiment designs a frequency domain interpolation scheme to design a baseband transmission signal that meets the requirements. Figure 4 As shown in the figure, in frequency domain interpolation, the original ZC sequence is first converted to the frequency domain using Fast Fourier Transform (FFT), then multiple zeros are symmetrically filled outside its original frequency components for interpolation, and finally the interpolated signal is converted back to the time domain using Inverse Fast Fourier Transform (IFFT). The bandwidth B of the interpolated signal is: Calculation formula (4) in, is the required signal bandwidth, is the signal sampling rate, set to 48KHz, and are the original ZC sequence length and the interpolated sequence length, It is set to 2048 to ensure that the signal can be transmitted over a distance of at least 7 meters, so the bandwidth B is determined by the original sequence length. In order to select the appropriate original sequence length (i.e., the first sequence length), the parameters of the ZC sequence are set to They are (32, 63), (64, 127) and (128, 255) respectively. The frequency domain interpolation scheme is used to interpolate the sequence length to 2048, and they are up-converted into passband signals with a carrier frequency of 20.25KHz. Figure 5 As shown, the original ZC (32, 63), ZC (64, 127) and ZC (128, 255) occupy bandwidths of 1.5KHz, 3KHz and 6KHz respectively. On the one hand, the rich multipath effects within the long-distance perception range will aggravate the frequency selective fading of the signal. If the transmitted signal only covers a 1.5KHz bandwidth, it will lead to insufficient channel measurement and fail to provide rich gesture characteristic information. On the other hand, although the 6KHz bandwidth helps to capture sufficient channel measurements and theoretically meets the frequency range that is inaudible to the human ear, the filters applied in the signal up-conversion and down-conversion processing have imperfect frequency responses, so the noise generated by the filters will inevitably leak into the nearby audible frequency bands. After comprehensive consideration, the final choice is = 64 and = 127 as the parameter of the original ZC sequence, which is then interpolated into a sequence of length 2048 to cover the 3KHz band from 18.75KHz to 21.75KHz.

[0025] (2) Channel measurement and multi-person gesture detection The Channel Impulse Response (CIR) can be used to describe the impact of multipath in the channel on the transmitted signal, and divide the multipath signal into different taps according to the propagation delay of the multipath signal. Each propagation delay corresponds to a different propagation distance range, which can accurately locate the signal affected by a specific target object. By identifying the number of taps affected by a specific target, the fluctuations in delay and attenuation of the signal can be studied, thereby effectively reflecting the specific movement of the target. This embodiment uses a CIR-based method to perform channel measurement to characterize the channel affected by gestures. For a ZC sequence (complex signal), the CIR can be measured by calculating the cyclic cross-correlation between the transmitted baseband signal and the received baseband signal. The formula is as follows: Calculation formula (5) in, is the measured CIR, represents the transmitted complex baseband signal, Indicates that the receiver receives a signal The conjugate of is the multipath delay, yes Length. The CIR directly measured by this formula includes line of sight (LOS) transmission, reflection from static objects in the environment, and reflection from other dynamic objects that are not related to gestures. The present invention extracts effective CIR measurement results in two steps: (1) First, by reasonably setting the number of channel taps, the influence of reflected signals from objects outside a specific distance range is filtered out. This step can ensure reliable CIR measurement within a specific range without being interfered with by reflected signals outside the specific range (i.e., from the channel impulse response, dynamic signal interference outside the preset range in the channel impulse response is eliminated). (2) The difference in CIR between the two adjacent signal frames received is calculated to eliminate the signal transmitted by the line of sight and reflected by static objects (i.e., the difference in the channel impulse response of two adjacent signal frames is calculated; the difference is used to eliminate the interference of the line of sight transmission and the interference of the reflected signal from static objects in the channel impulse response). Next, the effective CIR measurement is visualized to intuitively display the time domain characteristics of the CIR in the form of a heatmap. As shown in FIG. Figure 6 As shown, Figure 6 (a) is a schematic diagram of the gesture. Figure 6(b) is the channel measurement result of the double-click gesture, i.e., the CIR heat map. The horizontal axis of the heat map represents time in seconds (s), and the vertical axis represents the distance between the gesture and the system in meters (m). Figure 6 In (b), an amplitude point group appears at 3 meters on the vertical axis. This is the characteristic pattern of the double-click gesture, indicating that the CIR-based channel measurement method has effectively extracted the gesture features.

[0026] When multiple users perform gestures at the same time, although all reflected signals are aliased at the receiving end, the position of each user is different, and the arrival angle of the reflected signal corresponding to the gesture to the microphone at the receiving end is (angle of arrival, AoA) is different, so the present invention is based on the angle of arrival The traditional beamforming technology performs omnidirectional scanning of the entire space ( ) to determine the number and location of users, which leads to high algorithm complexity and long scanning time, and cannot meet the real-time requirements of gesture recognition. In addition, in the gesture recognition system, the human body will occupy a certain width. At the same time, in the multi-person gesture interaction scenario, users will maintain an appropriate spatial interval to achieve a better human-computer interaction experience. Therefore, the traditional beam scanning scheme is not suitable for this system. CIR can separate the received multipath signals based on the signal delay size, providing a basis for detecting gestures performed at different distances. In order to further distinguish gestures at different angles, this embodiment is inspired by the binary search algorithm and designs a two-stage progressive beam scanning scheme.

[0027] First, it is necessary to determine the appropriate beam scanning resolution. The core idea is to balance the scanning resolution and the accuracy of azimuth estimation, that is, to minimize the algorithm complexity and scanning time while ensuring that gesture detection can be completed. This embodiment uses the geometric relationship between the human body and the acoustic device during the gesture execution process to determine the minimum resolution of the beam scanning. Since the beam scanning resolution is lower with increasing distance, the beam scanning angle at the system's farthest sensing distance of 7 meters is used as the standard for calculation to ensure that the minimum beam scanning angle calculated in this way is still applicable at other distances. Figure 7 As shown in the figure, the geometric relationship between the user 7 meters away from the system and the system can be approximated as a fan-shaped area (i.e., the first model is established). In the actual long-distance gesture perception scenario, multiple users will maintain a certain distance, so the distance on both sides of the user's shoulders is extended by 1 meter (that is, the user's shoulder width and the space requirement for multi-person gesture interaction are determined together as the range of the gesture action), and the fan-shaped area angle formed after the extended distance is used as the As the scanning resolution. In order to reduce individual differences, the sector angle was obtained after multiple volunteers of different genders, heights and body shapes participated in the angle measurement. The distribution is in the range of 18° to 22°. In order to maintain generality, the present invention finally sets the minimum resolution of beam scanning to 20°.

[0028] In order to improve the efficiency of beam scanning, this embodiment applies the theoretical ideas of the classic sorting algorithm to the spatial scanning technology to reduce the algorithm complexity and scanning time, thereby meeting the real-time requirements of gesture detection. Figure 8 As shown, using the idea of ​​recursive binary search, the 180° scanning space in front of the system is first divided into three 60° sector areas (i.e., three first areas), and a scan is performed in each sector area to determine the approximate area where the gesture is located. Each 60° sector is then further divided into three 20° (i.e., lowest resolution) sub-sectors, and a beam scan is performed on each sub-sector to accurately locate the exact direction associated with the gesture. This scheme divides the beam scanning into two stages. The scanning area of ​​the second stage depends on the scanning results of the first stage, that is, the beam scanning of the sub-sector in stage two is only performed within the parent sector where the gesture has been detected in stage one. Based on this two-stage progressive beam scanning scheme, it takes at most 6 scans to determine the direction of a gesture. This scheme reduces the scanning complexity for detecting gestures from . Reduce to , which greatly reduces the algorithm complexity and shortens the scanning time. In this scheme, the user can be located at a random position within the 180° area in front of the system without being restricted to a specified sector.

[0029] This example verifies the effectiveness of the two-stage progressive beam scanning scheme in detecting multi-person gestures. The experimental settings of this experiment are as follows: Fig. 9 As shown in the figure, users A and B are both standing 3 meters away from the system, but A is located at 45° at 3 meters and B is located at 135° at 3 meters. The experiment was conducted in three rounds. In the first round of experiments, user A performed two push and pull gestures at 45°, and user B remained still. In the second round of experiments, user B performed two double-click gestures at 135°, and user A remained still. In the third round of experiments, user A performed push and pull gestures, while user B performed a double-click gesture. In each round of experiments, beamforming was performed at 45°, 90°, and 135°, and CIR measurements were performed to observe gesture characteristics. The experimental results are shown as follows: Fig.10 CIR heat map in Fig.10 (a)- Fig.10 (c) Fig.10 (d)- Fig.10 (f) Fig.10 (g) - Fig.10(i) CIR measurement results for the first, second, and third rounds of experiments. The experimental results can be summarized into the following two key conclusions: (1) When the beam scan matches the direction of the gesture, the gesture effect presented in the CIR is significantly greater than that in other areas, i.e. Fig.10 (a) Fig.10 (f) Fig.10 (g) and Fig.10 The CIR measurement value of (i) is significantly larger than that of the others. In addition, different gestures show distinguishable characteristics in CIR measurement, namely, Fig.10 (a) and Fig.10 (g) is a push forward, pull back gesture, and Fig.10 (f) and Fig.10 (i) is a double-click gesture; (2) When two users perform gestures at the same time, different CIR measurements will be generated after beamforming in the directions of different users, such as Fig.10 (g) and Fig.10 (i) shown. Fig.10 The other bright paths not mentioned in the CIR measurement are multipath reflections from the long-distance space. The above experimental results fully prove that the two-stage progressive beam scanning and beamforming scheme designed by the present invention can effectively distinguish multiple gestures occurring simultaneously at the same distance.

[0030] (3) Gesture recognition In this embodiment, the gesture detection result is a CIR heat map image, and the gesture features are included in the CIR amplitude point group. The gesture recognition in this embodiment is a multi-classification task, and it is oriented to the human-computer interaction needs of smart devices, which requires the gesture classification model to have the characteristics of high recognition accuracy, real-time response and low computational overhead. MobileNetV2 is a lightweight deep neural network proposed by Google in 2018. It is designed for mobile devices and embedded systems with insufficient computing resources. It provides efficient computing performance on a small model and performs visual recognition tasks. Therefore, this embodiment selects MobileNetV2 as the gesture classification model of the system. The gesture classification model is trained and tested based on the PyTorch framework, all of which are completed on a computer with 32GB of memory, a 12th-generation Intel Core i7-2700H CPU model and an NVIDIA GeForce RTX 3060 GPU model. To ensure that the model can obtain diverse samples during training, 70% of all data are randomly selected for training, and the remaining 30% are used for testing.

[0031] (4) System evaluation In order to evaluate the system's ability to recognize simultaneous gestures of multiple people, this embodiment conducts a test experiment of multi-person gestures. In the experiment, three volunteers were located at different distances and angles relative to the system. The experiment was conducted in two rounds. The first round tested a two-person scene. Two volunteers randomly performed three different gestures, and each gesture was repeated 100 times. At this time, the third volunteer did not make a gesture; the second round tested a three-person scene. Three volunteers randomly performed three different gestures at the same time, and each gesture was repeated 100 times. Under the same experimental environment, this embodiment also tested the single-person gesture recognition performance for comparison. The experimental test results are shown in the figure. Fig.11 As shown in the figure, in the scenario where two or three people make gestures at the same time, the system's gesture recognition accuracy (Presicion) is 98.4% and 95.4% respectively, and the recall rate (Recall) and F1 score (F1-score) are both over 95.0%, and the three indicators are only slightly lower than the single-person scenario. The experimental results show that the system can still effectively detect and distinguish each gesture in the multi-user scenario in the multi-user scenario, and has good recognition and classification performance.

[0032] In order to verify the robustness of the system, that is, the recognition of gestures from different positions, this embodiment tests the average recognition accuracy of the system for 9 gestures performed by the user at different distances and angles. The 9 gestures are: push forward, pull back, double click, slide up, slide down, single click, slide left, slide right, and wave. Fig.12 (b). Fig.12 (a) is the reference coordinate system definition for the nine gesture directions. During the experiment, the volunteers stood Fig.13 15 positions (indicated by solid circles) were used to perform 9 different gestures at each position, and each gesture was repeated 100 times. The test results are shown in Figure 1. Fig.13 As shown in the figure, the gesture recognition accuracy of the system for the 15 positions of the volunteers was higher than 94.0%. At the same time, the results also show that compared with the average gesture recognition accuracy of each position at 1 meter from the system, the average gesture recognition accuracy of each position at 5 meters only decreased by about 0.7%, which proves that the system has strong robustness and can accurately recognize gestures made by users at different positions. In addition, although the gesture recognition accuracy will decrease as the distance between the volunteer's position and the system placement increases, its average recognition accuracy at 7 meters is still maintained at 95.4%, which reflects the system's powerful long-distance recognition ability.

[0033] This embodiment proposes the selection and design of key parameters of the signal frame (ZC sequence), which increases the signal perception distance and satisfies practical factors such as the signal should be within the bandwidth range supported by the system and the frequency band that is inaudible to human ears. Based on the idea of ​​binary search algorithm, a two-stage progressive beam scanning scheme is designed to detect multi-person gestures, reducing the complexity of gesture scanning from Reduce to , greatly reducing the number and time of beam scanning. Based on existing commercial speakers and microphone arrays, a multi-person gesture recognition system with a larger sensing range (>2 meters) was designed and implemented.

[0034] This embodiment can realize real-time recognition of multi-person gestures within a larger range (>2 meters) based on acoustic signals while using existing commercial speakers and microphones, and can develop more innovative applications in the fields of entertainment games, smart homes, medical health, virtual reality and augmented reality.

[0035] After testing, it is proven to be feasible. This embodiment evaluates the system performance when different users (one, two, and three) make gestures at the same time. In the scenarios where two and three people make gestures at the same time, the system's gesture recognition accuracy (Presicion) is 98.4% and 95.4% respectively, proving that this system can recognize gestures performed by multiple users at the same time. In addition, this embodiment also evaluates the gesture recognition performance at different positions. The experimental results confirm that the system has a gesture recognition accuracy of more than 94.0% at different distances and angles, and has good gesture recognition performance.

[0036] This embodiment can be used for contactless control (human-computer interaction) of various smart devices, including but not limited to gaming and entertainment, smart home, medical health, smart driving, virtual reality and augmented reality.

[0037] The multi-person gesture recognition method based on acoustic signals provided in the embodiment of the present invention determines the target transmission distance of the target transmission signal, and determines the target sequence length of the target transmission signal according to the target transmission distance; determines the target bandwidth of the target transmission signal, and determines the first sequence length of the original signal according to the target bandwidth; obtains the original signal, and converts the original signal into frequency domain data of the first sequence length using fast Fourier transform; symmetrically fills the frequency domain data with zeros so that the sequence length of the filled data is the target sequence length; converts the filled data into the target transmission signal using fast Fourier inverse transform and then sends it; receives the reflected signal of the target transmission signal after the gesture action is reflected, and extracts the gesture features in the reflected signal; based on the gesture features, determines the gesture action corresponding to the gesture features using a gesture recognition model; the gesture recognition model is a deep neural network model. The solution provided by the present invention can perceive and recognize the gestures of multiple users at the same time in a large range, without any hardware modification or upgrade of the existing speakers and microphones.

[0038] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a multi-person gesture recognition system based on acoustic signals, including: a signal sending device, a signal receiving device and a processor; the signal sending device is used to send a target transmission signal; the signal receiving device is used to receive a reflected signal; the processor is used to execute the steps of the above method.

[0039] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0040] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0041] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Fig.14As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0042] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0043] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0044] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0045] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0046] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0047] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0048] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0049] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0050] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0051] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0052] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0053] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for multi-person gesture recognition based on acoustic signals, characterized in that: The method comprises: Determining a target transmission distance of a target transmission signal, and determining a target sequence length of the target transmission signal according to the target transmission distance; Determining a target bandwidth of the target transmission signal, and determining a first sequence length of the original signal according to the target bandwidth; Acquire an original signal, and convert the original signal into frequency domain data of a first sequence length using a fast Fourier transform; symmetrically padding the frequency domain data with zeros so that the sequence length of the padded data is the target sequence length; The padded data is converted into a target transmission signal by using an inverse fast Fourier transform and then sent; Receiving a reflected signal of the target transmission signal after being reflected by a gesture action, and extracting gesture features from the reflected signal; Based on the gesture feature, a gesture recognition model is used to determine the gesture action corresponding to the gesture feature; the gesture recognition model is a deep neural network model.

2. The method for multi-person gesture recognition based on acoustic signals according to claim 1, characterized in that: The determining the target sequence length of the target transmission signal according to the target transmission distance includes: The target sequence length is calculated using the following formula according to the target transmission distance: Calculation formula (1) in, d is the target transmission distance; L is the target sequence length, c is the propagation speed of sound signal, is the signal sampling rate.

3. The method for multi-person gesture recognition based on acoustic signals according to claim 1, characterized in that: The determining the first sequence length of the original signal according to the target bandwidth comprises: The first sequence length of the original signal is determined according to the target bandwidth using the following calculation formula: Calculation formula (2) in, B is the target bandwidth, is the signal sampling rate, is the length of the first sequence, is the target sequence length.

4. The method for multi-person gesture recognition based on acoustic signals according to claim 1, characterized in that: Receiving a reflected signal of the target transmission signal after the target transmission signal is reflected by the gesture action, comprising: Determining a minimum resolution for performing beam scanning based on a target transmission distance of the target transmission signal; Divide the receiving area of ​​the reflected signal into a plurality of first areas, perform a first beam scan on each of the first areas, and determine the first area where the reflected signal after the gesture action is reflected is located; For the first area, performing a second beam scan at the lowest resolution to determine a second area where the reflected signal after being reflected by the gesture action is located; The signal in the second area is collected, and the collected signal in the second area is used as a reflection signal.

5. The method for multi-person gesture recognition based on acoustic signals according to claim 4, characterized in that: Determining a minimum resolution for performing beam scanning based on a target transmission distance of the target transmission signal includes: A first model is established based on the target transmission distance of the target transmission signal; the first model is fan-shaped; the angle corresponding to the arc length of the fan-shaped is the minimum resolution of the beam scanning.

6. The method for multi-person gesture recognition based on acoustic signals according to claim 5, characterized in that: Establishing a first model based on the target transmission distance of the target transmission signal includes: The receiving point of the reflected signal is taken as the center point of the sector, the target transmission distance is taken as the side length of the sector, and the range of the gesture action is taken as the straight-line distance between the farthest two points on the arc of the sector.

7. The method for multi-person gesture recognition based on acoustic signals according to claim 1, characterized in that: Extracting gesture features from the reflection signal includes: Calculating a channel impulse response based on the reflected signal; removing interference components in the channel impulse response to obtain an effective channel impulse response; Displaying the effective channel impulse response in a heat map; Perform feature extraction on the displayed heat map image to obtain gesture features.

8. The method for multi-person gesture recognition based on acoustic signals according to claim 7, characterized in that: Calculating a channel impulse response based on the reflected signal includes: Based on the reflected signal, the channel impulse response is calculated using the following formula: Calculation formula (3) in, is the channel impulse response, Transmitting signals to the target, is the reflected signal received at the receiver The conjugate of is the multipath delay, yes The target sequence length.

9. The method for multi-person gesture recognition based on acoustic signals according to claim 7, characterized in that: Removing interference components in the channel impulse response, including: Eliminating dynamic signal interference outside a preset range in the channel impulse response from the channel impulse response; The difference of the channel impulse responses of two adjacent signal frames is calculated; and the visible path transmission interference and static object reflection signal interference in the channel impulse response are eliminated by using the difference.

10. A multi-person gesture recognition system based on acoustic signals, characterized in that: include: A signal sending device, a signal receiving device and a processor; The signal sending device is used to send a target transmission signal; The signal receiving device is used to receive the reflected signal; The processor is used to execute the steps of the method according to any one of claims 1 to 9.

Images