Virtual data generation and classification method for few-sample ultrasonic gesture recognition

Abstract

The invention discloses a virtual data generation and classification method for few-sample ultrasonic gesture recognition. According to the method, ultrasonic channel perception and the conditional diffusion generation model are deeply fused, stable and reliable gesture recognition with good generalization ability is realized on the premise that large-scale real user data acquisition is not needed, the problem that an existing ultrasonic gesture recognition technology depends on a large number of user samples is effectively relieved, and the user experience is improved. The robustness and adaptability of the model under complex practical application conditions such as cross-user, cross-gesture speed and cross-interaction distance are remarkably improved, the practicability and expandability of the ultrasonic interaction technology of the mobile equipment end are improved, and the method has high engineering application value and industrial popularization potential. According to the method, the data acquisition time and the calculation overhead are remarkably reduced, the user experience is improved, rapid adaptation to new users, new gestures and new use scenes is realized, and powerful support is provided for popularization and application of an ultrasonic gesture recognition technology in large-scale mobile terminals.

Description

Technical Field

[0001] This invention belongs to the field of user authentication, specifically relating to a reliable few-sample gesture recognition method based on ultrasound signals that uses the speaker and microphone of a device to collect ultrasonic signals and generates virtual samples using a conditional diffusion model. Background Technology

[0002] Gesture interaction technology has been widely used in smartphones, wearable devices, smart homes, and other scenarios in recent years, and can be used for various interaction methods such as control interfaces, assisted input, and touchless operation. Existing gesture recognition methods can be mainly divided into visual signal-based gesture recognition and wireless signal-based gesture recognition, both of which have their own limitations.

[0003] Vision-based gesture recognition technology typically relies on cameras to capture images or videos of users' actions, and then uses visual models such as convolutional neural networks or Transformers to classify gestures. While this method is convenient to use, it has significant drawbacks: camera capture is greatly affected by lighting and environmental occlusion, making it difficult to work stably in dark or complex environments; furthermore, visual data can easily leak user privacy, making it unsuitable for privacy-sensitive scenarios or scenarios requiring low power consumption.

[0004] Gesture recognition technology based on wireless signals uses WiFi, millimeter waves, radar, or ultrasonic signals to sense hand gestures and classifies them by analyzing changes in signal reflection. Wireless sensing methods have advantages such as being independent of lighting conditions and offering high privacy and security. However, the high hardware cost, high power consumption, or difficulty in deploying many wireless technologies on mobile devices limit their widespread application.

[0005] To achieve low-power, mobile-deployable gesture recognition capabilities with high privacy protection, researchers have begun to focus on leveraging existing ultrasonic hardware in smartphones or headphones for gesture perception. Existing ultrasonic gesture recognition methods typically require extensive user data for training to maintain good performance under varying user conditions, speeds, and distances. However, in practical applications, acquiring large numbers of user samples is both time-consuming and impractical, making it difficult for recognition models to quickly adapt to new users and resulting in weak generalization ability. Furthermore, ultrasonic multipath reflections generate unstable channel fluctuations with changes in gesture speed and hand position, limiting recognition accuracy under limited sample conditions.

[0006] With the continuous development of mobile device hardware, existing smartphones and headphones are now widely equipped with high-quality speakers and microphones, capable of stably transmitting and receiving high-frequency ultrasonic signals. Meanwhile, recent advancements in deep generative modeling techniques (such as diffusion models) have enabled the generation of virtual training data that conforms to physical laws even with low sample sizes, providing a new approach to enhancing the generalization ability of gesture recognition models. Therefore, it is possible to leverage the existing ultrasonic hardware of mobile devices in conjunction with deep generative models to automatically expand user training samples, achieving a gesture recognition technology that requires minimal user data yet achieves high-precision recognition results.

[0007] This invention utilizes a device's speaker to emit a constant ultrasonic signal and a microphone to collect ultrasonic multipath variations caused by gestures, generating a Differential Channel Impulse Response (DiffCIR) as a stable dynamic feature of the gesture. Based on this, a conditional vector is constructed using velocity and distance encoding, and a conditional diffusion model is used to generate virtual samples consistent with real physical characteristics, thereby improving gesture recognition performance under few-sample conditions. This invention proposes a reliable few-sample gesture recognition method based on ultrasonic signals, enabling natural, low-power, and privacy-friendly gesture interaction on mobile and wearable devices. Summary of the Invention

[0008] The beneficial effects of this invention are as follows:

[0009] This invention proposes a complete ultrasonic gesture recognition solution for scenarios with few samples at the system level. By deeply integrating ultrasonic channel sensing with a conditional diffusion generation model, it achieves stable, reliable gesture recognition with good generalization ability without requiring large-scale real user data collection. This solution effectively alleviates the dependence of existing ultrasonic gesture recognition technologies on a large number of user samples, significantly improves the robustness and adaptability of the model under complex practical application conditions such as cross-user, cross-gesture speed, and cross-interaction distance, reduces the cost of system deployment and user customization, and enhances the practicality and scalability of ultrasonic interaction technology on mobile devices. It has high engineering application value and industrial promotion potential.

[0010] This invention takes the ultrasonic multipath changes caused by user gestures as the modeling object. It constructs differential channel impulse response features through low-pass filtering, range truncation, and response calculation. Compared with directly using the original channel features, it effectively suppresses static environment and equipment noise interference, and highlights the spatiotemporal characteristics of the dynamic behavior of gestures themselves, thereby improving the discriminativeness and stability of feature expression and providing high-quality input for subsequent virtual data generation and classification recognition.

[0011] This invention proposes a multi-dimensional conditional vector construction method based on velocity encoding and distance encoding, explicitly introducing key physical factors such as gesture execution speed and interaction distance into the training and generation process of the diffusion model. This conditional encoding mechanism ensures that the generated virtual samples are highly consistent with real-world scenarios in both statistical distribution and physical meaning, effectively enhancing the model's adaptability to different user operating habits and changes in the usage environment, and significantly improving cross-conditional generalization performance.

[0012] This invention addresses the structural characteristics of ultrasonic channel impulse response by designing and training a dedicated graph-based conditional diffusion model, CIR-ShiftNet. By introducing ordered pairs of differential channel features and multidimensional conditional constraints during the diffusion process, it achieves precise modeling of the channel variation patterns caused by gestures. Compared to traditional data augmentation or generation methods, this model can generate virtual samples with strong continuity and high physical consistency, effectively avoiding sample distortion and distribution shift problems.

[0013] This invention combines a small number of real differential channel impulse response features from users with virtual samples generated by a diffusion model to construct a training set that drives subsequent classification model learning, significantly improving recognition accuracy and stability without increasing the user's data collection burden. Furthermore, this framework is compatible with various classification models such as support vector machines, random forests, convolutional neural networks, and long short-term memory networks, possessing good model versatility and application flexibility, facilitating deployment and expansion based on different device computing power and application requirements.

[0014] This invention requires only a small number of customized gesture samples from users to complete model training and recognition, significantly reducing data collection time and computational overhead, and improving user experience. Simultaneously, through a virtual data generation mechanism, it enables rapid adaptation to new users, new gestures, and new usage scenarios, providing strong support for the widespread application of ultrasonic gesture recognition technology in large-scale mobile terminals. Attached Figure Description

[0015] Figure 1 This is a flowchart of the process of this invention;

[0016] Figure 2 This is a schematic diagram of an ultrasonic signal transmission model;

[0017] Figure 3 This is a structural diagram of the CIR-ShiftNet network;

[0018] Figure 4 This is a diagram illustrating the authentication performance. Detailed Implementation

[0019] This invention addresses the shortcomings of existing ultrasonic gesture recognition technologies, such as insufficient generalization ability under few-sample conditions, weak cross-user adaptability, and sensitivity to environmental changes. It proposes a reliable few-sample gesture recognition method on mobile devices (such as headphones or smartphones) that utilizes ultrasonic signals to capture stable physical dynamic features and combines this with a diffusion model to generate virtual training samples. The method is further described below with reference to the accompanying drawings and embodiments.

[0020] Figure 2 For the theoretical model of ultrasound-based gesture recognition, it is assumed that the transmitter transmits based on a frequency-hopping carrier complex baseband. signal The transmitted passband signal can be represented as:

[0021]

[0022] Without loss of generality, consider a reflection point on the hand, whose position follows the time-varying trajectory of hand movements. By aggregating the reflections contributed by all such points individually, the overall influence of the entire hand can be obtained. Let the initial height of the reflection point be... at a constant speed Move downwards. Its vertical position is:

[0023]

[0024] The horizontal offset of the reflection point from the transceiver baseline is a constant. Therefore, a two-dimensional trajectory can be represented as:

[0025]

[0026] Assume the transmitter and receiver are located at... and The propagation distance is:

[0027]

[0028] Round-trip delay is:

[0029]

[0030] Therefore, the amplitude term considering geometric diffusion and reflectivity can be expressed as:

[0031]

[0032] in, This represents a constant gain, such as skin reflectivity. After reflection and down-conversion, the received complex baseband signal can be expressed as:

[0033]

[0034] in This represents additive noise. The equation shows that the received signal is determined by the following factors:

[0035] 1. Speed-related delay Its time variation is determined by hand speed Control and decide on items

[0036] 2. Distance-related magnitude It changes as the hand moves closer to or further away.

[0037] Therefore, the speed of motion and the distance between the hand and the transceiver are directly reflected in the received waveform. This naturally leads to differences in gesture features within the class.

[0038] To comprehensively represent the speed changes throughout the entire gesture execution process, a Short-Time Fourier Transform (STFT) is used to extract the time-spectrum of the signal. Specifically, given two spectra... and They correspond to For two differential channel impulse response (DiffCIR) instances on consecutive frames, calculate the frame-by-frame Wasserstein distance to quantify the velocity change at each time step. Assume... as well as Represents time frame The amplitude spectrum at each position, each amplitude spectrum is composed of It consists of several frequency ranges.

[0039] Each spectrogram frame is normalized to form an effective probability distribution:

[0040]

[0041] In frame At a given point, the Wasserstein distance between two distributions is defined as:

[0042]

[0043] in, Indicates marginal distribution and The set of all valid joint distributions, Indicates the frequency range and The quadratic cost of transmission quality between them. The resulting velocity offset embedding is a dimensional vector It captures the frame-by-frame velocity distribution difference between two DiffCIR instances.

[0044] To effectively characterize the temporal evolution of the distance offset between two DiffCIR instances, this invention employs frame-by-frame difference quantization of the distance difference based on the delay centroid. Specifically, given two DiffCIR data sequences corresponding to real instances... and Calculate each time frame Delayed centroid:

[0045]

[0046] in, This indicates the first DiffCIR vector. The propagation delay corresponding to each element It is the total number of delay bins after distance truncation. Indicates time Time The magnitude of the delay. Then, the frame distance offset is calculated as the difference between the centroids of the two instances:

[0047]

[0048] The resulting distance offset embedding It captured the temporal evolution of distance differences.

[0049] A few-sample gesture recognition method based on ultrasound signals, the brief process of which is as follows: Figure 1 As shown, the specific steps are as follows:

[0050] Customization stage:

[0051] Step 1): Signal acquisition: The user makes a specific gesture within a certain range of the data acquisition device, and the device's speaker and microphone acquire ultrasound signals containing the user's gesture characteristics;

[0052] Specifically, the data acquisition equipment needs to be equipped with a speaker and a microphone. During the acquisition time of each data point, the waveform output by the speaker consists of the following two sequential parts in the time domain:

[0053] Speaker Warm-up Signal: Before the official FM signal is sent, a single-frequency sine wave signal lasting 0.5 seconds is played, with a frequency equal to the starting frequency of the FM signal (18 kHz). This warm-up signal is used to eliminate the influence of mechanical inertia and nonlinear response during the speaker startup phase on the subsequent signal quality, thereby ensuring the stability and repeatability of the FM signal.

[0054] Repeated Chirp Signal Sequence: After the warm-up phase, the speaker continuously plays a repeated chirp signal multiple times, each chirp cycle lasting 0.1 seconds, repeated 20 times. Within a single cycle, the instantaneous frequency of each chirp signal linearly increases from 18 kHz to 22 kHz, forming an up-chirp waveform. There is no time interval between adjacent chirp signals; the chirp signals are sequentially connected, forming a continuous chirp excitation sequence. The user needs to repeatedly make specific gestures several times near the speaker and microphone (within <1m), while the microphone captures the channel impulse response characteristics including the user's gestures at a sampling rate of 48 kHz.

[0055] Step 2): Signal preprocessing: Based on the ultrasound signal containing user gesture features, the differential channel impulse response features of the training set are extracted through low-pass filtering, range truncation, and response calculation.

[0056] Specifically, in order to obtain clean and representative gesture features from the received signal, the present invention performs three preprocessing steps: channel impulse response feature extraction, range truncation, and differential calculation.

[0057] In an ultrasonic human motion capture (HGR) system, signals are received. It can be represented as a transmitted signal Channel Impulse Response (CIR) Convolution: ,in This represents a linear convolution operation. Based on known... and The CIR can be estimated by solving an optimization problem. The specific method is as follows: Based on the cross-correlation function, locate the starting position of the chirp in the signal obtained in step 1), ensuring that the received segment of each chirp cycle is strictly aligned with the transmitted signal; extract the received signal corresponding to a single chirp from the signal obtained in step 1); remove low-frequency environmental noise through ultrasonic band filtering, retaining the ultrasonic components around 18–22 kHz emitted by the loudspeaker; demodulate by multiplying the received single chirp signal with the signal emitted by the loudspeaker; remove the high-frequency cross terms generated after demodulation through low-pass filtering, retaining only the frequency range corresponding to the physical propagation distance; finally, obtain the CIR vector through Fast Fourier Transform (FFT).

[0058] To focus on gesture-related signal components and reduce interference from distant moving objects, this invention further performs range-based truncation on the original CIR signal. In a CIR vector, each element corresponds to a specific propagation delay that can be directly mapped to the physical distance to the transceiver. Therefore, this invention truncates each CIR vector by retaining only elements within a certain range to preserve reflections within the target range.

[0059] After truncation, the frame-by-frame absolute difference of the truncated CIR vector is calculated, and DiffCIR is calculated to capture changes caused by dynamic motion while minimizing the effects of static reflections.

[0060] Step 3): Conditional coding: Based on the ordered pairs of differential channel impulse response features in the training set, calculate the multidimensional conditional vector of the training set using velocity coding and distance coding algorithms;

[0061] Specifically, the velocity encoding of the present invention obtains a temporal spectrum by applying STFT to the ordered DiffCIR pairs obtained in step 2) and calculating the Wasserstein distance frame by frame on the temporal spectrum to quantize the velocity difference; the distance encoding calculates the frame-by-frame difference of the delay centroid between DiffCIR instances by using the ordered DiffCIR pairs obtained in step 2) to quantize the distance difference.

[0062] Step 4): Diffusion model training: Based on the ordered pairs of differential channel impulse response features in the training set and the multidimensional conditional vectors in the training set, train the graph-based conditional diffusion model CIR-ShiftNet, which is specifically designed for ultrasound signals;

[0063] Specifically, the graph-based diffusion model CIR-ShiftNet adapted to ultrasound signals designed in this invention is as follows: Figure 3 As shown. The training method is to use the ordered DiffCIR pairs obtained in step 2) to specify the positive noise start point and positive noise end point respectively, and the condition vector is the multi-dimensional condition vector obtained in step 3) to train the diffusion model.

[0064] Data Augmentation Phase

[0065] Signal acquisition: The user makes a specific gesture within a certain range of the data acquisition device, and the device's speaker and microphone collect ultrasonic signals containing the user's gesture characteristics;

[0066] Signal preprocessing: Based on the ultrasound signal containing user gesture features, the user differential channel impulse response features are extracted through low-pass filtering, range truncation, and response calculation;

[0067] Virtual data generation: Based on the multidimensional conditional vectors of the training set, the graph-generated graph conditional diffusion model CIR-ShiftNet, and the user differential channel impulse response features, virtual differential channel impulse response feature data is generated;

[0068] Specifically, this invention utilizes the conditionalUnet code provided by HuggingFace as a diffusion model prototype. Using the DiffCIR obtained in step 2) as the starting point for reverse denoising, the multidimensional conditional vector obtained in step 3) of the customization phase as the diffusion condition, and the conditional diffusion model obtained in step 4) of the customization phase, the denoising process is executed from the starting point for reverse denoising to obtain virtual differential channel impulse response characteristic data.

[0069] Gesture recognition: Based on the user's differential channel impulse response characteristics and virtual differential channel impulse response characteristics, different types of models such as convolutional neural networks and long short-term memory networks are selected according to user needs to train a classification model for a specific gesture recognition task.

[0070] Specifically, the present invention first uses the user differential channel impulse response features obtained in step 2) to train a classification model, and obtains an initialized gesture recognition model after the loss converges; then, the virtual differential channel impulse response feature data in step 3) is introduced into the training set, and the final gesture recognition model is obtained after the loss converges.

[0071] This invention addresses the problems of high data acquisition costs and difficulty in achieving stable recognition with small sample sizes in existing ultrasound gesture recognition technologies. It proposes a method for generating highly diverse virtual data based on ultrasound signals and achieving gesture recognition with a small sample size. This invention does not require users to provide large amounts of data; users only need to collect 2–6 real gesture samples to achieve stable recognition through virtual data augmentation. Figure 4 As shown, the average recognition accuracy of 2-shot, 4-shot and 6-shot methods reached 86.0%, 89.5% and 96.0% respectively, which is more than 20 percentage points higher than the existing methods.

[0072] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for virtual data generation and classification for few-sample ultrasound gesture recognition, characterized in that, Includes the following steps: Customization stage Signal acquisition: The user makes a specific gesture within a certain range of the data acquisition device, and the device's speaker and microphone collect ultrasonic signals containing the user's gesture characteristics; Signal preprocessing: Based on the ultrasound signal containing user gestures, differential channel impulse response features of the training set are extracted through channel impulse response feature extraction, range truncation, and differential calculation; Conditional coding: Based on the ordered pairs of differential channel impulse response features in the training set, a multidimensional conditional vector of the training set is calculated using velocity coding and distance coding algorithms; Diffusion Model Training: Based on the ordered pairs of differential channel impulse response features in the training set and the multidimensional conditional vectors in the training set, a graph-generated graph conditional diffusion model CIR-ShiftNet, specifically designed for ultrasound signals, is trained. Data Augmentation Phase Signal acquisition: The user makes a specific gesture within a certain range of the data acquisition device, and the device's speaker and microphone collect ultrasonic signals containing the user's gesture characteristics; Signal preprocessing: Based on the ultrasound signal containing user gestures, the user differential channel impulse response features are extracted through channel impulse response feature extraction, range truncation, and differential calculation; Virtual data generation: Based on the multidimensional conditional vectors of the training set, the graph-generated graph conditional diffusion model CIR-ShiftNet, and the user differential channel impulse response features, virtual differential channel impulse response feature data is generated; Gesture recognition: Based on the user's differential channel impulse response characteristics and virtual differential channel impulse response characteristics, different types of models such as convolutional neural networks and long short-term memory networks are selected according to user needs to train a classification model for a specific gesture recognition task.

2. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 1, characterized in that, In the channel impulse response feature extraction method of the signal preprocessing steps in the customization stage and data enhancement stage, a low-wavelength filter is used to control the response distance, with a cutoff frequency of 150-250Hz.

3. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 1, characterized in that, The range truncation method in the signal preprocessing steps of the customization stage and data enhancement stage is to retain the effective distance interval, specifically by retaining bins 10-40.

4. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 1, 2, or 3, characterized in that, The response calculation method in the signal preprocessing steps of the customization stage and data enhancement stage is to calculate the frame-by-frame difference of the channel impulse response feature vector to obtain the differential channel impulse response feature, which is used to describe the dynamic changes of micro multipath caused by gestures.

5. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 4, characterized in that, The speed coding algorithm in the customized stage conditional coding step is the Wasserstein distance of the ordered pairs of differential channel impulse response features of the training set frame by frame, and the distance coding algorithm is the centroid difference of the ordered pairs of differential channel impulse response features of the training set frame by frame.

6. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 5, characterized in that, The graph-generated graph conditional diffusion model CIR-ShiftNet is a complex-domain differentiable model that uses a complex attention mechanism to achieve conditional injection and backpropagation training.

7. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 6, characterized in that, The graph-generated graph conditional diffusion model CIR-ShiftNet is trained by using ordered pairs of differential channel impulse response features from the training set as the start and end points of the diffusion process, respectively, and using the multidimensional condition vectors corresponding to the ordered pairs of differential channel impulse response features from the training set as the condition vectors for the diffusion process.

8. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 1, 2, 3, 5, 6, or 7, characterized in that, The virtual data generation method described in the above steps includes the following steps: Based on the user differential channel impulse response characteristics, noise perturbation is added forward to obtain the user differential channel impulse response characteristics with added noise perturbation; Based on the user differential channel impulse response features with added noise perturbation, the multidimensional conditional vector of the training set, and the graph-generated graph conditional diffusion model CIR-ShiftNet, virtual differential channel impulse response features are generated using the inverse denoising process of the diffusion model.

9. The virtual data generation and classification method for few-sample ultrasound gesture recognition according to claim 8, characterized in that, The process of training the gesture recognition model in the gesture recognition step includes: The user differential channel impulse response feature samples are used for initial learning to obtain the initialized gesture recognition model; Based on the initialized gesture recognition model and virtual differential channel impulse response characteristics, a lightweight or deep model is selected according to the computing power of the user equipment, and the training distribution is expanded using generated virtual samples.

Images