Air gesture recognition method, device, apparatus and storage medium

Abstract

The application discloses an air gesture recognition method and device, equipment and storage medium, and relates to the technical field of gesture recognition, which comprises the following steps: collecting high-frequency acoustic signals generated by a user performing a gesture in the air through a microphone; performing time-frequency conversion on the high-frequency acoustic signals to generate a two-dimensional time-frequency spectrum; performing spectrum clipping on the two-dimensional time-frequency spectrum to retain target spectrum information in a preset high-frequency band; filtering the target spectrum information to extract potential gesture segments; inputting the potential gesture segments into a gesture recognition model to output a gesture category, and triggering a corresponding interaction instruction according to the gesture category. The application can improve the high-precision recognition of gestures.

Description

Technical Field

[0001] This application relates to the field of gesture recognition technology, and in particular to methods, devices, equipment and storage media for air gesture recognition. Background Technology

[0002] With the widespread adoption of smart wearable devices in daily life, headphones have gradually evolved from traditional audio playback tools into smart terminals integrating multiple interactive functions. However, limited by their compact physical structure and fixed wearing position, existing interaction methods based on physical buttons or touch panels not only have limited operating space and single recognition dimensions, but also easily cause ear discomfort due to repeated touching. Especially in sports, commuting, or noisy environments, the limitations and inconveniences of traditional interaction methods are more prominent. In recent years, non-contact interaction methods based on air gestures have received widespread attention due to their advantages such as natural operation, intuitive response, and avoidance of contact contamination. Among them, the acoustic perception-based technology path has become a research hotspot due to its low power consumption, privacy-friendly, and easy deployment characteristics. However, most existing acoustic gesture recognition systems use active sound sources (such as speakers) to emit ultrasonic signals and receive reflected echoes to detect hand movements. This method not only increases system power consumption, but may also cause auditory interference to users due to sound wave leakage. Summary of the Invention

[0003] The main purpose of this application is to provide a method, device, equipment and storage medium for air gesture recognition, which aims to solve the technical problem that existing acoustic gesture recognition systems are prone to causing auditory interference to users due to sound wave leakage.

[0004] To achieve the above objectives, this application proposes a method for air gesture recognition, the air gesture recognition method comprising:

[0005] High-frequency acoustic signals generated when a user performs gestures in the air are collected via a microphone;

[0006] The high-frequency acoustic signal is subjected to time-frequency transformation to generate a two-dimensional time-frequency spectrum.

[0007] The two-dimensional time-frequency spectrum is cropped to retain the target spectrum information within a preset high-frequency band;

[0008] The target spectrum information is filtered to extract potential gesture fragments;

[0009] The potential gesture fragments are input into the gesture recognition model, which outputs the gesture category and triggers the corresponding interaction command based on the gesture category.

[0010] In one embodiment, the step of acquiring high-frequency acoustic signals generated when a user performs a gesture in the air via a microphone includes:

[0011] Acoustic signals generated when the user performs gestures in the air are collected according to a preset sampling frequency.

[0012] The frequency band of the acoustic signal that is within the range of low hearing sensitivity of the human ear is defined as the high-frequency acoustic signal.

[0013] In one embodiment, the step of performing time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-spectrum graph includes:

[0014] Based on preset parameter configuration, the high-frequency acoustic signal is subjected to frame segmentation and windowing processing to obtain a preprocessed signal frame. The preset parameter configuration includes the fast Fourier transform length, frame shift, and Hamming window function.

[0015] The preprocessed signal frame is converted into a two-dimensional time-spectrum graph using a short-time Fourier transform.

[0016] In one embodiment, the step of filtering the target spectral information and extracting potential gesture fragments includes:

[0017] The target spectrum information is subjected to noise separation to obtain the calculated signal energy and noise energy. The signal-to-noise ratio is determined based on the calculated signal energy and noise energy, and an energy threshold is determined based on the signal-to-noise ratio.

[0018] Calculate the average signal energy of each time frame in the target spectrum information;

[0019] Extract time frame sequences from the target spectrum information where the average signal energy continuously exceeds the energy threshold and the duration reaches a preset number of frames, as potential gesture segments.

[0020] In one embodiment, before the step of inputting the potential gesture fragment into the gesture recognition model and outputting the gesture category, the method further includes:

[0021] Construct an initial gesture recognition model, wherein the initial gesture recognition model includes at least a deep residual network, a domain adversarial neural network, and a gradient inversion layer, and the domain adversarial neural network includes at least a gesture classifier and a user domain classifier;

[0022] Based on the training dataset, the initial gesture recognition model is trained to obtain the first gesture recognition model after training.

[0023] The first gesture recognition model is validated based on the test dataset, and the model parameters of the first gesture recognition model are adjusted based on the validation results to obtain the final gesture recognition model.

[0024] In one embodiment, the step of training the initial gesture recognition model based on the training dataset to obtain the trained first gesture recognition model includes:

[0025] Training samples are randomly obtained from the training dataset, wherein the training samples include training gesture fragments, real gesture category labels, and user identity labels;

[0026] The training gesture fragments are input into the initial gesture recognition model, and the deep residual network is used to extract features from the training gesture fragments to obtain sample features.

[0027] The sample features are processed by the gesture classifier and the user domain classifier to obtain gesture prediction results and user prediction results.

[0028] Based on the gesture prediction results, the user prediction results, the real gesture category labels, and the user identity labels, the initial gesture recognition model is trained to obtain the trained first gesture recognition model.

[0029] In one embodiment, the step of training the initial gesture recognition model based on the gesture prediction result, the user prediction result, the real gesture category label, and the user identity label to obtain the trained first gesture recognition model includes:

[0030] The gesture classification loss is calculated based on the gesture prediction result and the actual gesture category label, and the user classification loss is calculated based on the user prediction result and the user identity label.

[0031] During backpropagation, the gradient of the user classification loss is inverted through the gradient inversion layer to obtain the target user gradient, and the target user gradient is combined with the gradient of the gesture classification loss to obtain the target gradient.

[0032] The model parameters of the initial gesture recognition model are updated according to the target gradient to obtain the first gesture recognition model after training.

[0033] Furthermore, to achieve the above objectives, this application also proposes a gesture recognition device, which includes:

[0034] The acquisition module is used to acquire high-frequency acoustic signals generated when a user performs gestures in the air via a microphone;

[0035] The generation module is used to perform time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-frequency spectrum.

[0036] The retention module is used to perform spectrum cropping on the two-dimensional time-spectrum diagram and retain the target spectrum information within a preset high-frequency band;

[0037] The extraction module is used to filter the target spectrum information and extract potential gesture fragments;

[0038] The output module is used to input the potential gesture fragments into the gesture recognition model, output the gesture category, and trigger the corresponding interaction command according to the gesture category.

[0039] In addition, to achieve the above objectives, this application also proposes an air gesture recognition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the air gesture recognition method as described above.

[0040] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the air gesture recognition method described above.

[0041] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the air gesture recognition method described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] The air gesture recognition method, device, equipment, and storage medium proposed in this application acquire high-frequency acoustic signals generated when a user performs gestures in the air using a microphone; perform time-frequency transformation on the high-frequency acoustic signals to generate a two-dimensional time-frequency spectrum; perform spectrum clipping on the two-dimensional time-frequency spectrum to retain target spectrum information within a preset high-frequency band; filter the target spectrum information to extract potential gesture segments; input the potential gesture segments into a gesture recognition model, output the gesture category, and trigger corresponding interaction commands based on the gesture category. This application addresses the technical problem of existing acoustic gesture recognition systems causing auditory interference due to sound wave leakage. Compared to existing technologies, this application fundamentally solves the sound wave leakage interference problem caused by the active emission of sound waves in existing acoustic gesture recognition systems through a purely passive acoustic sensing scheme. This not only eliminates the potential impact on users' hearing but also significantly reduces system power consumption. By employing a signal processing mechanism that combines spectrum clipping and energy threshold filtering, environmental noise interference is effectively suppressed, maintaining stable recognition performance even in complex usage scenarios. Simultaneously, the gesture recognition model based on a deep residual network can accurately extract gesture features from the time-spectrum image, achieving high-precision recognition of various gestures. This solution fully utilizes the existing hardware resources of the headphones without adding any additional acoustic components, significantly reducing hardware costs and implementation complexity. It provides consumer-grade smart headphones with an accurate, natural user experience, low power consumption, and easy-to-deploy air gesture interaction solution. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an embodiment of the air gesture recognition method of this application.

[0047] Figure 2 A comparison diagram before and after frequency band clipping provided in Embodiment 1 of the air gesture recognition method of this application;

[0048] Figure 3 The following are the gesture response spectra of different people provided in Embodiment 1 of the air gesture recognition method of this application;

[0049] Figure 4 This is a system architecture diagram provided for Embodiment 1 of the air gesture recognition method of this application;

[0050] Figure 5 This is a flowchart illustrating Embodiment 2 of the air gesture recognition method of this application;

[0051] Figure 6 This is a schematic diagram of the module structure of the air gesture recognition device according to an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the air gesture recognition method in the embodiments of this application.

[0053] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0055] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of this application embodiment is as follows: high-frequency acoustic signals generated when a user performs gestures in the air are collected by a microphone; the high-frequency acoustic signals are transformed by time and frequency to generate a two-dimensional time-frequency spectrum; the two-dimensional time-frequency spectrum is spectral cropped to retain target spectrum information within a preset high-frequency band; the target spectrum information is filtered to extract potential gesture segments; the potential gesture segments are input into a gesture recognition model to output the gesture category, and corresponding interaction commands are triggered according to the gesture category.

[0057] As can be seen from the above embodiments, this application collects high-frequency acoustic signals generated when a user performs gestures in the air using a microphone; performs time-frequency transformation on the high-frequency acoustic signals to generate a two-dimensional time-frequency spectrum; performs spectrum clipping on the two-dimensional time-frequency spectrum to retain target spectrum information within a preset high-frequency band; filters the target spectrum information to extract potential gesture segments; inputs the potential gesture segments into a gesture recognition model, outputs the gesture category, and triggers corresponding interaction commands based on the gesture category. This application addresses the technical problem of existing acoustic gesture recognition systems causing auditory interference due to sound wave leakage. Compared to existing technologies, this application fundamentally solves the sound wave leakage interference problem caused by the active emission of sound waves in existing acoustic gesture recognition systems through a purely passive acoustic sensing scheme. This not only eliminates the potential impact on users' hearing but also significantly reduces system power consumption. By employing a signal processing mechanism that combines spectrum clipping and energy threshold filtering, environmental noise interference is effectively suppressed, maintaining stable recognition performance even in complex usage scenarios. Simultaneously, the gesture recognition model based on a deep residual network can accurately extract gesture features from the time-spectrum image, achieving high-precision recognition of various gestures. This solution fully utilizes the existing hardware resources of the headphones without adding any additional acoustic components, significantly reducing hardware costs and implementation complexity. It provides consumer-grade smart headphones with an accurate, natural user experience, low power consumption, and easy-to-deploy air gesture interaction solution.

[0058] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or air gesture recognition device capable of performing the above functions. The following description uses an air gesture recognition device as an example to illustrate this embodiment and the subsequent embodiments.

[0059] Based on this, embodiments of this application provide a method for air gesture recognition, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the air gesture recognition method of this application.

[0060] In this embodiment, the air gesture recognition method includes steps S10 to S50:

[0061] Step S10: Collect high-frequency acoustic signals generated when the user performs gestures in the air using a microphone;

[0062] It should be noted that the high-frequency acoustic signals naturally generated when the user performs air gestures such as swiping, clicking, and rubbing near the ear can be passively collected through the external microphone of the earphone. Based on the high-frequency acoustic signals, air gesture recognition and interactive control functions can be realized without active sound source emission, without additional hardware support, and with user-independent recognition capabilities.

[0063] In one feasible implementation, the step of acquiring the high-frequency acoustic signal generated when the user performs a gesture in the air via a microphone includes: acquiring the acoustic signal generated when the user performs a gesture in the air according to a preset sampling frequency; and taking the frequency band of the acoustic signal that is in the low hearing sensitivity of the human ear as the high-frequency acoustic signal.

[0064] It should be noted that, firstly, the external microphone of the headphones (such as a noise-canceling microphone) continuously collects audio signals from the surrounding environment at a sampling rate of 48kHz (i.e., the preset sampling frequency). These signals include short-term high-frequency sound wave components generated by physical actions such as friction and collision when the user's fingers move in the air. Their frequencies are mainly concentrated between 12kHz and 22kHz, which are in the frequency band where human hearing sensitivity is low, so they are not easy to attract the user's attention. At the same time, they are significantly distinguishable from everyday environmental noise (such as voice, air conditioning, traffic noise, etc.) in the spectrum.

[0065] Step S20: Perform time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-frequency spectrum.

[0066] In one feasible implementation, the step of performing time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-spectrum graph includes: performing frame-segmentation and windowing processing on the high-frequency acoustic signal based on a preset parameter configuration to obtain a preprocessed signal frame, wherein the preset parameter configuration includes frame shift and Hamming window function; and converting the preprocessed signal frame into a two-dimensional time-spectrum graph through short-time Fourier transform.

[0067] It should be noted that the acquired raw audio stream (i.e., high-frequency acoustic signal) is framed and windowed, and then converted into a two-dimensional time-frequency spectrum using a short-time Fourier transform (STFT). In this process, a parameter configuration of 512-point FFT length (i.e., fast Fourier transform length) and 256-point frame shift (i.e., frame shift) is used to ensure a balance between time and frequency resolution. Furthermore, the Hamming window function is used to suppress spectral leakage, thereby improving the spectrum quality and feature clarity.

[0068] Step S30: Perform spectrum cropping on the two-dimensional time spectrum diagram to retain the target spectrum information within the preset high-frequency band;

[0069] It should be noted that after obtaining the two-dimensional time spectrum, a spectrum clipping and energy filtering mechanism can be implemented to effectively suppress the interference of environmental noise and non-interactive segments. Specifically, the spectrum region with frequencies between 12kHz and 22kHz (i.e., within the preset high-frequency band) can be retained by spectrum clipping, thereby filtering out most of the low-frequency environmental noise (such as human voices, electrical appliance operation sounds, etc.).

[0070] Step S40: Filter the target spectrum information to extract potential gesture segments;

[0071] In one feasible implementation, the step of filtering the target spectrum information and extracting potential gesture segments includes: performing noise separation on the target spectrum information to obtain calculated signal energy and noise energy; determining the signal-to-noise ratio based on the calculated signal energy and noise energy; determining an energy threshold based on the signal-to-noise ratio; calculating the average signal energy of each time frame in the target spectrum information; and extracting a time frame sequence from the target spectrum information where the average signal energy continuously exceeds the energy threshold and the continuous duration reaches a preset number of frames, as a potential gesture segment.

[0072] It should be noted that the average signal energy of each frame in the target spectrum information can be calculated, and a signal-to-noise ratio threshold (i.e., energy threshold) can be set to 5dB. Only spectrum segments with energy consistently above the energy threshold and appearing consecutively for more than five frames are retained as potential gesture regions (i.e., potential gesture segments). This ensures that the signal segments subsequently entering the gesture recognition model have high reliability and representativeness, while also significantly reducing the system's computational burden and the risk of misidentification. Furthermore, as... Figure 2 As shown, in order to achieve real-time processing and response, a sliding window with a length of 0.5 seconds is used to continuously segment the audio stream with a step size of 0.05 seconds. Each time, only the newly added spectrogram data is updated, thereby achieving a low-latency real-time interactive experience while ensuring recognition accuracy.

[0073] Step S50: Input the potential gesture fragment into the gesture recognition model, output the gesture category, and trigger the corresponding interaction command according to the gesture category.

[0074] It should be noted that after signal preprocessing and feature extraction, a deep learning-based gesture recognition model is used to classify and identify potential gesture segments. This model uses a ResNet-18 network as its backbone and adapts the number of channels in its first convolutional layer to accommodate single-channel spectral input, thereby predicting the probability of gesture categories. The model output includes seven interactive gesture categories (single, double, swipe up, swipe down, swipe left, swipe right) and an empty class (indicating no gesture occurred), thus covering the most commonly used operation types in daily media control. To further improve the model's generalization ability among different users, a Domain-Adversarial Neural Network (DANN) training strategy is introduced. By constructing a gesture classifier and a user domain classifier in parallel in the model, and introducing a Gradient Reversal Layer (GRL) during backpropagation to invert the gradient of the domain classifier, the feature extractor is forced to learn a universal feature representation with user invariance. This allows the model to maintain high recognition performance without retraining or personalized calibration when facing new users, significantly improving the system's versatility and practicality.

[0075] It's important to note that during gesture recognition, we categorize gestures into eight types: seven interactive and seven non-interactive. We found that these seven interactive gestures have fixed patterns, while the patterns of non-interactive samples are random, making it impossible to exhaustively enumerate all patterns of non-interactive samples. Therefore, in the prediction phase, we employ a confidence strategy to handle unseen non-interactive samples. Specifically, besides having the gesture classifier directly predict non-interactive samples, we also use the probability distribution provided by the Softmax layer to determine whether a gesture belongs to the non-interactive category. If the probability does not exceed a set threshold in any of the seven categories (i.e., the model's prediction confidence for that sample is low), it is classified as a non-interactive sample. Figure 3 As shown, we set the threshold to 0.5 through experiments. When a potential gesture fragment is input into the gesture recognition model, it outputs the predicted probabilities of all interactive gesture categories. If the predicted probabilities of all interactive gesture categories are lower than the probability threshold, the final gesture category can be determined as a non-interactive gesture category. If any of the predicted probabilities of all interactive gesture categories is higher than the probability threshold, then the interactive gesture category with the highest predicted probability is taken as the final gesture category.

[0076] It should be noted that after gesture recognition is completed, corresponding interactive commands will be triggered based on the recognition results. The control commands will be transmitted to the media player or operating system through the communication interface between the headphones and the terminal device, realizing functions such as single-click to pause / play, double-click to favorite, swiping up and down to adjust the volume, and swiping left and right to switch tracks. The entire recognition and response process is completed locally without uploading any audio or user data to the cloud, ensuring the security of user privacy and the real-time response of the system. At the same time, since the system does not need to emit any sound wave signals or rely on image or video acquisition during operation, it has extremely high privacy friendliness and energy efficiency, making it particularly suitable for wearable device scenarios with high requirements for power consumption, privacy, and natural interaction.

[0077] In specific implementations, such as Figure 4 The system architecture diagram shown illustrates that the air gesture recognition method primarily consists of a signal preprocessing module and a gesture recognition module. The signal preprocessing module acquires audio signals via a headset microphone, converts them into a time-frequency representation, and performs preliminary noise and gap filtering. The gesture recognition module, based on the time-frequency representation generated by the signal preprocessing module, uses a deep learning model to predict gestures. To ensure the model's universality for new users, the system employs an adversarial training strategy, forcing the model to tolerate differences between users. The prediction result includes seven finger gestures and one invalid category (no detected gesture), outputting the recognized air gesture.

[0078] This embodiment acquires high-frequency acoustic signals generated when a user performs gestures in the air using a microphone; performs time-frequency transformation on the high-frequency acoustic signals to generate a two-dimensional time-frequency spectrum; performs spectrum cropping on the two-dimensional time-frequency spectrum to retain target spectrum information within a preset high-frequency band; filters the target spectrum information to extract potential gesture segments; inputs the potential gesture segments into a gesture recognition model, outputs the gesture category, and triggers corresponding interaction commands based on the gesture category. This application addresses the technical problem of existing acoustic gesture recognition systems causing auditory interference due to sound wave leakage. Compared to existing technologies, this application fundamentally solves the sound wave leakage interference problem caused by the active emission of sound waves in existing acoustic gesture recognition systems through a purely passive acoustic sensing scheme. This not only eliminates the potential impact on users' hearing but also significantly reduces system power consumption. By employing a signal processing mechanism that combines spectrum clipping and energy threshold filtering, environmental noise interference is effectively suppressed, maintaining stable recognition performance even in complex usage scenarios. Simultaneously, the gesture recognition model based on a deep residual network can accurately extract gesture features from the time-spectrum image, achieving high-precision recognition of various gestures. This solution fully utilizes the existing hardware resources of the headphones without adding any additional acoustic components, significantly reducing hardware costs and implementation complexity. It provides consumer-grade smart headphones with an accurate, natural user experience, low power consumption, and easy-to-deploy air gesture interaction solution.

[0079] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Before step S50, steps S501 to S503 are also included:

[0080] Step S501: Construct an initial gesture recognition model, wherein the initial gesture recognition model includes at least a deep residual network, a domain adversarial neural network, and a gradient inversion layer, and the domain adversarial neural network includes at least a gesture classifier and a user domain classifier;

[0081] It should be noted that the gesture recognition model is a deep residual network based on the ResNet-18 network, and the number of channels in its first convolutional layer has been modified to adapt to the input of a single-channel spectral map. To further improve the model's generalization ability among different users, a Domain-Adversarial Neural Network (DANN) training strategy is introduced. By constructing a gesture classifier and a user domain classifier in parallel in the model, and introducing a gradient reversal layer (GRL) during backpropagation to invert the gradient of the user domain classifier, the feature extractor is forced to learn a general feature representation with user invariance. This allows the model to maintain high recognition performance when facing new users without retraining or personalized calibration, significantly improving the system's versatility and practicality.

[0082] Step S502: Based on the training dataset, train the initial gesture recognition model to obtain the trained first gesture recognition model;

[0083] It should be noted that by constructing a training dataset containing multi-user gesture samples, the parameters of the initial gesture recognition model are optimized using an end-to-end training method. During the training process, the model calculates the loss value between the predicted result and the real label through forward propagation, and then updates the network weight parameters through the backpropagation algorithm. After multiple rounds of iterative training, the model gradually learns the accurate mapping relationship from the input acoustic features to the gesture category, and finally obtains the first gesture recognition model with stable recognition capabilities. This training process ensures that the model can effectively extract the discriminative features of gestures, laying the foundation for high-precision gesture recognition in subsequent practical applications.

[0084] In one feasible implementation, the step of training the initial gesture recognition model based on the training dataset to obtain the trained first gesture recognition model includes: randomly obtaining training samples from the training dataset, wherein the training samples include training gesture segments, real gesture category labels, and user identity labels; inputting the training gesture segments into the initial gesture recognition model, and extracting features from the training gesture segments using the deep residual network to obtain sample features; processing the sample features using the gesture classifier and the user domain classifier to obtain gesture prediction results and user prediction results; and training the initial gesture recognition model based on the gesture prediction results, the user prediction results, the real gesture category labels, and the user identity labels to obtain the trained first gesture recognition model.

[0085] It should be noted that a domain adversarial training framework was constructed by introducing user identity labels and a user domain classifier. A deep residual network serves as a shared feature extractor, providing feature representations for both gesture classification and user identity recognition. During training, the recognition accuracy is improved by minimizing the gesture classification loss, while the feature extractor is forced to learn user-independent general gesture features by maximizing the recognition difficulty of the user domain classifier. This adversarial training mechanism effectively solves the model's generalization problem among different users, enabling the final first gesture recognition model to overcome recognition biases caused by individual differences. This achieves consistent recognition performance for new users without requiring retraining with user-specific data, significantly improving the system's practicality and adaptability.

[0086] In one feasible implementation, the step of training the initial gesture recognition model based on the gesture prediction result, the user prediction result, the real gesture category label, and the user identity label to obtain the trained first gesture recognition model includes: calculating the gesture classification loss based on the gesture prediction result and the real gesture category label, and calculating the user classification loss based on the user prediction result and the user identity label; during backpropagation, inverting the gradient of the user classification loss through the gradient inversion layer to obtain the target user gradient, and combining the target user gradient with the gradient of the gesture classification loss to obtain the target gradient; updating the model parameters of the initial gesture recognition model based on the target gradient to obtain the trained first gesture recognition model.

[0087] It should be noted that domain adversarial training is implemented through a gradient reversal layer (GRL), which acts as an identity function during forward propagation. The gradient inversion layer inverts the gradient of the user classification loss during backpropagation by multiplying it by a negative factor.

[0088]

[0089] in, It is a hyperparameter used to control adversarial weights, and its value is usually gradually increased as training progresses.

[0090] This embodiment reveals the core mechanism of domain adversarial training. By simultaneously calculating the gesture classification loss and the user classification loss, and using a gradient inversion layer to invert the gradient of the user classification loss, an ingenious adversarial learning process is created: In the gesture classification task, the model improves recognition accuracy by minimizing the gesture classification loss; while in the user recognition task, the gradient inversion layer, by inverting the user classification gradient, actually forces the feature extractor to learn how to "deceive" the user classifier, that is, to extract features that make it difficult for the user classifier to distinguish the user's identity; this bidirectional gradient action enables the feature extractor to ultimately learn universal gesture features that are representative of different users, thereby significantly improving the model's generalization ability on new users and achieving the technical effect of maintaining stable recognition performance without user-specific calibration.

[0091] Step S503: Based on the test dataset, the first gesture recognition model is verified, and the model parameters of the first gesture recognition model are adjusted based on the verification results to obtain the final gesture recognition model.

[0092] It should be noted that an independent test dataset is introduced to objectively evaluate the first gesture recognition model after initial training. This test dataset contains user and environmental scene data not found in the training set, which can effectively simulate the model's performance in real-world application environments. During the verification process, the system evaluates the model's generalization ability and robustness by calculating performance indicators such as recognition accuracy and confusion matrix on the test set. Based on the verification results, the model parameters can be further optimized through hyperparameter tuning, model structure fine-tuning, or supplementary training to ultimately obtain a gesture recognition model that maintains high accuracy while possessing good generalization performance, ensuring that it can adapt to different users' usage habits and diverse environmental conditions in actual deployment.

[0093] This embodiment constructs an initial gesture recognition model, which includes at least a deep residual network, a domain adversarial neural network, and a gradient inversion layer. The domain adversarial neural network includes at least a gesture classifier and a user domain classifier. Based on a training dataset, the initial gesture recognition model is trained to obtain a trained first gesture recognition model. Based on a test dataset, the first gesture recognition model is validated, and its parameters are adjusted based on the validation results to obtain the final gesture recognition model. This approach constructs a complete model training and optimization process: the deep residual network serves as the backbone network for feature extraction, effectively extracting multi-level gesture features from the time-spectrum image; the domain adversarial neural network architecture, through collaborative training of the gesture classifier and the user domain classifier, combined with the inverse optimization mechanism of the gradient inversion layer, enables the model to learn universal feature representations insensitive to individual user differences while maintaining high-precision gesture recognition capabilities; finally, validation on the test set and parameter tuning ensure that the model has good generalization performance and robustness in practical application scenarios, ultimately achieving the technical effect of maintaining stable recognition accuracy among different users without user calibration, significantly improving the practicality and user experience of the air gesture interaction system.

[0094] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the air gesture recognition method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0095] This application also provides a gesture recognition device; please refer to [reference needed]. Figure 6 The air gesture recognition device includes:

[0096] Acquisition module 10 is used to acquire high-frequency acoustic signals generated when a user performs gestures in the air via a microphone;

[0097] The generation module 20 is used to perform time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-frequency spectrum.

[0098] The retention module 30 is used to perform spectrum cropping on the two-dimensional time spectrum diagram and retain the target spectrum information within a preset high-frequency band;

[0099] Extraction module 40 is used to filter the target spectrum information and extract potential gesture fragments;

[0100] The output module 50 is used to input the potential gesture fragment into the gesture recognition model, output the gesture category, and trigger the corresponding interaction command according to the gesture category.

[0101] The air gesture recognition device provided in this application, employing the air gesture recognition method in the above embodiments, can solve the technical problem that existing acoustic gesture recognition systems are prone to causing auditory interference to users due to sound wave leakage. Compared with the prior art, the beneficial effects of the air gesture recognition device provided in this application are the same as those of the air gesture recognition method provided in the above embodiments, and other technical features in the air gesture recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0102] This application provides a gesture recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the gesture recognition method in the above embodiment 1.

[0103] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the air gesture recognition device in the embodiments of this application. The air gesture recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The air gesture recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 7As shown, the air gesture recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the air gesture recognition device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the air gesture recognition device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show air gesture recognition devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0105] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0106] The air gesture recognition device provided in this application, employing the air gesture recognition method in the above embodiments, can solve the technical problem that existing acoustic gesture recognition systems are prone to causing auditory interference to users due to sound wave leakage. Compared with the prior art, the beneficial effects of the air gesture recognition device provided in this application are the same as those of the air gesture recognition method provided in the above embodiments, and other technical features in this air gesture recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the air gesture recognition method in the above embodiments.

[0110] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0111] The aforementioned computer-readable storage medium may be included in the air gesture recognition device; or it may exist independently and not assembled into the air gesture recognition device.

[0112] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the air gesture recognition device, cause the air gesture recognition device to: (independent solution).

[0113] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0116] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described air gesture recognition method. This solves the technical problem that existing acoustic gesture recognition systems are prone to causing auditory interference to users due to sound wave leakage. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the air gesture recognition method provided in the above embodiments, and will not be repeated here.

[0117] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the air gesture recognition method described above.

[0118] The computer program product provided in this application can solve the technical problem that existing acoustic gesture recognition systems are prone to causing auditory interference to users due to sound wave leakage. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the air gesture recognition method provided in the above embodiments, and will not be repeated here.

[0119] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for recognizing air gestures, characterized in that, The air gesture recognition method includes: The system collects high-frequency acoustic signals generated when a user performs gestures in the air using a microphone, wherein the microphone is an earphone microphone; The step of acquiring high-frequency acoustic signals generated when a user performs gestures in the air via a microphone includes: Acoustic signals generated when the user performs gestures in the air are collected according to a preset sampling frequency. The frequency band of the acoustic signal that is in the low hearing sensitivity of the human ear is defined as the high-frequency acoustic signal; The high-frequency acoustic signal is subjected to time-frequency transformation to generate a two-dimensional time-frequency spectrum. The step of performing time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-frequency spectrum includes: Based on preset parameter configuration, the high-frequency acoustic signal is subjected to frame segmentation and windowing processing to obtain a preprocessed signal frame. The preset parameter configuration includes the fast Fourier transform length, frame shift, and Hamming window function. The preprocessed signal frame is converted into a two-dimensional time-spectrum graph using a short-time Fourier transform; The two-dimensional time-frequency spectrum is cropped to retain the target spectrum information within a preset high-frequency band; The target spectrum information is filtered to extract potential gesture fragments; The step of filtering the target spectrum information and extracting potential gesture fragments includes: The target spectrum information is subjected to noise separation to obtain the calculated signal energy and noise energy. The signal-to-noise ratio is determined based on the calculated signal energy and noise energy, and an energy threshold is determined based on the signal-to-noise ratio. Calculate the average signal energy of each time frame in the target spectrum information; Extract time frame sequences from the target spectrum information where the average signal energy continuously exceeds the energy threshold and the continuous duration reaches a preset number of frames, as potential gesture segments; The potential gesture fragments are input into the gesture recognition model, which outputs the gesture category and triggers the corresponding interaction command based on the gesture category. Prior to the step of inputting the potential gesture fragment into the gesture recognition model and outputting the gesture category, the method further includes: Construct an initial gesture recognition model, wherein the initial gesture recognition model includes at least a deep residual network, a domain adversarial neural network, and a gradient inversion layer, and the domain adversarial neural network includes at least a gesture classifier and a user domain classifier; Based on the training dataset, the initial gesture recognition model is trained to obtain the first gesture recognition model after training. The first gesture recognition model is validated based on the test dataset, and the model parameters of the first gesture recognition model are adjusted based on the validation results to obtain the final gesture recognition model.

2. The air gesture recognition method as described in claim 1, characterized in that, The step of training the initial gesture recognition model based on the training dataset to obtain the trained first gesture recognition model includes: Training samples are randomly obtained from the training dataset, wherein the training samples include training gesture fragments, real gesture category labels, and user identity labels; The training gesture fragments are input into the initial gesture recognition model, and the deep residual network is used to extract features from the training gesture fragments to obtain sample features. The sample features are processed by the gesture classifier and the user domain classifier to obtain gesture prediction results and user prediction results. Based on the gesture prediction results, the user prediction results, the real gesture category labels, and the user identity labels, the initial gesture recognition model is trained to obtain the trained first gesture recognition model.

3. The air gesture recognition method as described in claim 2, characterized in that, The step of training the initial gesture recognition model based on the gesture prediction result, the user prediction result, the real gesture category label, and the user identity label to obtain the trained first gesture recognition model includes: The gesture classification loss is calculated based on the gesture prediction result and the actual gesture category label, and the user classification loss is calculated based on the user prediction result and the user identity label. During backpropagation, the gradient of the user classification loss is inverted through the gradient inversion layer to obtain the target user gradient, and the target user gradient is combined with the gradient of the gesture classification loss to obtain the target gradient. The model parameters of the initial gesture recognition model are updated according to the target gradient to obtain the first gesture recognition model after training.

4. A gesture recognition device, characterized in that, The air gesture recognition device includes: The acquisition module is used to acquire high-frequency acoustic signals generated when a user performs gestures in the air via a microphone, wherein the microphone is an earphone microphone; The acquisition module is also used to acquire acoustic signals generated when the user performs gestures in the air, according to a preset sampling frequency. The frequency band of the acoustic signal that is in the low hearing sensitivity of the human ear is defined as the high-frequency acoustic signal; The generation module is used to perform time-frequency transformation on the high-frequency acoustic signal to generate a two-dimensional time-frequency spectrum. The generation module is further configured to perform frame segmentation and windowing processing on the high-frequency acoustic signal based on preset parameter configuration to obtain a preprocessed signal frame. The preset parameter configuration includes the fast Fourier transform length, frame shift, and Hamming window function. The preprocessed signal frame is converted into a two-dimensional time-spectrum graph using a short-time Fourier transform; The retention module is used to perform spectrum cropping on the two-dimensional time-spectrum diagram and retain the target spectrum information within a preset high-frequency band; The extraction module is used to filter the target spectrum information and extract potential gesture fragments; The extraction module is further configured to perform noise separation on the target spectrum information to obtain calculated signal energy and noise energy, determine the signal-to-noise ratio based on the calculated signal energy and noise energy, and determine the energy threshold based on the signal-to-noise ratio. Calculate the average signal energy of each time frame in the target spectrum information; Extract time frame sequences from the target spectrum information where the average signal energy continuously exceeds the energy threshold and the continuous duration reaches a preset number of frames, as potential gesture segments; The output module is used to input the potential gesture fragments into the gesture recognition model, output the gesture category, and trigger the corresponding interaction command according to the gesture category; The output module is also used to construct an initial gesture recognition model, wherein the initial gesture recognition model includes at least a deep residual network, a domain adversarial neural network, and a gradient inversion layer, and the domain adversarial neural network includes at least a gesture classifier and a user domain classifier. Based on the training dataset, the initial gesture recognition model is trained to obtain the first gesture recognition model after training. The first gesture recognition model is validated based on the test dataset, and the model parameters of the first gesture recognition model are adjusted based on the validation results to obtain the final gesture recognition model.

5. A gesture recognition device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the air gesture recognition method as described in any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the air gesture recognition method as described in any one of claims 1 to 3.

Images