Small sample acoustic gesture recognition method based on cross-modal learning

Abstract

The application discloses a small sample acoustic wave perception gesture recognition method based on cross-modal learning. The method comprises the following steps: acquiring a time-frequency graph of a user-defined gesture, inputting a trained recognition model, and generating a prototype vector of the user-defined gesture category; detecting an interactive action gesture and acquiring a time-frequency graph, inputting the time-frequency graph into the recognition model, and generating a query vector of the interactive action gesture; and comparing the prototype vector and the query vector according to a cosine similarity to identify the category of the interactive action gesture. The recognition model is a cross-modal small sample learning model trained based on a support set and a query set, the support set is obtained by randomly selecting part of samples of a category from a public data set, the query set is obtained by randomly selecting different data in the same category from the public data set, and the time-frequency graph is an acoustic wave perception time-frequency graph. The application allows the training data set and the test data set to be different modalities, significantly reduces the data acquisition cost, and can achieve excellent performance with only a small amount of samples.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and more specifically, to a few-sample acoustic wave perception gesture recognition method based on cross-modal learning. Background Technology

[0002] With the development of sensors and the widespread adoption of smart technology, researchers frequently utilize sensors in devices such as smartphones, smart headphones, and smartwatches to conduct research aimed at identifying user activities and meeting user needs. Existing user activity recognition systems typically rely on machine learning models, which require training on large-scale labeled sensor data. Collecting large amounts of labeled sensor data is time-consuming and necessitates manual annotation, hindering rapid model deployment and preventing the timely application of new sensors.

[0003] Currently, there are many technologies for sound wave sensing, but they typically require pre-defined gesture sets and the recruitment of experimenters to collect large datasets of labeled sensor data before training deep learning network models. Because training neural networks requires massive amounts of data, and users need to follow pre-defined gestures to use the gesture recognition system, this not only consumes significant time and manpower but also prevents users from freely defining gestures according to their preferences and habits. This hinders researchers from quickly deploying systems; researchers must first collect large amounts of data before they can study the sensor's applications.

[0004] In existing technologies, research on cross-modal gesture recognition is limited. In 2020, Hyeokhien Kwon et al. proposed using video modeling of human activity to create virtual streams of inertial measurement unit (IMU) data, representing acceleration measurements at various human positions. This approach utilizes a large amount of existing video data of human activity to generate virtual IMU data for training a sensor-based activity recognizer, achieving good recognition performance. In 2021, Sejal Bhalla et al. proposed using IMU data to cross-modally assist in training machine learning models based on Doppler data converted from millimeter-wave radar. This approach uses a machine learning model to learn the feature mapping between sensors worn on the user's wrist and surrounding millimeter-wave radar sensors. This allows for the construction of an activity recognition system without requiring a large amount of millimeter-wave radar data; a system achieving 70% accuracy across 10 activity categories can be achieved using only 14 seconds of labeled Doppler data. In 2022, Zou Yongpan et al. proposed a gesture recognition method based on a small sample size. In this method, a decent recognition performance can be achieved by utilizing existing homomodal datasets and providing multiple samples for each category in the target data.

[0005] Analysis reveals that existing gesture recognition solutions have the following main drawbacks:

[0006] 1) The categories of the selected open-source dataset need to be basically consistent with the categories that the sensors across are to identify. Therefore, it still requires a lot of manpower and time to search for relevant open-source data for each identification category.

[0007] 2) A small amount of data from the sensors involved needs to be collected in advance to assist in training or fine-tuning the model.

[0008] 3) It is necessary to find a dataset with the same modality to train the model. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a few-sample acoustic wave perception gesture recognition method based on cross-modal learning. This method includes the following steps:

[0010] Obtain the time-frequency graph corresponding to the user's custom gesture, input it into the trained recognition model, and generate the prototype vector of the custom gesture category;

[0011] During the user's gesture interaction with the smart device, the interaction action is detected and the corresponding time-frequency map is obtained, which is then input into the recognition model to generate a query vector of the interaction action;

[0012] The prototype vector and the query vector are compared based on cosine similarity to identify the gesture category of the interactive action;

[0013] The recognition model is a cross-modal few-shot learning model trained on a support set and a query set. The support set is obtained by randomly selecting samples of some categories from a public dataset, and the query set is obtained by randomly selecting different data of the same category from a public dataset. The time-frequency map is a time-frequency map of sound wave perception, and the public dataset contains handwritten character and number images.

[0014] Compared with existing technologies, the advantages of this invention lie in proposing a cross-modal acoustic wave sensing gesture recognition system. Considering the abundance of publicly available datasets, this system can select and utilize these datasets based on the characteristics of the data collected by acoustic wave sensors, directly training the model using these datasets without requiring acoustic wave sensing gesture data during model training. At the user level, users can input one or more custom gesture samples through smart devices equipped with this system, such as smartphones. The feature vectors obtained after the samples are processed by the model are averaged and saved as the prototype vector in the system for final gesture recognition. This achieves the goal of gesture recognition without requiring acoustic wave sensing gesture data during model training.

[0015] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0017] Figure 1 This is a schematic diagram of the overall process of a few-sample acoustic wave perception gesture recognition method based on cross-modal learning according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of a few-sample acoustic wave perception gesture recognition method based on cross-modal learning according to an embodiment of the present invention;

[0019] Figure 3 This is a flowchart of open-source dataset processing according to an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of a cross-modal few-shot learning model according to an embodiment of the present invention;

[0021] Figure 5 This is a flowchart of acoustic sensing dataset processing according to an embodiment of the present invention.

[0022] Figure 6 This is a time-frequency diagram generated according to an embodiment of the present invention;

[0023] Figure 7 This is a schematic diagram illustrating the process of identifying user gesture categories according to an embodiment of the present invention;

[0024] Figure 8 This is a schematic diagram of a confusion matrix for 10 types of digit recognition according to an embodiment of the present invention;

[0025] Figure 9 This is a schematic diagram of a confusion matrix for recognizing 26 types of letter gestures according to an embodiment of the present invention. Detailed Implementation

[0026] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0028] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0029] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] The purpose of this invention is to train a sound wave perception gesture recognition model using existing open-source datasets of different modalities, thereby reducing the time and labor costs associated with collecting sound wave data; to train the model using open-source datasets to meet the need for rapid deployment of the model to smart terminals; and to enable the trained model to achieve excellent gesture recognition performance with a small number of samples and without the need for fine-tuning.

[0032] The few-sample acoustic wave perception gesture recognition method based on cross-modal learning provided by this invention can be implemented on commonly used smart devices, such as smartphones, smart headphones, or wearable devices. Its overall application process is as follows: Figure 1 As shown, the system involves three parts: model training, smart devices, and device software. It mainly includes: training a cross-modal few-shot learning model using an open-source dataset and importing the model into the software; then, users collect custom gestures as samples using the speaker and microphone on the smart device. The device software implements key functions such as data processing, generating gesture feature vectors, saving prototype vectors, and recognizing gesture categories.

[0033] Specifically, see Figure 2 As shown, the provided few-sample acoustic wave perception gesture recognition method based on cross-modal learning includes the following steps:

[0034] Step S210: Select a public dataset for preprocessing and data augmentation.

[0035] Figure 3 This is a flowchart for processing open-source datasets (i.e., public datasets). For the subsequent time-frequency maps of gesture data obtained based on sound wave perception, considering that the biggest difference between the various gesture time-frequency maps lies in their shape, the MINIST handwritten digit dataset was selected as the dataset for model training.

[0036] To reduce the difference between the gesture time-frequency image and the MINIST image, and to facilitate model generalization to gesture time-frequency image data, firstly, unlike previous models that only treated MINIST images as black and white (0 and 1), this model reads the MINIST dataset in color according to a set color bar, while reading the background as black. Secondly, considering that the recognition regions on the gesture time-frequency image may be distributed to the left or right and that noise may occur, noise is randomly added to the read MINIST dataset, and black fill is randomly applied to the left or right sides of the image. It should be understood that adding noise makes the model more robust to recognition, while black fill simulates the left or right deviation of gesture data during custom gesture processing. Finally, since the main purpose of training the model is to enable it to classify based on the shape comparison of content in the image, the contrast and hue of the image are randomly varied to make the model more inclined to learn the shape of the recognition regions in the image.

[0037] It should be noted that the publicly available datasets used in this invention are not limited to the MINIST dataset; datasets such as the Omniglot dataset and the EMINIST dataset can also be used. These datasets contain images of handwritten characters and numbers, which helps the subsequently trained model to focus more on the shape of the image content. Furthermore, other preprocessing methods can also be used. For example, the image can be converted to grayscale, i.e., from three channels to a single channel, to save computational resources.

[0038] Step S220: Construct a cross-modal few-shot learning model and train it with the set loss function as the optimization objective.

[0039] This invention obtains a gesture recognition model by training a cross-modal few-shot learning model. Figure 4 This is a schematic diagram of a cross-modal few-shot learning model, which mainly includes a feature extractor module, a global feature extraction module, a local feature extraction module, a global similarity comparison module, and a local similarity comparison module.

[0040] exist Figure 4 In this embodiment, a prototype network few-shot learning model is trained using an open-source dataset. During data input into the model, the data is divided into a support set and a query set. The model's feature extractor can be a simple convolutional neural network, such as a four-layer convolutional network. This shallow network is used to avoid the network learning overly deep feature distributions from the open-source dataset. To effectively transfer the model trained on the open-source dataset to the acoustic wave perception gesture dataset from only a few samples, the model utilizes globally invariant features and local discriminative details of the images. This allows it to learn both the global and local distributions of the open-source dataset, thereby improving the model's cross-modal recognition accuracy.

[0041] Specifically, after the feature extractor extracts global features, it further explores more recognizable local features under the supervision of the global features. For the global features extracted by the feature extractor, they first pass through a global feature extraction module (e.g., containing two convolutional blocks) to learn general knowledge of global representations in the image. Then, a cosine similarity function is used to measure the similarity between the query set and the support set. For the local feature extraction module, an attention mechanism is employed. First, the number of channels in the global features is reduced in dimensionality using a convolutional block. Then, the vector obtained through an adaptive average pooling layer is multiplied by the original dimensionality-reduced matrix to enhance the feature channels that need strengthening, forming a weight matrix. Finally, the weight matrix is ​​multiplied row-by-row by the global feature matrix to obtain the local features.

[0042] During model training, a meta-training method based on few-shot learning was employed. First, a small number of samples from a subset of classes were randomly selected from a chosen public dataset as the support set. Then, different data points within the same class were randomly selected as the query set, combining the two into a single task. The support set was processed by the model to obtain global and local feature vectors. The average of the support sets within the same class was used as the global and local prototype vectors. Similarly, the query set, after processing, also yielded corresponding feature vectors as the global and local query vectors. The global prototype vector and global query vector were compared using a global similarity comparison module, and the local prototype vector and local query vector were compared using a local similarity comparison module to obtain corresponding measurement results (e.g., using cosine similarity distance). Finally, the measurement results and the true labels of the query vectors were processed using the cross-entropy loss function to calculate the total training loss, which was then used for backpropagation to update the model parameters.

[0043] It should be noted that this paper does not impose restrictions on the specific structure of the cross-modal few-shot learning model. For example, the number of convolutional layers and the size of the convolutional kernels can be set appropriately.

[0044] Step S230: For user-defined gestures, collect gesture data based on sound wave perception, process it into a time-frequency graph, and use a trained cross-modal few-shot learning model to generate prototype vectors corresponding to the user-defined gestures.

[0045] Figure 5 This describes the processing flow of the acoustic wave sensing dataset. First, a smart device, such as a smartphone, is controlled to emit acoustic wave signals at a modulated specific frequency through its built-in speaker. Considering that the selected emission frequency must be imperceptible to the user and unaffected by the surrounding environment to facilitate filtering and noise reduction, the emission frequency is set to 19kHz. Since the sampling frequency must satisfy the Nyquist sampling theorem, the sampling rate is set to 44.1kHz.

[0046] Next, the user performs personalized predefined gestures in any size, speed, and orientation in the area near the speaker of the smart device.

[0047] Then, the predefined gesture data is preprocessed, including filtering, windowing, short-time Fourier transform, Gaussian filtering, threshold filtering, and time-frequency graph resizing. Since this process involves user-defined gestures, motion detection is not required. Specifically, first, the dataset is filtered. A third-order Butterworth point-stop filter with a band-stop range of [18985, 19015] Hz is used, primarily to remove the 19 kHz center frequency. Furthermore, considering that the gesture data is distributed within a frequency range, bandpass filtering is also performed, with a bandpass range of [18700, 19300] Hz. Next, the acquired sound signal sequence is processed by sliding windowing to frame it, and a short-time Fourier transform is performed on each frame. The sliding window type is set to Hanning window, with a width of 8192 sample points and a step value of 1024 sample points. Through these operations, the one-dimensional sound signal sequence is transformed into a two-dimensional time-frequency graph. The time-frequency graph data is then processed. In order to separate the identified data from the background of the time-frequency graph, a Gaussian filter and a threshold filter are performed. Here, the Gaussian convolution kernel is 5, the standard deviation is 1, and the threshold is set to 0.72.

[0048] Furthermore, considering that the time it takes for a user to complete each gesture is not fixed, it is also necessary to standardize the size of the time-frequency graph (e.g., 84×84) to facilitate input into the model. Figure 6 This is the generated time-frequency plot. After inputting a custom gesture into a trained cross-modal few-shot learning model, the model generates a prototype vector for the custom gesture data.

[0049] Finally, to improve the performance of the model recognition, considering that the speed of users making gestures may vary, the gesture samples provided by users were augmented, turning one gesture sample into multiple gesture samples. The sample data augmentation methods used included horizontal translation and scaling. Blank areas that appeared during the processing were directly filled with black (0,0,0).

[0050] It should be noted that, in the preprocessing stage of acoustic wave sensing data, the parameters of the sliding window and the type of the window function in windowing and framing can be adjusted appropriately. Furthermore, methods such as wavelet transform and Fourier transform can be used to convert one-dimensional time-series signals into two-dimensional time-frequency graphs. Gaussian filtering parameters and filtering methods can be adjusted appropriately; median filtering or mean filtering can also be used. The threshold for threshold filtering can be adjusted as needed.

[0051] Step S240: During the user's gesture interaction with the smart device, a query vector corresponding to the interactive gesture is generated using a trained cross-modal few-shot learning model, and then the gesture recognition result is obtained by comparing it with the stored prototype vector.

[0052] The process of identifying user gesture categories is as follows Figure 7 As shown. After the user customizes the gesture, the prototype vector of each custom gesture data has been saved (the prototype vector includes a global prototype vector and a local prototype vector). In the actual gesture recognition scenario, the user only needs to perform the gesture action in the direction of the previously completed gesture in front of the speaker near the smart device. After the gesture data preprocessing process described in step S230, a time-frequency map of the gesture that the user wants to recognize is generated. After being trained by the cross-modal few-shot learning model, a query vector is generated (the query vector includes a global query vector and a local query vector). The cosine similarity distance is calculated with all the prototype vectors of the previously saved custom gesture data. Then, the probability of belonging to each gesture category is calculated using the Softmax function. Finally, the probability predicted by the global similarity comparison module and the probability predicted by the local similarity comparison module are summed, and the category with the highest confidence is taken as a predicted category of the gesture to be recognized. After the time-frequency map is generated, since the user's gesture is an impromptu process, in order to prevent the system from recognizing gestures at all times, an action detection process needs to be added. The action detection process is as follows:

[0053] Considering that the background color of the time-frequency graph after data preprocessing has become black, i.e., the pixel value is (0,0,0), and the non-black part is basically the data of the gesture action, and considering the speed at which a person makes a gesture, when five consecutive frames, i.e., a window of about 0.1 seconds, all show data with a threshold greater than 0, the first frame is determined to be the start frame of the activity. After the activity frame is detected, if five consecutive frames do not show data with a threshold greater than 0, the first frame is determined to be the end frame of the activity. The start frame and the end frame are pushed forward and backward by five frames respectively, and this part is extracted as the complete time-frequency graph corresponding to the gesture.

[0054] It should be noted that motion detection can also be performed by scanning the time-frequency map from bottom to top. That is, if more than four consecutive color block values ​​are found to be greater than 0, the active frame is determined to be an active frame, and the same applies to the end frame.

[0055] To further verify the feasibility of this invention, experimental results were conducted. The results show that when users provided 1, 3, or 5 samples for each custom gesture (for convenience, 10 categories of numeric gestures and 26 categories of alphabetic gestures were used), the proposed method achieved recognition accuracies of 70.45%, 78.32%, and 81.14% for the 10 numeric gestures, respectively, and 60.99%, 68.99%, and 72.39% for the 26 alphabetic gestures, respectively. The top-3 accuracies were 83.68%, 88.78%, and 90.91%, respectively. Figure 8 and Figure 9 The confusion matrix diagrams for 10 types of digit gesture recognition and 26 types of letter gesture recognition are shown respectively under 3 samples.

[0056] It should be understood that the method proposed in this invention, in addition to using sound waves as a cross-modal object for gesture recognition, can also be applied to other sensor data, such as millimeter-wave radar sensors, which have been extensively explored recently, as long as the differences between different categories of data collected by the sensor can be distinguished by the shape of the area to be identified.

[0057] In summary, compared with the prior art, the present invention has the following advantages:

[0058] 1) This invention proposes a cross-modal small-sample acoustic wave perception gesture recognition scheme. This scheme allows the model to use open-source datasets of different modalities during the training phase without requiring the dataset to be of the same category as the dataset of the cross-modality, thereby reducing the human and time costs of data collection and enabling the model to be deployed quickly.

[0059] 2) This invention addresses the cross-modal problem by transforming and augmenting the open-source dataset according to the characteristics of the cross-modal data, and introducing local features, enabling the model to focus on both global invariance and local details. By learning the distribution of these two types of features, the generalization performance of the model is improved.

[0060] 3) Based on models trained on open-source datasets of different modalities, this invention allows users to freely define their own gestures without limiting the number of gestures. Furthermore, users are not required to interact according to predefined gesture categories during use, thus improving the user experience.

[0061] 4) This invention does not require the introduction of data from multiple sensors or fine-tuning during training. The open-source dataset can be directly processed and input into the model for training, allowing the model to use data from multiple sensors. This reduces the burden on users and achieves a high accuracy rate in gesture recognition.

[0062] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0063] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0064] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0065] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0066] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0068] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0070] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A few-sample acoustic wave-sensing gesture recognition method based on cross-modal learning, comprising the following steps: Obtain the time-frequency graph corresponding to the user's custom gesture, input it into the trained recognition model, and generate the prototype vector of the custom gesture category; During the user's gesture interaction with the smart device, the interaction action is detected and the corresponding time-frequency map is obtained, which is then input into the recognition model to generate a query vector of the interaction action; The prototype vector and the query vector are compared based on cosine similarity to identify the gesture category of the interactive action; The recognition model is a cross-modal few-shot learning model trained on a support set and a query set. The support set is obtained by randomly selecting samples of some categories from a public dataset, and the query set is obtained by randomly selecting different data of the same category from a public dataset. The time-frequency graph is a time-frequency graph of sound wave perception, and the public dataset contains handwritten character and number images. The cross-modal few-shot learning model includes a feature extractor module, a local feature extraction module, a global feature extraction module, a global similarity comparison module, and a local similarity comparison module. The feature extractor module extracts features based on the input support set and query set, and passes them to the local feature extraction module and the global feature extraction module, respectively. The global feature extraction module outputs a global prototype vector and a global query vector, and the local feature extraction module outputs a local prototype vector and a local query vector. The global similarity comparison module is used to measure the similarity between the global prototype vector and the global query vector, and the local similarity comparison module is used to compare the similarity between the local prototype vector and the local query vector. The cross-modal few-shot learning model is trained according to the following steps: Randomly select samples from a subset of categories in the chosen public dataset as the support set, and then randomly select different data from the same category as the query set. The two are combined into one task. The support set is processed by a cross-modal few-shot learning model to obtain global and local feature vectors. The average value of the support sets of the same class is taken as the global prototype vector and local prototype vector. The query set is processed through a cross-modal few-shot learning model to obtain global query vectors and local query vectors; The cosine similarity function is used to obtain the measurement results between the global prototype vector and the global query vector, as well as between the local prototype vector and the local query vector. The total training loss is then calculated using the cross-entropy loss function and the true label of the query vector to backpropagate and update the parameters of the cross-modal few-shot learning model.

2. The method according to claim 1, characterized in that, The feature extractor module is a multi-layer convolutional network. The global feature extraction module contains multiple convolutional blocks. The global similarity comparison module and the local similarity comparison module use a cosine similarity function for measurement. The local feature extraction module extracts local features based on a channel attention mechanism.

3. The method according to claim 1, characterized in that, The time-frequency diagram is obtained according to the following steps: When a user makes a gesture, the smart device's speaker emits sound waves at a set frequency, and the smart device's microphone collects signals at a set sampling rate. The acquired signal is subjected to point resistance filtering, windowing and framing, and time-frequency transformation to generate a two-dimensional time-frequency graph. The two-dimensional time-frequency graph is subjected to Gaussian filtering, threshold filtering, and data augmentation processing, wherein data augmentation includes horizontal translation and scaling, and blank areas appearing during the processing are filled with black.

4. The method according to claim 1, characterized in that, The publicly available dataset is a preprocessed dataset, and the preprocessing process includes: Choose a public dataset that contains images of handwritten digits; The selected public dataset is read in color for the numeric areas and in black for the background, according to the set color bars. The publicly available dataset is randomly noiseed and randomly filled with black on the left or right side of the image. The contrast and hue of the image are also randomly transformed.

5. The method according to claim 1, characterized in that, The detection interaction actions include: For the acquired signal, when a set number of consecutive frames show data with a threshold greater than 0, the first detected frame is determined to be the start frame of the action. After the start frame is detected, if a set number of consecutive frames do not show data with a threshold greater than 0, the first frame is determined to be the end frame of the action. The start frame and the end frame are pushed forward and backward by several frames respectively to extract the complete time-frequency map corresponding to the gesture.

6. The method according to claim 1, characterized in that, The smart device is a smartphone, smart headphones, or wearable device.

7. A computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images