A gesture recognition method based on small samples

Abstract

The present invention discloses a gesture recognition method based on small samples. The method includes: the user customizes the gesture category and collects multiple samples for each gesture category; the collected samples are input into a trained small sample learning model to extract the feature vector corresponding to each sample, and the feature vectors belonging to the same gesture are integrated to obtain the average feature vector corresponding to each gesture as a prototype vector; for the target gesture performed by the user, the corresponding samples are collected and input into the small sample learning model to obtain the feature vector of the target gesture as a query vector; the similarity between the query vector and the prototype vector is calculated, and the gesture category corresponding to the prototype vector with high similarity is selected as the predicted category of the target gesture. The present invention allows users to customize any gesture action, and only requires a very small number of samples to achieve a gesture input solution with high accuracy, strong generalization ability, and strong scalability.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and more particularly, to a gesture recognition method based on small samples. Background Art

[0002] Traditional gesture recognition technology often requires pre-setting a set of gestures when building a system. Experimenters are then recruited to perform each gesture repeatedly to collect a large amount of sensor data to form a training set for training the recognition model (such as a deep neural network). This imposes significant labor costs on the recognition system builders (such as the companies that release the technology). Furthermore, due to the diversity of real-world environments, the environment in which training data is collected can differ significantly from the environment in which the system is actually used. For example, there may be differences in ambient noise, equipment, and personnel. Furthermore, each user desires the freedom to define new gesture sets based on their own habits and preferences (the categories and number of gestures can differ from the pre-defined set). The resulting problem is that the trained model cannot meet diverse real-world needs, and directly using the model will significantly reduce recognition accuracy, indicating that the model's generalization and scalability are weak.

[0003] In the existing technology, gesture recognition solutions mainly include the following categories:

[0004] 1) Extract feature quantities that are independent of environmental factors based on underlying signal processing. For example, the underlying physical quantity of the WiFi signal - Channel State Information (CSI) is used to physically model human behavioral activities, thereby extracting feature quantities that are independent of factors such as the location, direction, and indoor environment of the behavior - body-coordinate velocity profile (BVP). Subsequently, this feature is used as the input of the deep network to identify behavioral actions. This method is only applicable to behavioral perception based on multi-link WiFi connections, has a narrow scope of application, and does not support users to freely define new gesture sets, that is, the number and type of gestures in the training set and the test set need to be consistent.

[0005] 2) Domain transfer methods based on generative adversarial networks. For example, by introducing a generative adversarial network to train feature extractors, gesture recognizers, and domain discriminators, the model can extract essential feature expressions that are independent of the domain, thereby improving the model's cross-domain recognition capabilities. For another example, the image domain style transfer model is borrowed to synthesize samples of the target domain, and the network is updated based on the synthesized samples. Domain transfer methods based on generative adversarial networks require the collection of a large amount of data during the model training phase, and the adversarial model is generally very complex, making it difficult to converge during training and having poor real-time performance when deployed on commercial smart mobile devices. In addition, this type of method does not support users to freely define new gesture sets.

[0006] 3) Methods based on meta-learning models. For example, meta-learning methods can be used to solve cross-domain recognition problems. Alternatively, virtual gesture sample generation mechanisms can be used to automatically generate target domain data to update the model. Currently, these methods do not support users freely defining new gesture sets and require that the gesture categories and numbers used during model training must be consistent with actual user usage. Furthermore, fine-tuning of model parameters is required during actual use, resulting in poor real-time performance when deployed on mobile devices. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a gesture recognition method based on a small sample. The method comprises the following steps:

[0008] The user defines gesture categories and collects multiple samples for each gesture category;

[0009] The collected samples are input into the trained small sample learning model to extract the feature vector corresponding to each sample, and the feature vectors belonging to the same gesture are combined to obtain the average feature vector corresponding to each gesture as the prototype vector;

[0010] For the target gesture performed by the user, a corresponding sample is collected and input into the small sample learning model to obtain a feature vector of the target gesture as a query vector;

[0011] Calculating the similarity between the query vector and the prototype vector, and selecting the gesture category corresponding to the prototype vector with high similarity as the predicted category of the target gesture;

[0012] The small sample learning model includes a feature extraction module and a distance classifier. The prototype vector and the query vector are obtained using the feature extraction module, and the similarity between the query vector and the prototype vector is obtained using the distance classifier.

[0013] Compared with existing technologies, the advantages of this invention lie in that the small-sample gesture recognition method can utilize existing open-source datasets for model training, reducing the labor cost of acquiring training datasets. Furthermore, the trained model can be quickly updated with a small number of samples to adapt to the usage environment. Furthermore, it can meet the user's need to freely define new gestures, thereby improving the model's generalization and scalability.

[0014] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0016] Figure 1 is a flowchart of a gesture recognition method based on a small sample according to an embodiment of the present invention;

[0017] Figure 2 is a general framework diagram of a gesture recognition method based on a small sample according to an embodiment of the present invention;

[0018] Figure 3 is a time-frequency graph corresponding to different gestures according to an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of an example of a small sample learning model according to an embodiment of the present invention;

[0020] Figure 5 is a schematic diagram of the structure of an encoder according to an embodiment of the present invention;

[0021] Figure 6 is a schematic diagram of an adaptive network structure according to an embodiment of the present invention;

[0022] In the figure, Conv-convolutional layer; Block-residual block; Avg-pool-average pooling; FC layer-fully connected layer; Set encoder-set encoder; Para-generator-parameter generator; NN Classifier-NN classifier, MLP-multi-layer perceptron; CNN-convolutional neural network. DETAILED DESCRIPTION

[0023] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0024] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0025] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0026] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0027] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0028] Combine Figure 1 and Figure 2 As shown, the provided gesture recognition method based on small samples includes the following steps:

[0029] Step S110 , collecting two data sets when the user performs gestures.

[0030] In the following, the present invention is described by taking the use of a smart device to collect sound wave samples as an example. The smart device can be various types of devices such as a smart phone, a smart watch, and a smart speaker.

[0031] Specifically, the dataset collection process includes the following steps:

[0032] Step S111 , controlling the built-in speaker of the smart device to transmit a sound wave signal of a specific frequency modulated in a certain modulation mode, and at the same time, controlling the built-in microphone of the smart device to receive an echo signal at a certain sampling frequency.

[0033] For example, the transmission frequency is set to exceed the human hearing range and be within the frequency response range of commercial equipment hardware, which can be set between 19KHz and 22KHz. The sampling frequency satisfies the Nyquist sampling theorem and can be set to 44.1KHz or 48KHz.

[0034] In step S112 , the user performs a user-defined gesture in an area near the smart device at any speed, any size, and at a certain angle relative to the device.

[0035] For example, the azimuth angle between the gesture and the smart device is 0 degrees or other angles. The gesture can be a general gesture or a gesture with a certain meaning, such as writing Arabic numerals or English letters.

[0036] By performing the gesture and collecting the sound wave signal as described above, a data set can be obtained, that is, the data set contains one or more sound wave sequence signals corresponding to the performed gesture.

[0037] Step S113: collect another data set.

[0038] Similarly, another data set may be collected, which differs from the first data set in that the orientation angle formed by the user and the device when performing the gesture is another angle, such as 90 degrees.

[0039] Step S120 , preprocessing the collected data set to construct training data set A and training data set B.

[0040] In this step S120, preprocessing operations are performed on the two collected data sets, including filtering, windowing, short-time Fourier transform, motion detection, time-frequency graph resizing, etc.

[0041] First, filter the data set. A third-order Butterworth notch filter with a band-stop range of [18985, 19015] Hz can be used to remove the 19 kHz center frequency. If the speaker emits a frequency other than 19 kHz, the notch filter parameters can be adjusted accordingly.

[0042] The acoustic signal sequence is then framed by sliding windowing, and each frame is subjected to a Fast Fourier Transform (FFT). The sliding window can be a Hanning window with a width of 8192 samples (corresponding to 0.1858 seconds at a sampling rate of 44.1 kHz) and a step size of 1024 samples. This process converts the one-dimensional acoustic signal sequence into a two-dimensional time-frequency plot.

[0043] Next, if the user performs multiple gestures continuously, each gesture needs to be automatically detected in order to extract a time-frequency graph corresponding to each gesture.

[0044] In one embodiment, the motion detection process involves scanning each frequency bin in each frame of the time-frequency graph from bottom to top (from low to high frequency). If more than four consecutive frequency bins with energy above -80dB are found, the frame is determined to be active and retained. In this way, the frames corresponding to the start and end of the gesture can be found, and the complete time-frequency graph corresponding to the gesture can be extracted.

[0045] Finally, considering that the gesture duration is not constant, the extracted time-frequency graph needs to be resized to a uniform size (such as 224×224) as the input of the subsequent model.

[0046] Figure 3 The figure below shows a time-frequency diagram of different gesture types. The two vertical dashed lines represent the start and end times of a gesture, and the time-frequency diagram between the two dashed lines corresponds to a gesture. Gesture types include numbers (such as 0-9) and letters, such as the 26 letters of the alphabet. Figure 3 (a) corresponds to the number 1, Figure 3 (b) corresponds to the number 3, Figure 3 (c) corresponds to the letter D, Figure 3 (d) corresponds to the letter H.

[0047] Through the above process, two training data sets corresponding to the two data sets can be obtained. For the convenience of description, they are marked as training data set A and training data set B.

[0048] Step 130: Use training dataset A to pre-train a basic classification model.

[0049] A basic classification model is trained using the training data set A. The basic classification model can be a common convolutional neural network, which as a whole includes a feature extraction part and a multi-layer perceptron classifier. Figure 4 As shown in , the base classification model can be a residual network, including multiple residual blocks, each of which can contain two convolutional layers. The process of training the base classification model is called pre-training.

[0050] The basic classification model can be various types of deep learning models. For example, it can be a residual network with different numbers of layers, such as ResNet10, ResNet18, etc., or other types of networks, such as CNN5, ShuffleNet, etc.

[0051] Through pre-training, the initial parameters of the basic classification model can be obtained, and a feature extractor with powerful feature extraction capabilities can be obtained. The feature extractor refers to the part remaining after the basic classification model removes the fully connected layer.

[0052] Step S140: Use the training data set B to train a small sample learning model. The small sample learning model is constructed based on the basic classification model and the training process freezes the parameters of the pre-trained basic classification model.

[0053] Still combined Figure 4 As shown in the figure, compared with the basic classification model, the small sample learning model further adds an encoder, a conversion layer (FiLM) and an adaptive network (or parameter generator, marked as Para.generator). That is, in this small sample learning model, the feature extractor in the basic classification model is transformed.

[0054] exist Figure 4 In, x j Represents an input sample, expressed in the form of a vector; represents the support sample set, the number of which is the product of the number of gesture categories N to be classified and the number of support samples K provided by each gesture, y i Indicates the corresponding gesture category. t Represents the output of the encoder. f pre (x j ) represents the input sample x extracted by the feature extractor in the basic classification model j The corresponding feature vector, or sample representation. is the output of the parameter generator, which is then input into two sets of parameter pairs of the two FiLM layers (so l = 1, 2), respectively. and The parameter pairs output by the FiLM layer are used to stretch and translate the feature vectors of the intermediate layers of the feature extractor, where represents the stretch factor, Represents the translation factor. represents the feature extractor in the few-shot learning model, and accordingly, and Represent the input samples x extracted by the feature extractor in the small sample learning model j The corresponding eigenvector. In this paper, i, j, t, b, l and t′ are all indices.

[0055] In one embodiment, the process of building and training a small sample learning model includes the following steps:

[0056] In step S141, an encoder is added to the front of the basic classification model. The encoder is structured as follows: first, graph convolution is performed on the input data, then the feature map obtained after convolution is reduced in dimension, and finally an encoded vector is obtained.

[0057] The structure of the encoder is as follows Figure 5 As shown in Figure 2, it includes multiple convolutional neural networks, which are used to extract the feature vectors of the corresponding input samples respectively, and after global pooling of each feature vector, output a single encoded representation vector z t .

[0058] Step S142: Add an adaptive network to the front of the basic classification model, take the encoding vector obtained in S131 as input, and output multiple paired vectors.

[0059] The structure of the adaptive network is as follows Figure 6 As shown, the output paired vector is One of the parameter vectors represents the stretch factor, Represents the translation factor. The stretch factor and the translation factor are combined to perform a linear transformation, i.e., stretching and translation, on the feature vectors of the intermediate layers of the few-shot learning model.

[0060] In step S143, all parameters of the pre-trained basic classification model are frozen, and a FiLM layer is added to the model. The parameters of the FiLM layer are the multiple paired vectors output in step S132.

[0061] The FiLM layer is used to perform linear transformation on the result of the convolutional layer in the original basic classification model. Figure 4 As shown in the figure, the FiLM layer acts on the two convolutional layers in each residual block, that is, the two sets of parameter pairs output by the adaptive network Each set of parameter pairs is used as the input of a FiLM layer, and a linear transformation (or affine transformation) is performed on the output of each convolutional layer of the residual block in the FiLM layer, that is, the stretching factor Multiply by the vector output by the convolutional layer, and add The new feature quantity is obtained as the input of the next layer in the residual block.

[0062] In step S144, the original multi-layer perceptron classifier of the pre-trained basic classification model is replaced with a distance classifier to form a modified basic model.

[0063] The distance classifier may be an L1 distance classifier, a Euclidean distance classifier (L2 distance classifier), or other types. The following description will take the L1 distance classifier as an example.

[0064] Step S145: Use the training data set B to train the small sample learning model composed of the encoder, the adaptive network, the FiLM layer and the modified basic model.

[0065] The parameters of the modified basic model have been determined in the pre-training phase and are frozen and not updated in this training phase.

[0066] When training a few-shot learning model, the few-shot learning method is used. First, a small number of samples from a certain category are randomly selected from the training dataset B. Then, different data from the same category are randomly combined into a single task. The average of the feature vectors obtained after the few samples have been run through the few-shot learning model is used as the prototype vector. The feature vectors obtained after the remaining data have been run through the few-shot learning model are used as the query vector. The prototype vector and the query vector are then subjected to an L1 distance classifier to obtain a prediction. The training loss is calculated with the ground-truth label of the query vector to update the model parameters. This training process is called meta-training.

[0067] It should be understood that the basic classification model parameters remain unchanged (frozen) during the meta-training phase. The meta-training phase is mainly to obtain the encoder and adaptive network parameters. Since the training data sets used in the pre-training and meta-training phases are different, the combination of these two phases enables the model to automatically learn how to adapt to different data sets. Compared with removing the training process of the basic classification model and directly training the modified feature extractor + distance classifier, the present invention adds an encoder, an adaptive network and a FiLM layer, so that the model can automatically adjust the features extracted by the feature extractor according to the input sample (perform an affine transformation on the feature vector), thereby adapting to new sample inputs and enhancing its migration ability.

[0068] In addition, by replacing the original multilayer perceptron classifier with a distance classifier, the following problems were solved: the multilayer perceptron classifier itself contains parameters, which need to be fine-tuned with customized gesture samples during the testing phase. Therefore, in order to achieve good results, the number of samples used to fine-tune the classifier needs to be large, which increases the user burden and reduces prediction efficiency; the use of the multilayer perceptron classifier requires that the number of gesture categories in the actual use phase must be consistent with the number of gesture categories in the training phase, that is, users cannot customize new gestures of any number of categories.

[0069] Step S150: predict the user gesture category using the trained small sample learning model.

[0070] After the parameters of the entire small-sample learning model are determined, that is, after the small-sample learning model training is completed, it can be used in the testing process or actual use. Specifically, the following steps are included:

[0071] In step S210, the user first defines the gesture category according to his or her own habits and preferences, and then collects a small amount of samples for each gesture.

[0072] For example, the number of samples corresponding to each gesture category may not exceed 3.

[0073] In step S220, the collected samples are fed into the trained small-sample learning model to extract the feature vector corresponding to each sample. Feature vectors belonging to the same gesture are summed to obtain the average feature vector corresponding to each gesture, known as the prototype vector. This feature extraction process does not use the L1 distance classifier.

[0074] In step S230, when the user performs a customized gesture, the corresponding sound wave sample is collected, and after preprocessing and input into the small sample learning model, a feature vector corresponding to the gesture is obtained, which is called the query vector. The query vector and the prototype vector are passed through the L1 distance classifier to calculate the similarity between the query vector and each prototype vector. The gesture category of the prototype vector with the highest similarity is selected as the category of the query vector, that is, as the predicted category of the user gesture.

[0075] To further validate the effectiveness of our method, we conducted experiments. The results showed that the proposed method achieved recognition accuracy of 69.4%, 79.0%, and 82.4% for 26 new user-defined gestures when the user provided only one, two, and three samples per gesture, respectively. These results significantly outperform existing methods (e.g., "Towards domain-independent complex and fine-grained gesture recognition with RFID. Proceedings of ACMCHI," 4(ISS):1–22, 2020; C. Dian, D. Wang, Q. Zhang, R. Zhao, and Y. Yu.).

[0076] It should be noted that, without departing from the spirit and scope of the present invention, those skilled in the art may make appropriate changes or modifications to the above embodiments. For example, the datasets used for pre-training and meta-training can be open-source datasets. In the data preprocessing stage, the point-stop filter can be replaced with a band-stop filter; the type of window function in windowed framing can be appropriately selected, and the window function parameters can be adjusted; the method for converting a one-dimensional time series signal into a two-dimensional time-frequency graph can be a Fourier transform, a wavelet transform, or a Hilbert-Huang transform, among other time-frequency transform methods. Furthermore, Doppler frequency shift can be used to detect gestures. This is because when a gesture is performed, relative motion occurs between the device and the user, and the magnitude of the frequency shift can be used to determine whether a gesture is present. Furthermore, in addition to gesture recognition using sound waves as the sensing medium, this method can also be applied to gesture recognition using other sensor data, such as WiFi signals and inertial measurement units.

[0077] In summary, compared with the prior art, the present invention has at least the following technical effects:

[0078] 1) The present invention can support users to freely define any category and any number of gestures, without requiring users to interact only according to preset gesture categories when using the system, which significantly improves scalability and user experience.

[0079] 2) The present invention only requires the user to provide a small number of customized samples when using the system in order for the model to achieve high recognition accuracy in the new environment. For example, each gesture has no more than 3 samples, and the total number of samples is 3N, where N is the number of categories of user-defined gestures.

[0080] 3) The present invention does not require fine-tuning of the model during use, thereby ensuring high real-time performance.

[0081] 4) The present invention allows the use of open source datasets in the model pre-training or meta-training stage to fully utilize existing datasets without recruiting personnel to collect them, thereby reducing the cost of the model training stage.

[0082] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0083] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0084] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0085] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, Python, and conventional procedural programming languages such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0086] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0088] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0089] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0090] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A gesture recognition method based on a small sample size, comprising the following steps: The user defines gesture categories and collects multiple samples for each gesture category; The collected samples are input into the trained small sample learning model to extract the feature vector corresponding to each sample, and the feature vectors belonging to the same gesture are combined to obtain the average feature vector corresponding to each gesture as the prototype vector; For the target gesture performed by the user, a corresponding sample is collected and input into the small sample learning model to obtain a feature vector of the target gesture as a query vector; Calculating the similarity between the query vector and the prototype vector, and selecting the gesture category corresponding to the prototype vector with high similarity as the predicted category of the target gesture; The small sample learning model includes a feature extractor and a distance classifier, the prototype vector and the query vector are obtained by using the feature extractor, and the similarity between the query vector and the prototype vector is calculated by using the distance classifier; The small sample learning model is trained according to the following steps: Pre-training a basic classification model using a first training data set, the first training data set reflecting a correspondence between samples collected when a user performs gestures and gesture categories, the basic classification model comprising a first feature extraction module and a multi-layer perceptron classifier; Using the second training data set to train the small sample learning model, the training process freezes the pre-trained basic classification model; The small sample learning model is constructed according to the following steps: An encoder, an adaptive network, and a conversion layer are added to the pre-trained basic classification model. The encoder is used to encode the input data to obtain a coding vector; the adaptive network takes the coding vector as input and outputs a parameter vector representing the linear transformation; the conversion layer performs a linear transformation on the convolutional layer results in the basic classification model based on the parameter vector; The multi-layer perceptron classifier of the basic classification model is replaced with a distance classifier to construct the small sample learning model.

2. The method according to claim 1, characterized in that In the process of training the small sample learning model, a small number of samples of some categories are randomly selected from the second training data set, and then different data of the same category are randomly selected to form a task; the average value of the feature vectors obtained after the small number of samples pass through the small sample learning model is used as the prototype vector, and the feature vectors obtained after other data pass through the small sample learning model are used as the query vector, and then the prototype vector and the query vector are passed through the distance classifier to obtain the category prediction result, and the training loss is calculated with the true label of the query vector.

3. The method according to claim 1, characterized in that The distance classifier is an L1 distance classifier or an L2 distance classifier.

4. The method according to claim 1, wherein The first training data set and the second training data set are obtained by the following steps: Control the built-in speaker of the smart device to transmit a sound wave signal of a specific frequency modulated according to a certain modulation method, and control the built-in microphone of the smart device to receive the echo signal at a certain sampling frequency; The user performs a user-defined gesture in a vicinity of the smart device at any speed and size, at a first orientation angle relative to the smart device, to collect a first data set; The user performs a gesture forming a second azimuth angle with the smart device to collect a second data set, wherein the first azimuth angle and the second azimuth angle are different, and the first data set and the second data set each include a plurality of collected one-dimensional acoustic signal samples; A preprocessing operation is performed on the first data set and the second data set to convert the one-dimensional acoustic signal sequence into a two-dimensional time-frequency graph, which is then constructed into a first training data set and a second training data set, wherein the first training data set and the second training data set reflect the correspondence between the two-dimensional time-frequency graph and the gesture category, and the two-dimensional time-frequency graph reflects the time-frequency features between the start frame and the end frame of the gesture.

5. The method according to claim 4, characterized in that When the user performs multiple gestures continuously, each gesture is detected according to the following steps to extract a two-dimensional time-frequency graph corresponding to each gesture: For each frame in the time-frequency graph, each frequency slot is scanned from low to high. When there are more than a set threshold of consecutive frequency slots with energy higher than the set energy threshold, the frame is determined to be an active frame, and then the gesture start frame and end frame are found to extract the two-dimensional time-frequency graph corresponding to different gestures.

6. The method according to claim 1, characterized in that The encoder is used to perform graph convolution on the input data, and then reduce the dimension of the feature map obtained after the convolution to obtain an encoded vector.

7. The method according to claim 1, characterized in that The basic classification model includes multiple residual blocks, each residual block contains multiple convolutional layers, the encoding vector output by the encoder is passed to the adaptive network, and the parameter vector output by the adaptive network includes a stretching factor and a translation factor and is provided to the conversion layer.

8. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer device comprising a memory and a processor, wherein a computer program capable of being executed on the processor is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images