Quasi-incremental gesture recognition method, electronic device and medium

Abstract

The present invention relates to a quasi-incremental gesture recognition method, electronic device, and medium. The method comprises: receiving input data; extracting intermediate features based on the input data and constructing task-specific index embeddings, each embedding corresponding to a gesture recognition task; calculating the similarity between the intermediate features and the index embeddings to generate combination weights; dynamically aggregating isolated parameters of multiple tasks using the weights to construct a final feature for the current task; inputting the final feature into a classifier to predict gesture categories; training an adapter module based on the prediction results; enhancing the task index embeddings and model parameters by calculating historical category prototypes and generating pseudo samples; combining the enhanced results to form a prototype-enhanced hybrid regularization loss to further optimize the model; and after completing training for all tasks, using the optimized model to classify newly input gesture data. This method is applicable to gesture recognition tasks with confusing characteristics and endows continuous learning based on parameter isolation with adaptive combination capabilities.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a quasi-incremental gesture recognition method, electronic device, and medium. Background Art

[0002] Nonverbal communication plays a crucial role in daily human interactions, accounting for over 65% of all communication (Mohamed, Mustafa, and Jomhari, 2021). Among the various forms of nonverbal communication, gestures have attracted considerable attention due to their unique intuitiveness and naturalness. Humans instinctively use gestures to communicate, perceive, and interact with their environments. With the rapid development of virtual reality (VR) and augmented reality (AR) technologies, gesture recognition, as an important bridge connecting the physical and virtual worlds, has played a key role in promoting core immersive and realistic experiences in the metaverse (Duan et al., 2023). In particular, gesture recognition technology has become an indispensable method of interaction in AR / VR wearable devices such as Meta Quest and VUZIX.

[0003] In recent years, thanks to the rapid development of deep neural networks, gesture recognition technology has achieved recognition accuracy close to 96% on standard datasets (Min et al., 2020), providing a highly immersive user experience in real virtual environments. However, current mainstream gesture recognition methods typically focus on improving test accuracy on a specific dataset through a single centralized training session. The limitations of this approach are obvious: in real-world applications, gesture recognition systems need to be able to continuously learn new knowledge and dynamically adapt to the needs of different users. For example, when gesture recognition systems are deployed on VR edge devices, users may gradually register new gesture categories based on their needs. Using traditional static methods to train samples containing new categories often leads to catastrophic forgetting of the system's existing knowledge. Furthermore, the acquisition of samples from previous categories in gesture recognition tasks is often restricted by privacy protection, making direct access to historical data impossible. Therefore, building a continuous learning model that supports the sequential registration of new gesture categories over the lifecycle, without playback data, has become a major challenge in the AR / VR field.

[0004] Continuous learning (CL) technology has been proposed to address the problem of catastrophic forgetting and has made significant progress in recognition tasks involving images, audio, and text (Qu et al., 2021; Biesialska et al., 2020; Parisi et al., 2019). However, research on class-incremental hand gesture recognition (CI-HGR) remains limited. Existing continuous learning methods can be roughly divided into the following three categories:

[0005] 1. Regularization-based methods: These methods retain knowledge from previous tasks by introducing explicit regularization terms (Li and Hoiem, 2017; Kirkpatrick et al., 2017). These methods are good at preventing catastrophic forgetting, but they often suffer from low efficiency and limited performance.

[0006] 2. Replay-based methods: Alleviate the forgetting problem by storing old class samples or generating pseudo samples (Rebuffi et al., 2017; Rolnick et al., 2019). However, this method may involve privacy issues, and the generated pseudo samples may introduce historical model bias, affecting model performance.

[0007] 3. Parameter isolation-based methods: Independent parameters are constructed for each task to avoid interference (Madotto et al., 2021). Although this method performs well in task-incremental learning (TIL), its performance is limited in class-incremental learning (CIL) due to the unknown test task.

[0008] In skeleton-based action recognition tasks, continuous learning methods have been initially applied due to their high robustness and compactness. For example, some studies have achieved incremental learning of tasks through dynamic network expansion or pseudo-sample replay techniques (Aich et al., 2023). However, these methods either fail to meet privacy requirements or are inferior to parameter isolation techniques in terms of time and memory efficiency. Summary of the Invention

[0009] The present invention provides a quasi-incremental gesture recognition method, electronic device and medium, which aims to solve the problems of existing gesture recognition systems being prone to catastrophic forgetting, having difficulty in dynamically adapting to new gesture categories, and confusion in recognition between gestures when there is no playback data.

[0010] To achieve the above object, the first aspect of the present invention provides a quasi-incremental gesture recognition method, comprising the following steps:

[0011] receiving input data representing a hand gesture, wherein the input data is three-dimensional skeleton data including a time series, each frame of the three-dimensional skeleton data including spatial coordinates of a plurality of hand joints;

[0012] extracting intermediate features according to the received input data, and constructing task-specific index embeddings based on the intermediate features, wherein each index embedding is associated with a specific gesture recognition task;

[0013] Calculate the similarity between the extracted intermediate features and the task-specific index embedding to generate the combination weight;

[0014] Use the generated combined weights to dynamically aggregate the isolated parameters of multiple tasks to construct the final features for the current task;

[0015] The constructed final features are input into the classifier to perform gesture category prediction;

[0016] Based on the prediction results, the adapter module is trained for each task;

[0017] During the adapter module training process, prototypes of historical gesture categories are calculated and pseudo-prototype samples are generated. The pseudo-prototype samples are used to embed task-specific indexes and enhance model parameters.

[0018] Combine index embedding enhancement and model parameter enhancement to form an overall prototype enhancement hybrid regularization loss and continue to tune the model;

[0019] After completing all task training, the adjusted model is used to classify and recognize new input gesture data.

[0020] Furthermore, the method of constructing task-specific index embedding based on the intermediate features includes:

[0021] Extracting three-dimensional spatial coordinate features of the hand joints of each frame from the input data, wherein the three-dimensional spatial coordinate features include the position of each hand joint in the three-dimensional space;

[0022] Extracting time series features from the three-dimensional spatial coordinate features using a time series modeling method to generate time series features representing dynamic changes in gestures;

[0023] The temporal features and the three-dimensional spatial features are fused through a feature fusion network to generate intermediate features of the gesture;

[0024] Based on the intermediate features, generating a task-specific index embedding vector associated with the current task, wherein the task-specific index embedding vector is constructed by a task-specific trainable module;

[0025] The task-specific index embedding vector is dynamically combined with the intermediate features to form a task-specific index embedding for subsequent task learning.

[0026] Furthermore, the similarity between the extracted intermediate features and the task-specific index embedding is calculated, and the method of generating the combination weight includes:

[0027] Calculate the cosine similarity between the generated intermediate features and each task-specific index embedding to obtain a similarity score;

[0028] Compute the above similarity scores for all task-specific index embeddings;

[0029] Divide each similarity score by a predefined scaling factor to obtain a scaled similarity score;

[0030] Apply an exponential function to all scaled similarity scores to obtain the unnormalized value of the weight;

[0031] Normalize all unnormalized weight values ​​and calculate the combined weight of each task-specific index embedding vector, which is calculated as follows:

[0032]

[0033] Among them, W t is the combined weight, exp is the exponential function, cos(f,K t ) is the intermediate feature representation f and the task-specific index embedding vector K t The cosine similarity between them, K is the index embedding pool, which contains the set of all task-specific index embedding vectors, K j is the fth task-specific index embedding vector in the set K, f represents the intermediate feature vector extracted from the input gesture data by the feature extractor, τ is the scaling factor, For similarity.

[0034] Furthermore, the dynamic aggregation includes weighted combination of the isolation parameters of the multiple tasks according to the combination weight, specifically including:

[0035] applying the combined weights to the isolated parameters of each relevant task;

[0036] Apply the weighted isolation parameters to the extracted intermediate features one by one;

[0037] The features after each task are accumulated to obtain the final features for the current task.

[0038] Furthermore, gesture category prediction includes the following specific steps:

[0039] The constructed final features are input into the pre-trained classifier module;

[0040] In the classifier module, the final features are linearly transformed using the fully connected layer;

[0041] By applying the Softmax function, the output of the linear transformation is converted into the probability distribution of each gesture category;

[0042] According to the probability distribution, the category with the highest probability is selected as the final gesture recognition result.

[0043] Furthermore, during the adapter module training process, the method of calculating the prototype of the historical gesture category and generating the prototype pseudo sample includes:

[0044] Extract features belonging to historical gesture categories through the feature extractor of the gesture recognition model;

[0045] According to the gesture samples belonging to the same category, the prototype and covariance matrix of the category are calculated. The specific formula is as follows:

[0046]

[0047] Among them, μ c is the prototype of category c, N c is the number of samples of category c, y i =c represents sample x i The label belongs to category c, Indicates input gesture x i Through feature extractor The intermediate features obtained later; is the covariance matrix of category c, T is the transpose operator;

[0048] Based on the calculated prototype and covariance matrix, a prototype pseudo sample is generated for each historical gesture category. The prototype pseudo sample follows a multivariate Gaussian distribution:

[0049]

[0050] in, is the generated prototype pseudo sample feature vector, representing the virtual sample of category c, is a multivariate Gaussian distribution with mean μ c , the covariance matrix is c∈{C K \C U} is category c that belongs to the learned historical category set C K Subtract the new category set C introduced by the current task U .

[0051] Further methods to enhance task-specific index embeddings using prototype pseudo samples include:

[0052] A metric learning strategy is used to treat prototype pseudo samples as negative samples, and compared them with the intermediate features of the current task to optimize indexable embeddings.

[0053] The task-specific index embedding is updated by maximizing the similarity between the original sample features and the correct task embedding and minimizing its similarity with the pseudo sample features. The task-specific index embedding is optimized using the following loss function:

[0054]

[0055] in, It is a prototype-driven embedding enhancement loss function used to measure and optimize task-specific index embeddings. m is a boundary parameter used to control the minimum distance between different categories. Prototype pseudo sample The cosine similarity between the task-specific index embedding K, cos(f,K) is the current gesture feature The cosine similarity between the embedding K and the task-specific index.

[0056] Furthermore, methods for enhancing model parameters using prototype pseudo samples include:

[0057] Based on the prototype pseudo samples of the historical task category and the characteristics of the training samples of the current task, the pseudo samples are used as an intermediary to transfer knowledge;

[0058] Design a loss function that aligns the category probabilities of the current task model and the historical task model;

[0059] The model parameters are optimized using the following loss function:

[0060]

[0061] in, and Represent the predictions of the current task model and the previous task model for the pseudo sample input, is the classifier for the current task, is the intermediate feature representation of the input gesture, The combined weights calculated for the indexable embeddings, is the adapter module pair feature of the i-th task The transformation output of is the predicted probability distribution after alignment of the current task model, is the predicted probability distribution of the current task model on the learned category set, C K is the learned category set, C U Add a new category set for the current task, It is a normalization operation, which means normalizing the historical category probabilities; The loss function for prototype-driven embedding enhancement is used to measure and optimize task-specific index embedding. KL represents the Kullback-Leibler divergence function, γ is the weight coefficient of the loss function, and is used to balance the loss value. CE(·) is the task loss function for gesture recognition tasks. is the label corresponding to the pseudo sample of the category prototype.

[0062] To achieve the above-mentioned object, a second aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the steps of the quasi-incremental gesture recognition method when executing a computer program stored in the memory.

[0063] To achieve the above-mentioned object, the third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the quasi-incremental gesture recognition method are executed.

[0064] Beneficial effects of the present invention:

[0065] Compared with the prior art, the present invention provides a quasi-incremental gesture recognition method, electronic device, and medium that effectively solves the problems of catastrophic forgetting and confusion between gestures in the absence of playback data by introducing a task-specific compositional mechanism (TCM) and prototype-enhanced hybrid regularization (PHR). TCM constructs a trainable task index embedding, utilizes the similarity between the features of the input gesture and the features of historical tasks, adaptively combines task-specific parameters, achieves dynamic fine-tuning and reasoning, and avoids the problem of difficult parameter selection when the task is unknown. At the same time, PHR uses prototype pseudo samples of historical gesture categories to improve the discriminative ability of task-specific embeddings through prototype-driven embedding enhancement (PEE), and realizes knowledge transfer and prevents over-adjustment of parameters through prototype-driven parameter enhancement (PPE), thereby significantly improving the stability and accuracy of the model and reducing the interference of new gesture learning on learned knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0067] Figure 1 This is a flowchart of a quasi-incremental gesture recognition method disclosed in an embodiment of the present invention.

[0068] Figure 2 This is a diagram of a PHR ablation study disclosed in an embodiment of the present invention.

[0069] Figure 3 This is a basic process and performance comparison diagram of a type of incremental gesture recognition task disclosed in an embodiment of the present invention.

[0070] Figure 4 This is a conceptual diagram of the working mechanism of a PENCIL method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0072] According to an embodiment of the present invention, it should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the following production method, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] In the prior art, gesture recognition systems usually rely on static datasets for centralized training, which leads to catastrophic forgetting problems when facing new gesture categories. In addition, existing continuous learning methods such as replay-based and regularization-based methods have data privacy and low efficiency issues in practical applications. The present invention provides a quasi-incremental gesture recognition method (hereinafter referred to as the "PENCIL method") through a task-specific combination mechanism (TCM) and a prototype-reinforced hybrid regularization (PHR) module to provide an efficient and privacy-friendly quasi-incremental gesture recognition solution that can dynamically adapt to the introduction of new gesture categories while retaining historical gesture knowledge to avoid forgetting.

[0074] The following is a detailed step-by-step description of the specific implementation steps of the PENCIL method, such as Figure 1 As shown, the method includes the following steps:

[0075] Step S100: receiving input data representing a hand gesture, wherein the input data is a three-dimensional skeleton data including a time series, and each frame of the three-dimensional skeleton data includes spatial coordinates of a plurality of hand joints;

[0076] The received gesture input data is a three-dimensional skeleton data containing a time series, and each frame of data includes the spatial coordinates of multiple hand joints. Specifically, the data is represented as Where T is the time dimension (number of frames), J is the number of hand joints, and C is the coordinate dimension (usually 3, representing the X, Y, and Z axes). Compared to high-dimensional video data, 3D skeleton data is more robust and compact, making it suitable for low-sample-size action recognition tasks, such as medical action recognition. Gestures have distinct dynamic characteristics, and time series modeling methods can effectively capture their temporal evolution.

[0077] Step S200: extracting intermediate features according to the received input data, and constructing task-specific index embeddings based on the intermediate features, wherein each index embedding is associated with a specific gesture recognition task;

[0078] Extract the 3D spatial coordinate features of each frame of hand joint from the input data, expressed as J = {j1, j2, ..., j V}, where V = |J| is the number of joints in the gesture skeleton. Time series modeling methods (such as RNN, Transformer, etc.) are used to extract temporal features from the three-dimensional spatial coordinate features to generate temporal features that represent the dynamic changes of the gesture. The temporal features are fused with the three-dimensional spatial features through a feature fusion network to generate the intermediate features f of the gesture. Based on the intermediate features f, a task-specific index embedding vector K associated with the current task is generated through a task-specific trainable module. t . Embed the task-specific index into the vector K t It is dynamically combined with the intermediate features f to form task-specific index embeddings for subsequent task learning.

[0079] Step S300: Calculate the similarity between the extracted intermediate features and the task-specific index embedding to generate a combination weight;

[0080] The intermediate feature f and the index embedding pool K = [K0, K1, ..., K n Each K in t Perform cosine similarity calculation to obtain the similarity score cos(f,K t ). Each similarity score is divided by a predefined scaling factor τ to obtain the scaled similarity score cos(f,K t ) / τ; Apply the exponential function to the scaled similarity score to obtain the unnormalized value of the weight exp(cos(f,K t ) / τ); normalize all unnormalized weight values ​​and calculate the combined weight W of each task-specific index embedding vector t , and its calculation formula is:

[0081]

[0082] Among them, W t is the combined weight, exp is the exponential function, cos(f,K t ) is the intermediate feature representation f and the task-specific index embedding vector K t The cosine similarity between them, K is the index embedding pool, which contains the set of all task-specific index embedding vectors, K j is the fth task-specific index embedding vector in the set K, f represents the intermediate feature vector extracted from the input gesture data by the feature extractor, τ is the scaling factor, For similarity.

[0083] Step S400: using the generated combined weights, dynamically aggregate the isolation parameters of multiple tasks to construct the final features for the current task;

[0084] According to the generated combined weight W t , the corresponding isolation parameter M in the index embedding pool K is M={M0,M1,...,M n} perform weighted combination, namely:

[0085]

[0086] Among them, W t is the task τ t By adding the weight W t Acts on the task isolation parameter M t , which realizes the dynamic participation of different task parameters in the current gesture. Task parameters with high weights will have a greater impact on the final feature generation.

[0087] Apply the weighted combined isolation parameters to the intermediate feature f and add them to f itself to obtain the final output feature Right now:

[0088]

[0089] Among them, f is the initial intermediate feature, M i (f) is the task τ i The result of the isolated parameter module on the feature f. i is the task τ i The combined weight of .

[0090] Final Features It is generated by superimposing the output of task-isolated parameters and intermediate features. This fusion approach not only preserves the global characteristics of the input gesture but also incorporates task-specific features. The accumulation process weights and integrates all task features, ensuring that the final feature reflects the comprehensive relationship between the input gesture and the characteristics of each task. Features that strongly correlate with the current task are given higher weights, while effectively preserving knowledge from previous tasks to avoid catastrophic forgetting.

[0091] Step S500: input the constructed final features into the classifier to perform gesture category prediction;

[0092] The final feature to be built Input to the pre-trained classifier module F c In the classifier module, the fully connected layer is used to classify the final features A linear transformation is performed to generate a category score. The Softmax function is applied to convert the output of this linear transformation into a probability distribution P for each gesture category. Based on this probability distribution P, the category with the highest probability is selected as the final gesture recognition result, thereby accurately classifying the input gesture. Specifically, the fully connected layer completes this process through matrix multiplication and bias addition, generating a score for each gesture category. These scores represent linear activation values ​​indicating whether the input gesture belongs to each category. The classifier then converts these linear activation values ​​into a probability distribution by applying the Softmax function.

[0093] Step S600: training the adapter module for each task based on the prediction results;

[0094] According to the prediction results of the classifier, the category of the current input gesture is determined and associated with the corresponding task, corresponding to the current task τ t , for its independent adapter module M t The model is trained to optimize its adaptability to the current task. Specifically, during the current task training phase, the model only updates the adapter module relevant to the current task, while the adapter parameters for other tasks remain frozen. This ensures that knowledge for the current task can be efficiently learned through the adapter module while avoiding interference with parameters from previous tasks, thereby mitigating the problem of catastrophic forgetting. Furthermore, through task-specific training, the model strengthens the gesture feature representation for the current task, improving classification performance while maintaining an efficient parameter isolation mechanism.

[0095] Step S700: During the adapter module training process, prototypes of historical gesture categories are calculated and pseudo prototype samples are generated, and the pseudo prototype samples are used to embed task-specific indexes and enhance model parameters.

[0096] Feature extractor through gesture recognition model Extract the features belonging to the historical gesture category, and calculate the prototype μ of each category C based on the gesture samples belonging to the same category c and the covariance matrix The specific formula is:

[0097]

[0098] Among them, μ c is the prototype of category c, N c is the number of samples of category c, y i =c represents sample x i The label belongs to category c, Indicates input gesture x i Through feature extractor The intermediate features obtained later; is the covariance matrix of category c, and T is the transpose operator.

[0099] Based on the calculated prototype μ c and the covariance matrix Generate prototype pseudo samples for each historical gesture category, and the pseudo samples follow a multivariate Gaussian distribution:

[0100]

[0101] in, is the generated prototype pseudo sample feature vector, representing the virtual sample of category c, is a multivariate Gaussian distribution with mean μ c , the covariance matrix is c∈{C K \C U} is category c that belongs to the learned historical category set C K Subtract the new category set C introduced by the current task U .

[0102] By utilizing the generated prototype pseudo samples, the task-specific index embedding and model parameters are enhanced through the task-specific combination mechanism (TCM) and prototype-enhanced hybrid regularization (PHR), which improves the model's ability to recognize historical categories and reduces catastrophic forgetting.

[0103] Index Embedding Enhancement (PEE): Using a metric learning strategy, the prototype pseudo sample is used as a negative sample, and the intermediate features of the current task are compared and learned to optimize the indexable embedding and use the loss function To optimize:

[0104]

[0105] in, It is a prototype-driven embedding enhancement loss function used to measure and optimize task-specific index embeddings. m is a boundary parameter used to control the minimum distance between different categories. Prototype pseudo sample The cosine similarity between the task-specific index embedding K, cos(f,K) is the current gesture feature The cosine similarity between the embedding K and the task-specific index.

[0106] By introducing prototype pseudo samples as negative samples, PEE improves the robustness of index embedding and enhances the ability to distinguish between tasks.

[0107] Model Parameter Enhancement (PPE): Use prototype pseudo samples as an intermediary to achieve knowledge transfer from the previous task model to the current task model and prevent over-adjustment of model parameters. Design knowledge transfer loss function As shown below:

[0108]

[0109] in, and Represent the predictions of the current task model and the previous task model for the pseudo sample input, is the classifier for the current task, is the intermediate feature representation of the input gesture, The combined weights calculated for the indexable embeddings, is the adapter module pair feature of the i-th task The transformation output of is the predicted probability distribution after alignment of the current task model, is the predicted probability distribution of the current task model on the learned category set, C K is the learned category set, C U Add a new category set for the current task, It is a normalization operation, which means normalizing the historical category probabilities; The loss function for prototype-driven embedding enhancement is used to measure and optimize task-specific index embedding. KL represents the Kullback-Leibler divergence function, γ is the weight coefficient of the loss function, and is used to balance the loss value. CE(·) is the task loss function for gesture recognition tasks. is the label corresponding to the pseudo sample of the category prototype.

[0110] Step S800: Combine index embedding enhancement and model parameter enhancement to form an overall prototype enhancement hybrid regularization loss, and continue to adjust the model;

[0111] The total loss is calculated by combining task loss, embedding enhancement loss and parameter enhancement loss. The total loss function integrates task recognition loss and two regularization losses:

[0112]

[0113] where α and β are and The trade-off factor is Represents the cross entropy loss of the gesture recognition task. Through the above combination, the model can effectively retain the knowledge of historical tasks while learning new tasks, improving the overall task robustness.

[0114] Step S900: After completing all task training, use the adjusted model to classify and recognize newly input gesture data.

[0115] After completing training for all tasks, the adjusted model is used to classify and recognize newly input gesture data. After training, the model integrates knowledge from all tasks and has the ability to continue learning. When a new gesture is input: Based on the input gesture data, the feature extractor extracts intermediate features, calculates the similarity with the task index embedding, and dynamically aggregates the task isolation parameters. The aggregated features are input into the classifier module, which outputs the probability distribution of each gesture category and selects the category with the highest probability as the recognition result. The model trained with prototype-augmented hybrid regularization can accurately recognize all learned gesture categories, avoid catastrophic forgetting, and maintain efficient performance and task scalability when learning new categories.

[0116] In this embodiment, as described in step S100 above, the input data is a time series of three-dimensional skeleton data representing the dynamic changes of the hand gesture. Each frame of skeleton data consists of the spatial coordinates of multiple human hand joints, where each joint coordinate is a point in a three-dimensional Cartesian coordinate system. Specifically:

[0117] 3D skeleton data is typically collected using motion capture devices or depth sensors (such as Microsoft Kinect, Intel RealSense, or Leap Motion). These devices generate hand skeleton models by capturing depth information, or use convolutional neural networks (such as OpenPose) to extract 3D hand skeleton data from video data. Before entering the model, this data needs to undergo preprocessing such as normalization (to reduce individual differences in gestures, such as hand size and range of motion), denoising (using smoothing filtering methods such as Gaussian filtering to reduce acquisition noise), frame alignment (unifying the time series length through interpolation or cropping), and coordinate transformation (aligning the coordinates to the center of the palm to reduce initial position deviation). 3D skeleton data refines the motion trajectory and spatial relationship of the hand joints and is an important feature of gesture recognition. This is because it discards irrelevant information such as color and background, retains core motion features, and has low dimensionality, high computational efficiency, and strong privacy protection capabilities, demonstrating excellent generalization.

[0118] In this embodiment, as described in step S200 above, the goal of step S200 is to extract dynamic, structured gesture features from the input 3D skeleton data and combine these features with task-related information to generate task-specific index embeddings to support dynamic parameter combination in quasi-incremental gesture recognition. The following details steps S201-S205:

[0119] Step S201: extracting the three-dimensional spatial coordinate features of the hand joints of each frame from the input data.

[0120] In this step, the model extracts the spatial coordinate information of the joints from the input 3D skeleton data frame by frame, including the position coordinates of the palm and finger joints. i =(x i ,y i ,z i ), these data represent the spatial shape and position of the gesture, capture the position information of the joints in three-dimensional space and the spatial relationship between the joints, thereby describing the basic shape of the gesture (such as the difference between an open palm and a clenched fist); at the same time, the joint coordinates are standardized (such as normalized with the palm as the center), and the noise that may be introduced by the acquisition equipment is removed by filtering methods.

[0121] Step S202: extracting time series features from the three-dimensional space coordinate features using a time series modeling method to generate time series features representing dynamic changes in gestures.

[0122] It's understandable that gestures represent dynamic movements, and the spatial coordinate information of each frame alone is insufficient to fully describe their dynamic characteristics. Time series modeling methods (e.g., those based on RNNs, LSTMs, or Transformers) are used to extract the temporal variations of gestures. RNNs (recurrent neural networks) or their improved versions (such as LSTMs and GRUs) are used to capture temporal dependencies in skeleton sequences.

[0123] The generated temporal features capture the dynamic changes in gestures over time, such as the gradual bending of a finger. By modeling the time series, the dynamic features of gestures are extracted, enabling the model to understand the continuous changes in motion, rather than the static state of a single frame.

[0124] Step S203: Fusing the temporal features with the three-dimensional spatial features through a feature fusion network to generate intermediate features of the gesture.

[0125] The spatial features extracted in step S201 and the temporal features extracted in step S202 are integrated through a feature fusion network. The fusion methods may include simple splicing, weighted combination or deep fusion through neural networks. The fusion network can be designed as a parallel structure, with two networks processing spatial features and temporal features respectively, and finally fusion output in a high-dimensional feature space. It should be noted that the intermediate feature is a high-dimensional representation that combines the static shape and dynamic movement of the gesture, which includes both the spatial layout information of the gesture and the dynamic changes in the time dimension; the fused intermediate feature has stronger expressive power and can comprehensively characterize the static and dynamic characteristics of the gesture.

[0126] Step S204: Based on the intermediate features, generate a task-specific index embedding vector associated with the current task.

[0127] Each task has an independent index embedding vector that represents the specific characteristics of the task. These embedding vectors are generated by a trainable module (e.g., a small indexable neural network) and are associated with the task-specific parameter isolation module. For each task τ t , construct a task-related index embedding vector K t The training of embedding vectors is based on the mapping relationship between the gesture intermediate features f and the task labels. These embedding vectors will be stored in the task retrieval pool K = [K0, K1, ..., K n ] to support subsequent dynamic parameter combination. The task-specific index embedding vector provides an independent feature representation for each task, ensuring that the model can distinguish gesture categories between different tasks during the task incremental learning process.

[0128] Step S205: Dynamically combine the task-specific index embedding vector with the intermediate features to form a task-specific index embedding for subsequent task learning.

[0129] The dynamic combination process enables the model to automatically select and activate the parameters of the corresponding task based on the characteristics of the input gesture, thereby supporting example-free incremental learning.

[0130] Through steps S201-S205 in S200, the model extracts static and dynamic features layer by layer from the 3D skeleton data, generates intermediate features through feature fusion, and further constructs task-specific index embedding vectors. Index embedding provides independent parameter support for each task through a dynamic combination mechanism, ensuring that the model can effectively learn and reason in the quasi-incremental gesture recognition task while avoiding catastrophic forgetting and confusion.

[0131] In this embodiment, as described in step S300 above, in order to dynamically combine task-specific parameters, it is necessary to generate a combination weight for each task based on the similarity between the gesture intermediate features and the task-specific index embedding. These combination weights reflect the relevance of the current input gesture to each task and are used to dynamically select a combination of task-specific parameters. The following will be explained in detail in conjunction with steps S301 to S305:

[0132] Step S301: Calculate the cosine similarity between the generated intermediate features and each task-specific index embedding to obtain a similarity score.

[0133] The intermediate feature vector is extracted from the input 3D skeleton data and represents the characteristics of the gesture. Each task-specific index embedding vector K t Representation and task τ t Use cosine similarity to measure f and each K t The cosine similarity reflects the similarity between f and Kt By calculating the cosine similarity between f and each task index embedding, the degree of association between the current input gesture and each task can be measured. A higher similarity means that the current gesture is more likely to belong to the gesture category of the task.

[0134] Step S302: Calculate the similarity score for all task-specific index embeddings.

[0135] Assume that the current model has learned n tasks τ0,τ1,...,τ n , then each task has a task-specific index embedding K0,K1,...,K n Extend the cosine similarity calculation in step S301 to all task-specific index embeddings K i ∈K(index embedding pooling). By calculating the similarity scores for all tasks, the input gestures can be associated with each other at the task level.

[0136] Step S303: Divide each similarity score by a predefined scaling factor to obtain a scaled similarity score.

[0137] The similarity scores are smoothed using a scaling factor τ. Smoothing avoids numerical instability caused by extreme differences between similarity values ​​(e.g., close to 1 or close to 0). The scaled similarity scores make the subsequent exponential function calculation more stable.

[0138] Step S304: applying an exponential function to all scaled similarity scores to obtain unnormalized values ​​of weights.

[0139] Here, the exponential function amplifies the weight of high similarity scores while suppressing the weight of low similarity scores. The exponential function amplifies the influence of similarity and further distinguishes between tasks that are more likely to be associated with the current gesture and those that are less likely to be associated. The unnormalized weights are the intermediate results that generate the final combined weights.

[0140] Step S305: Normalize all unnormalized weight values ​​and calculate the combined weight of each task-specific index embedding vector.

[0141] All unnormalized weights are normalized to obtain the combined weight.

[0142] Normalization allows each weight to be interpreted as a probability value, indicating the relationship between the current input gesture and the task τ i The combined weight provides a scaling factor for the subsequent dynamic parameter combination, enabling the model to dynamically activate task-related parameters based on the characteristics of the current gesture.

[0143] In step S300, the model uses cosine similarity calculation and a dynamic weighting mechanism based on the input intermediate features to generate combined weights for each task. These weights accurately reflect the degree of correlation between the current input gesture and each task, providing a basis for the subsequent step of dynamically aggregating the isolation parameters of multiple tasks (step S400). This approach avoids the limitations of directly selecting a single task index embedding, fully utilizes the specific characteristics of different tasks, and improves the flexibility and robustness of the model.

[0144] In this embodiment, as described in step S400 above, the core objective of step S400 is to dynamically aggregate the isolated parameters of multiple tasks based on the task combination weights generated in the previous step (step S300), thereby generating final features specific to the current task based on the input gesture data. In this way, the model can achieve dynamic adaptation, not only handling the current task but also effectively utilizing and integrating knowledge from previous tasks. The following details steps S401-S403.

[0145] Step S401: applying the combined weight to the isolation parameters of each relevant task.

[0146] Each task has an independent isolation parameter module, which stores task-specific parameters through an adapter mechanism. The isolation parameters of each task are weighted according to the task combination weight calculated in step S300.

[0147] The task weighting operation is formalized as: W i ·M i (f). Where W i is the task τ t By adding the weight W i Acts on the task isolation parameter M i , which realizes the dynamic participation of different task parameters in the current gesture. Task parameters with high weights will have a greater impact on the final feature generation.

[0148] Step S402: applying the weighted isolation parameters to the extracted intermediate features one by one.

[0149] For each task, the weighted isolation parameter module W i ·M i Acting on the extracted intermediate features, the input features are transformed through the isolation parameter module for each task to generate task-specific feature representations. The operations of the isolation parameter module can be processed in parallel, that is, the isolation parameters of all tasks are applied to the intermediate features simultaneously. The intermediate features are projected into a subspace specific to each task to obtain different task feature representations, ensuring that the model can fully exploit the characteristics of the task isolation parameters.

[0150] Step S403: Accumulate the features after each task is performed to obtain the final features for the current task.

[0151] The output features of all tasks in step S402 are weighted and accumulated to generate the final features for the current task.

[0152] In this embodiment, as described in step S500 above, the model inputs the final features generated in step S400 into a pre-trained classifier module for predicting gesture categories. The classifier module first performs a linear transformation on the final features using a fully connected layer, mapping them from a high-dimensional feature space to a dimension corresponding to the number of gesture categories.

[0153] In this embodiment, as described in step S600 above, in step S600, the model trains the adapter module for each task based on the prediction results to further optimize the task-specific isolation parameters. Specifically, during the current task training phase, the model only updates the adapter module related to the current task, while the adapter parameters of other tasks remain frozen. This ensures that the knowledge of the current task can be efficiently learned through the adapter module while avoiding interference with historical task parameters, thereby alleviating the problem of catastrophic forgetting. In addition, through task-specific training, the model can strengthen the gesture feature representation of the current task, improve classification performance, and maintain an efficient parameter isolation mechanism.

[0154] The following is a detailed description of the specific implementation process of the PENCIL method, including experimental steps, key technical points, experimental settings, and performance comparison.

[0155] 1. Experimental Dataset and Benchmark

[0156] 1. Experimental dataset: This experiment uses two public datasets: SHREC-2017 and EgoGesture3D.

[0157] SHREC-2017: Contains 14 gesture categories, captured using an Intel RealSense depth camera, including 22 hand keypoints (1 for the wrist, 1 for the palm, and 4 for each finger). The training set contains 1980 samples, and the test set contains 840 samples.

[0158] EgoGesture3D: Contains 83 one-hand and two-hand gesture categories, with a total of 14,416 training samples, 4,768 validation samples, and 4,977 test samples. It is derived from the EgoGesture dataset and uses MediaPipe to extract 3D skeletons.

[0159] 2. Baseline methods: To verify the effectiveness of the PENCIL method, it is compared with the following methods: Oracle, Base, Fine-tuning, Feature Extraction, LWF, LWFMC, DeepInversion, ABD, R-DFCIL, and BOAT-MI (one of the current state-of-the-art methods).

[0160] 2. Experimental settings and task details:

[0161] 1. Experimental task division: Using an incremental learning-like experimental setting, the gesture recognition task is divided into a basic task (task 0) and subsequent incremental tasks (tasks 1 to 6).

[0162] SHREC-2017: Task 0 includes 8 types of gestures, and Tasks 1 to 6 add 1 new type of gesture each time;

[0163] EgoGesture3D: Task 0 contains 59 types of gestures, and tasks 1 to 6 add 4 types of gestures each time.

[0164] 2. Model training details:

[0165] Optimizer: Same as BOAT-MI, using Adam optimizer;

[0166] Learning rate: SHREC-2017 is set to 6e-5, EgoGesture3D is set to 3e-5;

[0167] Hardware environment: All experiments are performed on an Nvidia 3090 24GB GPU.

[0168] 3. Performance evaluation indicators:

[0169] Global Accuracy (ACC↑): measures the overall accuracy of gesture recognition;

[0170] Performance degradation rate (PD↓): reflects the impact of the new task on the accuracy of historical category recognition.

[0171] 3. Experimental steps of the PENCIL method:

[0172] 1. Task 0 training: The initial model is trained on the base category dataset (Task 0), extracting task-specific index embeddings, building adapter modules and generating isolated parameter pools.

[0173] 2. Task incremental learning: Each time a new gesture category is introduced, the following steps are performed:

[0174] Extract intermediate features for new tasks; use a task-specific combination mechanism (TCM) to calculate combination weights based on the similarity between intermediate features and the index embedding pool, and dynamically aggregate isolation parameters; use prototype-reinforced hybrid regularization (PHR) to optimize index embeddings and model parameters through pseudo sample generation and knowledge transfer; while predicting new categories in the classifier, maintain the ability to recognize historical categories.

[0175] 3. Ablation experiment: Analyzing the effects of TCM and PHR: Removing TCM and using simple parameter averaging showed a significant decrease in accuracy; removing PHR also weakened the model's ability to distinguish similar categories.

[0176] 4. Experimental results and analysis

[0177] 1. Comparison of recognition accuracy:

[0178] The experimental results are shown in Tables 1 and 2. Tables 1 and 2 show the performance of PENCIL on the SHREC-2017 and EgoGesture3D datasets;

[0179] SHREC-2017: Outperformed BOAT-MI by 13.33% in Task 6, with an average accuracy of 79.89%;

[0180] EgoGesture3D: outperforms the BOAT-MI method by 22.43% in Task 6, achieving an average accuracy of 69.33%.

[0181] Table 1: Class-incremental learning results for six tasks in SHREC-2017

[0182]

[0183]

[0184] Table 2: Class-incremental learning results for six tasks in EgoGesture3D

[0185]

[0186] 2. Computational efficiency: PENCIL significantly outperforms BOAT-MI in terms of computational efficiency. The experimental results are shown in Table 3:

[0187] Table 3: Overall training time and memory overhead

[0188]

[0189] Table 3 shows the time and memory overhead comparison between PENCIL and BOAT-MI;

[0190] SHREC-2017: 93.9% reduction in processing time and 93.2% reduction in memory usage;

[0191] EgoGesture3D: Processing time reduced by 92.8% and memory usage reduced by 97%;

[0192] The results demonstrate the advantages of deploying PENCIL on edge devices.

[0193] 3. Ablation study:

[0194] Effectiveness of Task-Specific Combination Mechanism (TCM): The task-specific combination mechanism significantly improves the recognition ability of task-irrelevant samples; Table 4 shows the effectiveness of TCM. After removing TCM, the performance drops significantly:

[0195] Improvements of Prototype-enhanced Hybrid Regularization (PHR): By introducing pseudo samples, the confusion between similar categories is significantly reduced, and the model stability is improved. Figure 2 The ablation experimental results of PHR are presented. Removing the PEE or PPE module leads to performance degradation.

[0196] Table 4: Ablation studies of different aggregation methods

[0197]

[0198] V. Versatility of the Method and Available Options:

[0199] 1. Compatibility of adapter modules: As shown in Table 5, three adapter structures (convolutional adapter, compact adapter, and bottleneck adapter) were tested experimentally. PENCIL performed well in all structures, demonstrating its model independence.

[0200] 2. Pseudo sample generation method: The prototype pseudo sample generation can be extended to a method based on generative adversarial network (GAN) to further enhance the diversity and authenticity of pseudo samples;

[0201] 3. Multi-task expansion: For more complex task sequences, a hierarchical index embedding pool can be designed to improve efficiency and accuracy under multi-task conditions.

[0202] Table 5: Ablation studies using different adapter methods

[0203]

[0204] Figure 3This paper presents the basic process and performance comparison of the Class-Incremental Hand Gesture Recognition (CI-HGR) task. Part (a) illustrates the characteristics of the CI-HGR task, namely that the model first performs initial training on the base category gestures (task 0), and then sequentially learns new gesture categories (tasks 1 to N), without accessing samples from previous tasks during this process. Part (b) experimentally compares the performance of the proposed PENCIL method with the existing state-of-the-art method BOAT-MI on the SHREC-2017 dataset. The results show that the PENCIL method significantly outperforms BOAT-MI in global accuracy and has a lower performance degradation rate, verifying the significant advantages of the present invention in addressing catastrophic forgetting and improving learning efficiency.

[0205] Figure 4 The conceptual diagram of the working mechanism of the proposed PENCIL method includes two core components: (1) Task-Specific Compositional Mechanism (TCM), which is used to dynamically combine task-specific isolation parameters to adapt to different gesture inputs and solve the problem of task information agnosticism in quasi-incremental scenarios through adaptive parameter adjustment; (2) Prototype-Enhanced Hybrid Regularization (PHR), which uses prototype pseudo samples to optimize the performance of TCM from two aspects: first, improving the accuracy of TCM matching pseudo samples through prototype-driven embedding enhancement (PEE), and second, realizing the transfer of historical model knowledge to the current model through prototype-driven parameter enhancement (PPE) to avoid excessive parameter adjustment. The entire mechanism significantly improves the recognition accuracy and model stability in quasi-incremental gesture recognition tasks through the synergistic effect of TCM and PHR.

[0206] 6. Summary:

[0207] Experimental results demonstrate that the PENCIL method significantly outperforms existing state-of-the-art techniques on both datasets, achieving superior accuracy, performance degradation, and computational efficiency. By combining a task-specific combination mechanism with prototype-enhanced hybrid regularization, PENCIL effectively addresses the catastrophic forgetting and recognition confusion issues inherent in quasi-incremental learning, providing an efficient and reliable solution for gesture recognition tasks.

[0208] According to another aspect of an embodiment of the present application, an electronic device is provided, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0209] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0211] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0213] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A quasi-incremental gesture recognition method, characterized in that: The steps include: receiving input data representing a hand gesture, wherein the input data is three-dimensional skeleton data including a time series, each frame of the three-dimensional skeleton data including spatial coordinates of a plurality of hand joints; extracting intermediate features according to the received input data, and constructing task-specific index embeddings based on the intermediate features, wherein each index embedding is associated with a gesture recognition task; Calculate the similarity between the extracted intermediate features and the task-specific index embedding to generate the combination weight; Use the generated combined weights to dynamically aggregate the isolated parameters of multiple tasks to construct the final features for the current task; The constructed final features are input into the classifier to perform gesture category prediction; Based on the prediction results, the adapter module is trained for each task; During the adapter module training process, prototypes of historical gesture categories are calculated and pseudo-prototype samples are generated. The pseudo-prototype samples are used to embed task-specific indexes and enhance model parameters. Combine index embedding enhancement and model parameter enhancement to form an overall prototype enhancement hybrid regularization loss and continue to tune the model; After completing all task training, use the adjusted model to classify and recognize new input gesture data; Methods for enhancing model parameters using prototype pseudo samples include: Based on the prototype pseudo samples of the historical task category and the characteristics of the training samples of the current task, the pseudo samples are used as an intermediary to transfer knowledge; Design a loss function that aligns the category probabilities of the current task model and the historical task model; The model parameters are optimized using the following loss function: in, and Represent the predictions of the current task model and the previous task model for the pseudo sample input, is the classifier for the current task, is the intermediate feature representation of the input gesture, The combined weights calculated for the indexable embeddings, For the The adapter module for each task has features The transformation output of is the predicted probability distribution after alignment of the current task model, is the predicted probability distribution of the current task model on the learned category set, is the learned category set, Add a new category set for the current task, It is a normalization operation, which means normalizing the historical category probabilities; A loss function enhanced for prototype-driven embeddings to measure and optimize task-specific index embeddings, represents the Kullback-Leibler divergence function, is the weight coefficient of the loss function, which is used to balance the loss value. is the task loss function for the gesture recognition task, is the label corresponding to the pseudo sample of the category prototype.

2. The incremental gesture recognition method according to claim 1, wherein: Methods for constructing task-specific index embeddings based on this intermediate feature include: Extracting three-dimensional spatial coordinate features of the hand joints of each frame from the input data, wherein the three-dimensional spatial coordinate features include the position of each hand joint in the three-dimensional space; Extracting time series features from the three-dimensional spatial coordinate features using a time series modeling method to generate time series features representing dynamic changes in gestures; The temporal features and the three-dimensional spatial features are fused through a feature fusion network to generate intermediate features of the gesture; Based on the intermediate features, generating a task-specific index embedding vector associated with the current task, wherein the task-specific index embedding vector is constructed by a task-specific trainable module; The task-specific index embedding vector is dynamically combined with the intermediate features to form a task-specific index embedding for subsequent task learning.

3. The incremental gesture recognition method according to claim 2, wherein: The similarity between the extracted intermediate features and the task-specific index embedding is calculated, and the methods for generating the combination weights include: Calculate the cosine similarity between the generated intermediate features and each task-specific index embedding to obtain a similarity score; Compute the above similarity scores for all task-specific index embeddings; Divide each similarity score by a predefined scaling factor to obtain a scaled similarity score; Apply an exponential function to all scaled similarity scores to obtain the unnormalized value of the weight; Normalize all unnormalized weight values ​​and calculate the combined weight of each task-specific index embedding vector, which is calculated as follows: in, is the combined weight, is an exponential function, is the intermediate feature representation Embedding vectors with task-specific indexes The cosine similarity between is the index embedding pool, which contains the set of all task-specific index embedding vectors, For collection The task-specific index embedding vectors, represents the intermediate feature vector extracted from the input gesture data by the feature extractor, is the scaling factor, For similarity.

4. The incremental gesture recognition method according to claim 1, wherein: Dynamic aggregation includes weighted combination of the isolation parameters of the multiple tasks according to the combination weight, specifically including: applying the combined weights to the isolated parameters of each relevant task; Apply the weighted isolation parameters to the extracted intermediate features one by one; The features after each task are accumulated to obtain the final features for the current task.

5. The incremental gesture recognition method according to claim 1, wherein: Gesture category prediction includes the following specific steps: The constructed final features are input into the pre-trained classifier module; In the classifier module, the final features are linearly transformed using the fully connected layer; By applying the Softmax function, the output of the linear transformation is converted into the probability distribution of each gesture category; According to the probability distribution, the category with the highest probability is selected as the final gesture recognition result.

6. The incremental gesture recognition method according to claim 1, wherein: During the adapter module training process, the method of calculating the prototype of the historical gesture category and generating the prototype pseudo sample includes: Extract features belonging to historical gesture categories through the feature extractor of the gesture recognition model; According to the gesture samples belonging to the same category, the prototype and covariance matrix of the category are calculated. The specific formula is as follows: in, For category The prototype, For category The number of samples, Represents a sample The label belongs to the category , Indicates input gesture Through feature extractor The intermediate features obtained later; For category The covariance matrix of is the transpose operator; Based on the calculated prototype and covariance matrix, a prototype pseudo sample is generated for each historical gesture category. The prototype pseudo sample follows a multivariate Gaussian distribution: in, is the generated prototype pseudo sample feature vector, representing the category The virtual sample of is a multivariate Gaussian distribution with a mean of , the covariance matrix is , For category Belongs to the learned set of historical categories Subtract the set of new categories introduced by the current task .

7. The incremental gesture recognition method according to claim 1, wherein: Methods for enhancing task-specific index embeddings with prototype pseudo samples include: A metric learning strategy is used to treat prototype pseudo samples as negative samples, and compared them with the intermediate features of the current task to optimize indexable embeddings. The task-specific index embedding is updated by maximizing the similarity between the original sample features and the correct task embedding and minimizing its similarity with the pseudo sample features. The task-specific index embedding is optimized using the following loss function: in, A loss function enhanced for prototype-driven embeddings to measure and optimize task-specific index embeddings, is a boundary parameter used to control the minimum distance between different categories. Prototype pseudo sample Embedding with task-specific indexes The cosine similarity between Current gesture feature Embedding with task-specific indexes The cosine similarity between .

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to implement the steps of the incremental gesture recognition method according to any one of claims 1 to 7 when executing a computer program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the incremental gesture recognition method according to any one of claims 1 to 7 are executed.

Images