Lightweight semi-supervised incremental gesture recognition method and system for resource-constrained data gloves

Abstract

The lightweight semi-supervised incremental gesture recognition method and system for resource-constrained data gloves belong to the technical field of intelligent data glove control. In order to solve the problem of data glove recognition accuracy, the present application comprises a branch routing module, a semantic stability module, a drift detection module, a prototype memory module, an artificial correction module and a new class expansion module. Gesture data collected by the intelligent data glove is input into the branch routing module, the branch routing module is connected with the semantic stability module, the semantic stability module is connected with the drift detection module, the drift detection module is respectively connected with the prototype memory module and the artificial correction module, and the artificial correction module is connected with the new class expansion module. The present application realizes a complete lightweight semi-supervised incremental gesture recognition process and its system module combination, which is composed of heterogeneous branch reasoning, semantic stability, sub-position uncertainty gating, bounded prototype updating, token budget artificial correction and new class prototype expansion.

Description

Technical Field

[0001] This invention belongs to the field of intelligent data glove control technology, specifically relating to a lightweight semi-supervised incremental gesture recognition method and system for resource-constrained data gloves. Background Technology

[0002] The gesture recognition system based on smart data gloves collects finger bending and hand movement information through stretch sensors and inertial measurement units, and outputs control commands using a classification model. As wearable interaction moves from experimental environments to long-term use, the recognition model not only needs to process the initial training data, but also needs to adapt to continuous distribution shifts caused by different users, glove re-wearing, sensor position changes, muscle fatigue, and changes in execution habits. Users may also add new gesture categories after deployment that were not present during the training phase.

[0003] Existing personalized methods typically employ full model fine-tuning, empirical replay, regularized continuous learning, or server-assisted retraining. Full fine-tuning requires saving intermediate activations, gradients, and optimizer states; empirical replay requires continuously saving historical samples; regularization methods may also save a statistical matrix commensurate with the model parameters. These methods are suitable for servers or mobile devices, but are difficult to run long-term on microcontrollers with limited on-chip storage, fixed sampling periods, and battery power.

[0004] While generating pseudo-labels solely based on model predictions eliminates the need for manual annotation, it can easily introduce incorrect predictions into the model during situations like glove misalignment or cold starts with new users, leading to confirmation bias and contaminating subsequent updates. Using a fixed confidence threshold also struggles to adapt to varying probability distributions across different categories, users, and deployment phases. Furthermore, frequent requests for user annotation disrupt the continuity of interaction; therefore, manual labeling should only be used for high-risk situations such as persistent uncertainty or the emergence of new categories.

[0005] Therefore, an incremental identification method suitable for microcontrollers on the data glove side is needed: freezing the feature extraction network during normal operation, maintaining only the class prototypes with bounded capacity, support anchors, and confidence statistics; using the classifier's posterior probability and empirical quantiles to determine whether unlabeled samples are reliable; requesting artificial truth values ​​only when there is continuous drift and the query budget allows; and directly expanding the latent prototypes after obtaining a small number of new class labels without retraining the complete network. Summary of the Invention

[0006] The problem this invention aims to solve is that the recognition accuracy of data gloves decreases during long-term use due to new users, re-wearing, muscle fatigue, changes in movement habits, and sensor drift. Furthermore, fine-tuning of the complete model, experience playback, and regularized continuous learning require significant runtime storage and computing resources, making them difficult to deploy on microcontrollers. Unconstrained pseudo-label updates easily accumulate errors, and fixed thresholds struggle to adapt to changes in probability distribution. Frequent manual annotation increases the user burden, and closed-set classifiers cannot reliably add new gesture categories with a small number of labels. This invention proposes a lightweight semi-supervised incremental gesture recognition method and system for resource-constrained data gloves.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves includes the following steps:

[0009] S1. Collect reliable calibration data from the smart data glove, use the smart data glove to collect gesture data, preprocess to obtain valid gesture fragments and their static type labels or dynamic gesture type labels, and obtain preprocessed gesture data;

[0010] S2. The preprocessed gesture data obtained in step S1 is input into the branch routing module for static branching or dynamic branching to obtain latent features, class posterior probabilities and predicted labels.

[0011] S3. Input the latent features, class posterior probabilities and predicted labels obtained in step S2 into the semantic stabilization module for semantic stabilization processing to obtain the latent features, class posterior probabilities and predicted labels corresponding to stable and valid gesture segments;

[0012] S4. The drift detection module calculates the maximum class posterior probability, the confidence margin between the first class and the second class, and the posterior entropy based on the class posterior probability output in step S3. It then uses the empirical quantiles of the reliable calibration data collected in step S1 to determine the adaptive thresholds, including the high maximum class posterior probability threshold, the high confidence margin threshold, the low maximum class posterior probability threshold, the low confidence margin threshold, and the high entropy threshold. Finally, it establishes a joint uncertain state model and its time de-jittering conditions.

[0013] S5. Based on the adaptive threshold obtained in step S4, the prototype memory module accepts the predicted label as a pseudo-label for unlabeled stable and valid gesture segments that simultaneously satisfy the high maximum class posterior probability threshold and the high confidence margin threshold, calculates the prototype update coefficient, and updates the corresponding class prototype, class-by-class confidence statistic and class-by-class finite support memory in a closed manner, and outputs the current predicted label confirmed by the pseudo-label admission.

[0014] S6. Based on the adaptive threshold obtained in step S4, the manual correction module determines a stable and valid gesture segment as a drift candidate when it simultaneously satisfies the low maximum class posterior probability threshold, the low confidence edge threshold, and the high entropy threshold, and the joint uncertain state is satisfied continuously a predetermined number of times. When the query token and the adaptive cooling condition are satisfied at the same time, the module requests the manual truth value through Bluetooth Low Energy and outputs the manual truth value request and the state of the queried segment.

[0015] S7. The new class extension module judges the artificial truth value obtained in step S6. When the artificial truth value belongs to an existing class, it overwrites and updates the corresponding class prototype. When the artificial truth value belongs to a new class outside the current vocabulary, it is marked as a temporary class, and the temporary class prototype, support anchors in class-by-class limited support memory, and confidence statistics are initialized.

[0016] Furthermore, both the static branching and dynamic branching processes in step S2 include an offline-trained feature extractor and a base classifier, respectively. For the static branching process, the feature extractor and the base classifier are composed of the hidden layer and the output layer of a lightweight multilayer perceptron (MLP), respectively. For the dynamic branching process, the feature extractor and the base classifier are composed of the convolutional backbone and the output classification layer of an INT8 quantized one-dimensional convolutional neural network, respectively.

[0017] Furthermore, the semantic stability processing in step S3 includes merging short-interval segments with the same label, splitting excessively long bimodal segments, and correcting segment boundary artifacts. When adjacent valid gesture segments have the same predicted label and the time interval is not greater than a preset merging threshold, adjacent valid gesture segments are merged. When the dominant category confidence trajectory of an excessively long valid gesture segment is bimodal and the valley value is lower than a preset confidence threshold, the excessively long valid gesture segment is split. Static branching or dynamic branching is re-executed on the merged or split segments to obtain the latent features, category posterior probabilities, and predicted labels corresponding to stable valid gesture segments.

[0018] Furthermore, the expression for establishing the joint uncertain state model in step S4 is as follows:

[0019]

[0020] in, Let k be the joint uncertain state of the k-th stable and valid gesture segment. Let be the maximum class posterior probability of the k-th stable and valid gesture segment. This is a low maximum class posterior probability threshold. The logical AND / conjunction symbols, For the confidence edge of the k-th stable and valid gesture segment, For low confidence edge threshold, The posterior probability vector of the k-th stable and valid gesture segment entropy, It is a high-entropy threshold;

[0021] Time-based de-shaking conditions are In continuous Continuous capture of true values ​​from stable and effective gesture segments. It is judged as a drift candidate, among which The number of consecutive times the time-de-jitter condition is applied.

[0022] Furthermore, in step S5, when an unlabeled stable and valid gesture fragment satisfies the high maximum class posterior probability threshold and the high confidence margin threshold, the predicted label is accepted as a pseudo-label; the prototype update coefficient is calculated based on the similarity between the latent features of the stable and valid gesture fragment and the current prototype of the corresponding category of the predicted label, and the corresponding category prototype, class-by-class confidence statistics, and class-by-class finite support memory are updated in a closed manner. The latent features that have passed the admission are written into the finite support memory of the corresponding category as support anchors, and the current predicted label confirmed by the pseudo-label admission is output.

[0023] Furthermore, the condition for the artificial truth-triggered state in step S6 is:

[0024]

[0025] in, This is a manually triggered truth value state. For the k-th stable and valid gesture segment to continuously satisfy The number of stable and effective gesture segments, Let be the number of remaining query tokens corresponding to the k-th stable and valid gesture fragment. This represents the number of decision-making steps since the last manual query. This represents the cooling interval corresponding to the k-th stable and effective gesture segment.

[0026] Furthermore, in step S7, when the artificial truth value belongs to a new category outside the current vocabulary, the new category is marked as a temporary category, and the temporary category prototype, support anchors in the class-by-class limited support memory, and confidence statistics are initialized with the potential features of the queried fragment; after obtaining sufficient artificial truth value confirmation anchors or repeating high-confidence matching, the temporary category is converted into an active category.

[0027] Furthermore, in the optional mobile backup path, the feature extractor of the selected recognition branch in step S2 is frozen, and the lightweight classifier head is refreshed only using latent vectors that have been manually verified or have high confidence admission; the classifier head parameters are the weight parameters and bias parameters in the basic classifier that map latent features to the posterior probabilities of each gesture category; after the refresh is completed, the updated classifier head parameters are asynchronously sent back to the glove-side device.

[0028] A lightweight semi-supervised incremental gesture recognition system for resource-constrained data gloves is provided, which implements the aforementioned lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, including a branch routing module, a semantic stabilization module, a drift detection module, a prototype memory module, a manual correction module, and a new class extension module.

[0029] The smart data glove collects gesture data and inputs it into the branch routing module. The branch routing module is connected to the semantic stabilization module, which is connected to the drift detection module. The drift detection module is connected to the prototype memory module and the manual correction module, respectively. The manual correction module is connected to the new class extension module.

[0030] Furthermore, the smart data glove has stretch sensors installed at the five finger positions, and a six-axis inertial measurement unit containing a three-axis accelerometer and a three-axis gyroscope is installed on the back of the hand, forming an eleven-channel synchronous sensor sequence; the glove side is equipped with an ESP32-S3 microcontroller, 512KB on-chip SRAM, non-volatile memory and a low-power Bluetooth communication module.

[0031] The beneficial effects of this invention are:

[0032] This invention presents a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, reducing incremental adaptation storage on the microcontroller side. Complete fine-tuning, experience playback, and EWC-1D require approximately 1048 KB, 812 KB, and 486 KB of adaptation states, respectively, exceeding or approaching the available on-chip SRAM of the ESP32-S3. This invention only stores category prototypes, limited support anchors, and confidence statistics; the measured adaptation state is 4.40 KB, and the storage requirement does not increase indefinitely with the length of the historical data stream.

[0033] The present invention provides a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, meeting the update latency requirements of real-time data streams. The closed-loop pseudo-label update and manual truth update of this invention are approximately 0.252 ms and 0.180 ms respectively, while the complete update event, including forward inference and trigger judgment, is approximately 14.323 ms and 14.251 ms respectively, lower than the 20 ms single-cycle budget under 50 Hz sampling conditions.

[0034] This invention presents a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, capable of recovering accuracy lost due to glove re-wearing using unlabeled samples. Re-wearing caused the frozen model's accuracy to drop from 97.74% to 82.35%; after using 20 unlabeled samples, the naive pseudo-label method only recovered to 85.12%, while this invention recovers it to 96.50% through quantile gating and bounded prototype updates.

[0035] This invention presents a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, improving the personalization capabilities for new users during cold starts. In a 30-fold, one-subject-retention experiment, the average accuracy of the source user-trained model on unseen users was 80.62%. This invention, using only unlabeled data streams from target users, achieved a 94.88% fit in one round and 96.61% in five rounds, with the standard deviation decreasing from 6.41 percentage points to 1.25 percentage points.

[0036] This invention presents a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, reducing unnecessary updates and prototype contamination. Continuous updates trigger more than 500 times per day with an update duty cycle of approximately 14.50%, achieving an accuracy of 91.20%. This invention further reduces the update frequency to approximately 3.2 times per day by combining uncertainty and time debouncing, with a duty cycle below 0.15% and an accuracy of 97.82%.

[0037] This invention presents a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, supporting the insertion of new classes with few samples and reducing catastrophic forgetting. For new gestures not encountered during the training phase, the accuracy of new classes is 31.42% when using two labels for final fine-tuning, while the forgetting rate of existing classes exceeds 18.5%. This invention utilizes a prototype extension of artificial truth gating, achieving a new class accuracy of 90.89% with only two labels and a forgetting rate of less than 0.5%.

[0038] This invention discloses a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, which utilizes heterogeneous branches to reduce conventional inference overhead. The dynamic INT8 one-dimensional convolution branch achieves 97.74% accuracy in dynamic gestures with 10880 multiply-accumulate operations and a latency of 0.87 ms; the static multilayer perceptron branch achieves 100.0% accuracy under a controlled binary state static gesture protocol with fewer than 5000 multiply-accumulate operations and a latency of less than 0.10 ms. This branch routing reserves computational and storage margins for subsequent drift detection and incremental updates. Attached Figure Description

[0039] Figure 1 This is a flowchart of a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves as described in this invention;

[0040] Figure 2 This is a flowchart of a lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, as described in this invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0042] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0043] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 and attached Figure 2 Detailed explanation is as follows:

[0044] Example 1:

[0045] A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves includes the following steps:

[0046] S1. Collect reliable calibration data from the smart data glove, use the smart data glove to collect gesture data, preprocess to obtain valid gesture fragments and their static type labels or dynamic gesture type labels, and obtain preprocessed gesture data;

[0047] Furthermore, threshold segmentation, learning-based segmentation, or personalized multimodal continuous gesture segmentation methods are used for preprocessing of gesture data; the specific segmentation method does not constitute a necessary limitation on the incremental adaptation method of this invention.

[0048] Furthermore, the reliable calibration data is obtained from the same data glove during the offline training phase before deployment or during the initial calibration phase of first use. Specifically, it includes a small number of valid gesture fragments with manually verified labels and their posterior probability statistics output by the recognition branch. This calibration data is only used to estimate the empirical quantile thresholds for the maximum class posterior probability, confidence margins, and posterior entropy. It does not require additional sensors or is an independent data stream acquired additionally during runtime.

[0049] S2. The preprocessed gesture data obtained in step S1 is input into the branch routing module for static branching or dynamic branching to obtain latent features, class posterior probabilities and predicted labels.

[0050] Furthermore, both the static branching and dynamic branching processes in step S2 include an offline-trained feature extractor and a base classifier, respectively. For the static branching process, the feature extractor and the base classifier are composed of the hidden layer and the output layer of a lightweight multilayer perceptron (MLP), respectively. For the dynamic branching process, the feature extractor and the base classifier are composed of the convolutional backbone and the output classification layer of an INT8 quantized one-dimensional convolutional neural network, respectively.

[0051] Furthermore, for static gesture segments, the mean of the five stretched channels within the segment is calculated to obtain a five-dimensional finger pose vector, which is then input into a small multilayer perceptron. For dynamic gesture segments, the five-channel stretched data and six-axis inertial data are concatenated into an eleven-channel sequence, and the time length is unified to 50 frames through linear interpolation before being input into an INT8 quantized one-dimensional convolutional neural network.

[0052] Furthermore, when the glove is running normally, both branches only perform forward inference and do not update the complete network parameters.

[0053] (1)

[0054] in, This represents the k-th stable and valid gesture segment; This indicates the identification of branch markers; static gestures are used for identification. Dynamic gesture capture ; and These represent the fixed feature extractor and the base classifier for the selected branch, respectively. This represents the latent feature corresponding to the k-th stable and valid gesture segment; This represents the posterior probability of each gesture category; This represents the predicted category with the highest posterior probability. Let represent the total number of current activity categories at time t. Therefore, Equation (1) is a unified representation of the static multilayer perceptron branch and the dynamic INT8 quantized one-dimensional convolutional neural network branch.

[0055] S3. Input the latent features, class posterior probabilities and predicted labels obtained in step S2 into the semantic stabilization module for semantic stabilization processing to obtain the latent features, class posterior probabilities and predicted labels corresponding to stable and valid gesture segments;

[0056] Furthermore, the semantic stability processing in step S3 includes merging short-interval segments with the same label, splitting excessively long bimodal segments, and correcting segment boundary artifacts. When adjacent valid gesture segments have the same predicted label and the time interval is not greater than a preset merging threshold, adjacent valid gesture segments are merged. When the dominant category confidence trajectory of an excessively long valid gesture segment is bimodal and the valley value is lower than a preset confidence threshold, the excessively long valid gesture segment is split. Static branching or dynamic branching is re-executed on the merged or split segments to obtain the latent features, category posterior probabilities, and predicted labels corresponding to stable valid gesture segments.

[0057] Furthermore, the stable and valid gesture fragments, category posterior probabilities, and predicted labels output after semantic stabilization in step S3 are not used to update the complete recognition network, but are instead used to establish and maintain the following capacity-bounded incremental adaptation state, resulting in:

[0058] (2)

[0059] in, This represents the incremental adaptation state at time t; This represents a set of class-by-class prototypes, where each activity category stores a d-dimensional prototype to represent the center of that category in the latent space; This indicates that memory is supported in a limited manner for each category, with a maximum of R recent potential anchors stored per category to supplement multiple user-specific patterns within the category; This represents the class-by-class confidence statistic, used to maintain confidence moving statistics and thresholds. The above state is initialized with calibration data before deployment and is only updated via closed-form calculation in the high-confidence update of step S5 or the manual truth correction of step S7. The expression is:

[0060] (3)

[0061] in, This indicates a closed-form update operator that does not use gradient descent; This represents the latent feature corresponding to the k-th stable and valid gesture segment; This indicates the label actually used to update the incremental adaptation state; in step S5, the predicted category with the highest posterior probability is selected. As a pseudo-label, the artificial truth label is obtained in step S7. Each type supports a fixed memory capacity R, so the adaptation state will not grow indefinitely with the length of the data stream.

[0062] S4. The drift detection module calculates the maximum class posterior probability, the confidence margin between the first class and the second class, and the posterior entropy based on the class posterior probability output in step S3. It then uses the empirical quantiles of the reliable calibration data collected in step S1 to determine the adaptive thresholds, including the high maximum class posterior probability threshold, the high confidence margin threshold, the low maximum class posterior probability threshold, the low confidence margin threshold, and the high entropy threshold. Finally, it establishes a joint uncertain state model and its time de-jittering conditions.

[0063] Furthermore, the expression for establishing the joint uncertain state model in step S4 is as follows:

[0064] (6)

[0065] in, Let k be the joint uncertain state of the k-th stable and valid gesture segment. Let be the maximum class posterior probability of the k-th stable and valid gesture segment. This is a low maximum class posterior probability threshold. The logical AND / conjunction symbols, For the confidence edge of the k-th stable and valid gesture segment, For low-confidence edge thresholds and high-entropy thresholds, The posterior probability vector of the k-th stable and valid gesture segment entropy, It is a high-entropy threshold;

[0066] Time-based de-shaking conditions are In continuous Continuous capture of true values ​​from stable and effective gesture segments. =1, indicating a drift candidate. The number of consecutive times the time-de-jitter condition is applied.

[0067] Furthermore, the formulas for calculating the maximum class posterior probability, the confidence margin between the first and second classes, and the posterior entropy are as follows:

[0068] (4)

[0069] in, Represents the maximum class posterior probability of the k-th stable and valid gesture fragment; This represents the predicted category of the k-th stable and valid gesture segment; This represents the maximum posterior probability other than the predicted category, i.e., the posterior probability of the second largest category. Represents the confidence margin of the k-th stable and valid gesture segment; The Shannon entropy represents the posterior probability vector of the k-th stable and valid gesture segment; Let represent the total number of activity categories at time t. For items with a posterior probability of 0, their entropy contribution is defined. ; Indicates the confidence margin;

[0070] (5)

[0071] in, This represents the qth empirical quantile of the corresponding statistic in the trusted calibration data; the trusted calibration data comes from offline training before deployment or the initial calibration phase of first use, and is not data collected by additional sensors. and These represent the high maximum class posterior probability threshold and the high confidence margin threshold, respectively. , and These represent the low maximum class posterior probability threshold, the low confidence margin threshold, and the high entropy threshold, respectively; each quantile is determined by calibration data before deployment and can be refreshed at low frequencies using recently validated high-confidence samples.

[0072] Furthermore, the maximum class posterior probability is not higher than the low threshold, the confidence margin is not higher than the low margin threshold, and the posterior entropy is not lower than the high entropy threshold if and only if... Take the truth value. Only... The system only considers a candidate for drift after continuously taking true values ​​from K consecutive stable and valid gesture segments, thereby avoiding frequent manual queries triggered by instantaneous sensor noise or single erroneous predictions.

[0073] S5. Based on the adaptive threshold obtained in step S4, the prototype memory module accepts the predicted label as a pseudo-label for unlabeled stable and valid gesture segments that simultaneously satisfy the high maximum class posterior probability threshold and the high confidence margin threshold, calculates the prototype update coefficient, and updates the corresponding class prototype, class-by-class confidence statistic and class-by-class finite support memory in a closed manner, and outputs the current predicted label confirmed by the pseudo-label admission.

[0074] Furthermore, in step S5, when an unlabeled stable and valid gesture fragment satisfies the high maximum class posterior probability threshold and the high confidence margin threshold, the predicted label is accepted as a pseudo-label; the prototype update coefficient is calculated based on the similarity between the latent features of the stable and valid gesture fragment and the current prototype of the corresponding category of the predicted label, and the corresponding category prototype, class-by-class confidence statistics, and class-by-class finite support memory are updated in a closed manner. The latent features that have passed the admission are written into the finite support memory of the corresponding category as support anchors, and the current predicted label confirmed by the pseudo-label admission is output.

[0075] Furthermore, the specific implementation method of step S5 includes the following steps:

[0076] The system only uses predicted labels as pseudo-labels for unlabeled samples that meet the high-confidence admission criteria, and updates the prototype, limited support memory, and class-by-class confidence statistics using a similarity-aware update rate. The formula for calculating the similarity-aware update rate is as follows:

[0077] (7)

[0078] in, This represents the prototype update coefficient of the current sample; Indicates cosine similarity; Indicates the prediction category The current prototype; γ represents the base update rate; This indicates the sensitivity to similarity decay; the lower the similarity, the more exponentially the update coefficient decreases.

[0079] The system accepts pseudo-labels generated by the classifier under the following conditions:

[0080] (8)

[0081] The system only accepts pseudo-labels generated by the classifier when both the high maximum confidence and high confidence margin conditions of equation (8) are met simultaneously. Prototype similarity is only used to adjust the update magnitude and is not used as the sole basis for pseudo-label admission, thereby reducing the self-reinforcement of outdated prototypes on incorrect categories.

[0082] The confidence statistic for the predicted category is updated using an exponential moving average, expressed as:

[0083] (9)

[0084] in, Historical retention factor;

[0085] The lower bound is used to prevent the threshold from decreasing prematurely due to consecutive low-quality samples. The expression for a single-pass prototype update is:

[0086] (10)

[0087] The latent vectors for admission are simultaneously stored in the finite support memory of the corresponding category; when the capacity reaches R, the oldest or least reliable anchor is discarded. For categories with multiple user-specific patterns, consistency can be determined by combining the most recent prototype and the most recent support anchor.

[0088] Furthermore, in the high-confidence update path, pseudo-labels are only accepted when stable and valid gesture fragments satisfy equation (8). Subsequently, the similarity-aware update rate is calculated according to equation (7), and the class-wise confidence statistics, class prototypes, and limited support memory are updated according to equations (9) and (10), while the current predicted label is output. In the normal output path, the fragment neither satisfies the high-confidence admission condition nor forms a continuous drift. The system does not modify any adaptation state and directly outputs the current predicted label.

[0089] Furthermore, in the persistent uncertainty path, the joint uncertainty state defined by equation (6) needs to be satisfied K times consecutively, and further checked by the token balance and cooling interval in equation (14) before triggering a manual truth request. When the manual label belongs to an existing category, prototype overlay is performed and the corrected category is output; when it belongs to an unknown category, a temporary new category prototype is initialized, and the category is activated after manual confirmation of the anchor point or repeated high-confidence matching, and then the new category label is output. Thus, all three paths produce recognition results, but only the high-confidence path and the budget-controlled manual truth path will change the incremental adaptation state.

[0090] S6. Based on the adaptive threshold obtained in step S4, the manual correction module determines a stable and valid gesture segment as a drift candidate when it simultaneously satisfies the low maximum class posterior probability threshold, the low confidence edge threshold, and the high entropy threshold, and the joint uncertain state is satisfied continuously a predetermined number of times. When the query token and the adaptive cooling condition are satisfied at the same time, the module requests the manual truth value through Bluetooth Low Energy and outputs the manual truth value request and the state of the queried segment.

[0091] Furthermore, the condition for the artificial truth-triggered state in step S6 is:

[0092] (14)

[0093] in, This is a manually triggered truth value state. For the k-th stable and valid gesture segment to continuously satisfy The number of stable and effective gesture segments, Let be the number of remaining query tokens corresponding to the k-th stable and valid gesture fragment. This represents the number of decision-making steps since the last manual query. This represents the cooling interval corresponding to the k-th stable and effective gesture segment.

[0094] Furthermore, the specific implementation method of step S6 includes the following steps: When the uncertain state continues to meet the time debouncing condition, the system then combines the query token balance and the cooling interval to determine whether to request a manual truth value. The expression for querying the token balance is:

[0095] (12)

[0096] in, This indicates the number of valid gesture segments that have been processed within the current adaptation range; This represents the percentage of the query budget between 0 and 1. This indicates the maximum number of manual truth queries allowed within this adaptation range, determined by... The product of H and rounded up is obtained.

[0097] The expression for calculating the cooling interval is:

[0098] (13)

[0099] in, Indicates the basic cooling interval; This represents the number of query tokens remaining at decision step k; Indicates the maximum number of queries for the current adaptation range; Indicates recent Query density within each decision-making step; and These represent the budget consumption adjustment coefficient and the local query density adjustment coefficient, respectively. and These represent the lower and upper limits of the cooling interval, respectively; clip(·) represents the clipping function. Here, I represents the interval variable to distinguish it from the number of activity categories. And confidence statistics. The less remaining budget or the more frequent the recent queries, the longer the allowed interval for the next query.

[0100] Furthermore, This indicates an artificial truth-testing trigger state; an artificial truth-testing request is triggered only when all of the above conditions are met simultaneously. Upon triggering, a token is consumed, and low-dimensional latent features and predicted states are uploaded via Bluetooth Low Energy, instead of the complete original sensor sequence.

[0101] S7. The new class extension module judges the artificial truth value obtained in step S6. When the artificial truth value belongs to an existing class, it overwrites and updates the corresponding class prototype. When the artificial truth value belongs to a new class outside the current vocabulary, it is marked as a temporary class, and the temporary class prototype, support anchors in class-by-class limited support memory, and confidence statistics are initialized.

[0102] Furthermore, in step S7, when the artificial truth value belongs to a new category outside the current vocabulary, the new category is marked as a temporary category, and the temporary category prototype, support anchors in the class-by-class limited support memory, and confidence statistics are initialized with the potential features of the queried fragment; after obtaining sufficient artificial truth value confirmation anchors or repeating high-confidence matching, the temporary category is converted into an active category.

[0103] Furthermore, the specific implementation method of step S7 includes the following steps: when the artificial truth value belongs to an existing category, perform prototype coverage correction of formula (15); when the artificial truth value does not belong to the current active vocabulary, perform new category initialization of formula (16), and convert the temporary category to the active category after obtaining sufficient confirmation.

[0104] (15)

[0105] Among them, artificial truth labels Prototype overwrite update when it belongs to an existing category This represents the artificial truth absorption factor; the smaller the absorption factor, the stronger the effect of the new sample on the relocation of the prototype; the larger the absorption factor, the more historical prototype structure is retained.

[0106] (16)

[0107] in, This represents the artificial truth label for the k-th queried gesture fragment; when this label does not belong to the activity vocabulary at time t... Add it to the vocabulary list for the next moment. and with potential features confirmed by humans Initialize the new category prototype. The system also initializes the finite support memory for the category with the latent feature and initializes the corresponding confidence statistic based on the lower bound of the base confidence or the artificial truth prior.

[0108] New categories are initially labeled as temporary categories. They are only promoted to fully active categories when a sufficient number of human truth-verifying anchors are subsequently obtained, or when repeated high-confidence matches are obtained under the same strict admission conditions. Autonomous pseudo-labels cannot create new categories independently to prevent erroneous categories from being generated by transient drift or noise.

[0109] Furthermore, the high-confidence closed-form update in step S5, the artificial truth triggering in step S6, and the artificial truth correction in step S7 are implemented on the microcontroller side in a capacity-bounded state, expressed as:

[0110] (11)

[0111] in, d represents the number of activity categories at time t; d represents the dimension of latent features; R represents the upper limit of support anchors per category. Since R is a fixed value, the storage requirement only increases linearly with the number of activity categories, and does not increase with the total number of historical samples.

[0112] The glove-side uses a fixed feature extractor and base classifier by default, performing only prototype updates, supporting memory replacement and confidence statistics updates, and does not store complete training samples, backpropagation computation graphs, gradients, optimizer states, or Fisher information matrices of the same scale as the full network parameters. This allows for pre-allocation of adaptation state memory outside of the firmware, communication protocol stack, and inference working area.

[0113] Furthermore, in the optional mobile backup path, the feature extractor of the selected recognition branch in step S2 is frozen, and the lightweight classifier head is refreshed only using the latent vectors that have been manually verified or have high confidence admission; the classifier head parameters are the weight parameters and bias parameters in the basic classifier that map the latent features to the posterior probabilities of each gesture category; after the refresh is completed, the updated classifier head parameters are asynchronously sent back to the glove-side device, and the loss function in the refresh is:

[0114] (17)

[0115] The cross-entropy term refreshes the classifier head using artificial ground truth or accepted pseudo-labels, and the regularization term restricts the new classifier parameters from deviating from the old parameters; λ is the regularization weight; this process freezes the feature extractor and is executed asynchronously only when the mobile device is active and has accumulated enough reliable latent vectors, without changing the default glove-side closed update path.

[0116] Example 2:

[0117] A lightweight semi-supervised incremental gesture recognition system for resource-constrained data gloves implements the lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves described in Example 1, including a branch routing module, a semantic stabilization module, a drift detection module, a prototype memory module, a manual correction module, and a new class extension module;

[0118] The smart data glove collects gesture data and inputs it into the branch routing module. The branch routing module is connected to the semantic stabilization module, which is connected to the drift detection module. The drift detection module is connected to the prototype memory module and the manual correction module, respectively. The manual correction module is connected to the new class extension module.

[0119] Furthermore, the smart data glove has stretch sensors installed at the five finger positions, and a six-axis inertial measurement unit containing a three-axis accelerometer and a three-axis gyroscope is installed on the back of the hand, forming an eleven-channel synchronous sensor sequence; the glove side is equipped with an ESP32-S3 microcontroller, 512KB on-chip SRAM, non-volatile memory and a low-power Bluetooth communication module.

[0120] The verification results obtained from the experiments in this embodiment are as follows: Table 1 shows the peak state storage and update characteristics of different adaptation methods; Table 2 shows the incremental adaptation effect under typical deployment scenarios; and Table 3 shows the new category expansion and update scheduling effect.

[0121] Table 1

[0122]

[0123] Table 2

[0124]

[0125] Table 3

[0126]

[0127] The experimental results above validate the effectiveness of bounded prototype updates, label-free personalization, minimal manual truth correction, new category expansion, and drift-aware scheduling. The OPPORTUNITY experiment was used to validate the general adaptability across subject distribution shifts and is not intended as a direct comparison with the official challenge protocol for this dataset.

[0128] This embodiment achieves the following technical effects:

[0129] (1) In scenarios such as long-term use of data gloves, cold start of new users, re-wearing, and changes in action habits, the feature extractor and basic classifier after deployment are frozen, and online personalization is achieved only through the bounded adaptation state composed of category prototypes, class-by-class limited support anchors, and class-by-class confidence statistics.

[0130] (2) Select either a five-channel pose multilayer perceptron branch or an eleven-channel INT8 one-dimensional convolution branch based on the static or dynamic gesture type, and unify the outputs of the two branches into a heterogeneous branch recognition interface that includes latent features, posterior probabilities, and predicted labels.

[0131] (3) Before incremental update, perform short interval segment merging with the same label and splitting of excessively long segments based on confidence bimodal trajectory to reduce the impact of segmentation boundary error on pseudo-label and drift detection.

[0132] (4) Use the maximum class confidence, the confidence margin between the first and second classes and the posterior entropy together, and determine the high confidence admission threshold, low confidence drift threshold and high entropy threshold respectively based on the empirical quantile of the credible calibration data.

[0133] (5) Pseudo-labels are only accepted when both high confidence and high confidence margin are satisfied; prototype similarity is only used to calculate the update coefficient of exponential decay and to update the category prototype and class-by-class support memory with fixed capacity through closed weighting.

[0134] (6) Drift candidates are only confirmed when the joint conditions of low confidence, low confidence edge and high entropy are reached a predetermined number of times, so as to avoid unnecessary updates triggered by instantaneous noise.

[0135] (7) The manual truth cooldown interval is adjusted by using query token budget, recent query density and remaining token ratio, and manual labeling is requested only when there is persistent uncertainty, there are remaining tokens and the cooldown ends.

[0136] (8) When the artificial truth value belongs to an existing category, the corresponding prototype is overwritten and updated; when the artificial truth value belongs to an unknown category, a new category prototype, limited support memory and confidence statistics are initialized, and the creation of incorrect categories is prevented through the confirmation mechanism from temporary category to active category.

[0137] (9) Separate the real-time closed update path on the glove side from the optional mobile classifier head refresh path; the mobile terminal only receives low-dimensional latent features, freezes the feature extractor, and performs low-frequency asynchronous refresh of the classifier head with old parameter constraints.

[0138] (10) The complete lightweight semi-supervised incremental gesture recognition process and its system module combination consisting of the above heterogeneous branch reasoning, semantic stability, quantile uncertainty gating, bounded prototype updating, token budget manual correction and new category prototype extension.

[0139] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves, characterized in that, Includes the following steps: S1. Collect reliable calibration data from the smart data glove, use the smart data glove to collect gesture data, preprocess to obtain valid gesture fragments and their static type labels or dynamic gesture type labels, and obtain preprocessed gesture data; S2. The preprocessed gesture data obtained in step S1 is input into the branch routing module for static branching or dynamic branching to obtain latent features, class posterior probabilities and predicted labels. S3. Input the latent features, class posterior probabilities and predicted labels obtained in step S2 into the semantic stabilization module for semantic stabilization processing to obtain the latent features, class posterior probabilities and predicted labels corresponding to stable and valid gesture segments; S4. The drift detection module calculates the maximum class posterior probability, the confidence margin between the first class and the second class, and the posterior entropy based on the class posterior probability output in step S3. It then uses the empirical quantiles of the reliable calibration data collected in step S1 to determine the adaptive thresholds, including the high maximum class posterior probability threshold, the high confidence margin threshold, the low maximum class posterior probability threshold, the low confidence margin threshold, and the high entropy threshold. Finally, it establishes a joint uncertain state model and its time de-jittering conditions. S5. Based on the adaptive threshold obtained in step S4, the prototype memory module accepts the predicted label as a pseudo-label for unlabeled stable and valid gesture segments that simultaneously satisfy the high maximum class posterior probability threshold and the high confidence margin threshold, calculates the prototype update coefficient, and updates the corresponding class prototype, class-by-class confidence statistic and class-by-class finite support memory in a closed manner, and outputs the current predicted label confirmed by the pseudo-label admission. S6. Based on the adaptive threshold obtained in step S4, the manual correction module determines a stable and valid gesture segment as a drift candidate when it simultaneously satisfies the low maximum class posterior probability threshold, the low confidence edge threshold, and the high entropy threshold, and the joint uncertain state is satisfied continuously a predetermined number of times. When the query token and the adaptive cooling condition are satisfied at the same time, the module requests the manual truth value through Bluetooth Low Energy and outputs the manual truth value request and the state of the queried segment. S7. The new class extension module judges the artificial truth value obtained in step S6. When the artificial truth value belongs to an existing class, it overwrites and updates the corresponding class prototype. When the artificial truth value belongs to a new class outside the current vocabulary, it is marked as a temporary class, and the temporary class prototype, support anchors in class-by-class limited support memory, and confidence statistics are initialized.

2. The lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 1, characterized in that, The static branching and dynamic branching processes in step S2 each include an offline trained feature extractor and a base classifier. For static branching, the feature extractor and the base classifier are respectively composed of the hidden layer and the output layer of a lightweight multilayer perceptron (MLP). For dynamic branching, the feature extractor and the base classifier are respectively composed of the convolutional backbone and the output classification layer of an INT8 quantized one-dimensional convolutional neural network.

3. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 2, characterized in that, The semantic stability processing in step S3 includes merging short-interval segments with the same label, splitting excessively long bimodal segments, and correcting segment boundary artifacts. When adjacent valid gesture segments have the same predicted label and the time interval is not greater than a preset merging threshold, adjacent valid gesture segments are merged. When the dominant category confidence trajectory of an excessively long valid gesture segment is bimodal and the valley value is lower than a preset confidence threshold, the excessively long valid gesture segment is split. Static branching or dynamic branching is re-executed on the merged or split segments to obtain the latent features, category posterior probabilities, and predicted labels corresponding to stable valid gesture segments.

4. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 3, characterized in that, The expression for establishing the joint uncertain state model in step S4 is as follows: in, Let k be the joint uncertain state of the k-th stable and valid gesture segment. Let be the maximum class posterior probability of the k-th stable and valid gesture segment. This is a low maximum class posterior probability threshold. For logic and symbols, For the confidence edge of the k-th stable and valid gesture segment, For low confidence edge threshold, The posterior probability vector of the k-th stable and valid gesture segment entropy, It is a high-entropy threshold; Time-based de-shaking conditions are In continuous Continuous capture of true values ​​from stable and effective gesture segments. It is judged as a drift candidate, among which The number of consecutive times the time-de-jitter condition is applied.

5. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 4, characterized in that, In step S5, when an unlabeled, stable, and valid gesture fragment satisfies the high maximum class posterior probability threshold and the high confidence margin threshold, the predicted label is accepted as a pseudo-label. Based on the similarity between the latent features of the stable and valid gesture fragment and the current prototype of the corresponding category of the predicted label, the prototype of the corresponding category, the class-by-class confidence statistic, and the class-by-class finite support memory are updated in a closed manner. The latent features that have passed the admission are written into the finite support memory of the corresponding category as support anchors, and the current predicted label confirmed by the pseudo-label admission is output.

6. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 5, characterized in that, The condition for the artificial truth-triggered state in step S6 is: in, This is a manually triggered truth value state. For the k-th stable and valid gesture segment to continuously satisfy The number of stable and effective gesture segments, Let be the number of remaining query tokens corresponding to the k-th stable and valid gesture fragment. This represents the number of decision-making steps since the last manual query. This represents the cooling interval corresponding to the k-th stable and effective gesture segment.

7. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 6, characterized in that, In step S7, when the artificial truth value belongs to a new category outside the current vocabulary, the new category is marked as a temporary category, and the temporary category prototype, support anchors in the class-by-class limited support memory, and confidence statistics are initialized with the potential features of the queried fragment. After obtaining sufficient artificial truth value confirmation anchors or repeating high-confidence matching, the temporary category is converted into an active category.

8. A lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves according to claim 7, characterized in that, In the optional mobile backup path, the feature extractor of the selected identification branch in step S2 is frozen, and the lightweight classifier head is refreshed only using latent vectors that have been manually verified or have high confidence admission. The classifier head parameters are the weight and bias parameters in the basic classifier that map latent features to the posterior probabilities of each gesture category. After the refresh is complete, the updated classifier header parameters will be asynchronously sent back to the glove-side device.

9. A lightweight semi-supervised incremental gesture recognition system for resource-constrained data gloves, implementing the lightweight semi-supervised incremental gesture recognition method for resource-constrained data gloves as described in any one of claims 1-8, characterized in that, It includes a branch routing module, a semantic stabilization module, a drift detection module, a prototype memory module, a manual correction module, and a new class extension module; The smart data glove collects gesture data and inputs it into the branch routing module. The branch routing module is connected to the semantic stabilization module, which is connected to the drift detection module. The drift detection module is connected to the prototype memory module and the manual correction module, respectively. The manual correction module is connected to the new class extension module.

10. A lightweight semi-supervised incremental gesture recognition system for resource-constrained data gloves according to claim 9, characterized in that, The smart data glove has stretch sensors installed at the five finger positions, and a six-axis inertial measurement unit containing a three-axis accelerometer and a three-axis gyroscope is installed on the back of the hand, forming an eleven-channel synchronous sensor sequence; the glove side is equipped with an ESP32-S3 microcontroller, 512KB on-chip SRAM, non-volatile memory and a low-power Bluetooth communication module.

Images