Real-time gesture recognition method and system based on multi-feature fusion

Abstract

This invention discloses a real-time gesture recognition method and system based on multi-feature fusion, comprising: acquiring an image sequence containing a hand; performing hand skeleton detection on the current frame image, extracting hand key points, and determining an initial Region of Interest (ROI) for the hand based on the hand key points; performing superpixel segmentation within the initial ROI to generate a boundary mask for the hand region, and calculating the region confidence based on the deviation between the superpixel segmentation result and the initial ROI to assist in verifying the validity of the hand region; constructing a skin color template based on skin color features within the initial ROI, and using the skin color template to perform target tracking on subsequent frame images to dynamically update the hand ROI; determining the finger extension state using the relative positional relationship between hand key points, and outputting the gesture recognition result. This invention achieves an accuracy of 93% in partially occluded scenes and 95% in backgrounds with similar skin tones, meeting the real-time and accurate interaction requirements of consumer-grade terminals.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction technology, and in particular to a real-time gesture recognition method and system based on multi-feature fusion. It has applications in human-computer interaction scenarios such as medical care, industrial control, and smart homes. Background Technology

[0002] With the widespread adoption of smart terminals (such as smartphones, VR devices, and self-service kiosks) in fields such as healthcare, industrial control, and smart homes, natural human-computer interaction has become a core direction for improving service quality and user experience. Gestures, as an intuitive, natural, and contactless interaction method, are in increasingly urgent demand.

[0003] Currently, there are several ways to implement gesture recognition:

[0004] 1. Traditional computer vision gesture recognition

[0005] The hand region is extracted using a skin color thresholding method, and edge detection is used to distinguish the hand from the background.

[0006] Limitations: Relying solely on a single skin color feature results in a misclassification rate exceeding 40% for hand areas against backgrounds with similar skin tones; when the hand is partially occluded, the broken contour leads to a gesture misclassification rate exceeding 68%; and the skin color threshold fails under varying lighting conditions, causing recognition to interrupt. The root cause lies in the single feature dimension and the lack of multi-dimensional cross-validation.

[0007] 2. Sensor-based solutions

[0008] Electromyography or motion signals are collected by wearing devices such as EMG sensors and data gloves.

[0009] Disadvantages: The sensors are expensive (EMG modules cost approximately 500 RMB each), require close contact with the skin for comfortable wear, and have limited transmission distance (≤5 meters) while wireless solutions have latency exceeding 100ms. The root cause lies in the reliance on physical contact data collection, which contradicts the low-cost, contactless requirements of consumer-grade scenarios.

[0010] 3. Deep Learning Solution

[0011] Gesture features are learned end-to-end using deep networks such as CNN and LSTM.

[0012] Drawbacks: Requires massive amounts of labeled data for training, resulting in high labeling costs; large model parameters, easily triggering memory overflow when deployed on ordinary PCs; poor model generalization, with accuracy dropping by 25% when switching users (e.g., from adults to children). The root cause lies in the high model complexity and strong data dependency, making it impossible to achieve highly generalized real-time inference on resource-constrained edge devices.

[0013] 4. Existing multimodal fusion solutions

[0014] Try to simply overlay visual features with sensor data or different visual algorithms.

[0015] Defects: The "vision + sensor" fusion suffers from time asynchrony (latency difference > 30ms), causing lag in gesture recognition; the "deep learning + traditional vision" fusion lacks a calibration mechanism, and tracking offsets in one module can directly contaminate subsequent features. The root cause lies in simply stacking technologies without designing collaborative logic for feature complementarity and error calibration. Summary of the Invention

[0016] This invention addresses the problems of poor robustness in complex scenarios, reliance on dedicated hardware, and lack of collaborative calibration in simple feature overlay, making it difficult to achieve low-cost, high-generalization, real-time accurate gesture recognition on ordinary terminals. It proposes a real-time gesture recognition method and system based on multi-feature fusion. Through MediaPipe Hands' high-precision hand keypoint detection, superpixel segmentation contour optimization, and CamShift back-projection skin color tracking algorithm, it solves the robustness problems of traditional visual methods under physical occlusion, background color interference, inaccurate gestures, and finger occlusion, while achieving real-time processing on ordinary CPUs. Experimental results show that the accuracy of this invention is close to 100% in ideal environments, 93% in partially occluded scenarios, 95% in backgrounds with similar skin colors, and 92% in scenarios with hand pointing deviation / irregular finger shapes. The average processing time per frame is 78.52ms, with a frame rate of 12.74FPS, meeting the real-time and accurate interaction requirements of consumer terminals.

[0017] To achieve the above objectives, the present invention adopts the following technical solution:

[0018] This invention proposes a real-time gesture recognition method based on multi-feature fusion, comprising:

[0019] Obtain an image sequence containing hands;

[0020] Perform hand skeleton detection on the current frame image, extract hand key points, and determine the initial region of interest (ROI) of the hand based on the hand key points;

[0021] Superpixel segmentation is performed within the initial ROI to generate a boundary mask for the hand region. Based on the deviation between the superpixel segmentation result and the initial ROI, the region confidence is calculated to help verify the validity of the hand region.

[0022] A skin color template is constructed based on the skin color features within the initial ROI, and the skin color template is used to perform target tracking on subsequent frame images to dynamically update the hand ROI;

[0023] By utilizing the relative positional relationships between key points on the hand, the finger extension state is determined, and the gesture recognition result is output.

[0024] Furthermore, the method further includes at least one of the following calibration steps:

[0025] When the confidence level of a region falls below a preset confidence threshold, the initial ROI is readjusted.

[0026] When the area difference between the region covered by the boundary mask and the initial ROI exceeds a preset ratio threshold, the initial ROI is readjusted.

[0027] When the deviation between the center position of the dynamically updated hand ROI and the center position of the hand skeleton in the current frame exceeds a preset distance threshold, the target tracking area of ​​the skin color template is reinitialized using the hand key points in the current frame.

[0028] Furthermore, the hand skeleton detection uses the MediaPipe Hands framework to extract the two-dimensional or three-dimensional coordinates of 21 key hand points.

[0029] Furthermore, the determination of the initial region of interest (ROI) of the hand based on hand key points includes:

[0030] Calculate the minimum bounding rectangle of all hand keypoints in the image coordinate system;

[0031] The initial ROI is generated by expanding the boundary of the minimum enclosing rectangle outward by a preset pixel value.

[0032] Furthermore, performing superpixel segmentation within the initial ROI includes:

[0033] Convert the initial ROI from the RGB color space to the LAB color space;

[0034] In the LAB color space, the SLICO algorithm is used for superpixel segmentation. Superpixel blocks are generated through iterative clustering, and the boundaries between superpixel blocks are extracted to form the boundary mask.

[0035] Furthermore, the calculation of region confidence based on the deviation between the superpixel segmentation result and the initial ROI includes:

[0036] Calculate the expected number of superpixels based on the area of ​​the initial ROI;

[0037] Count the actual number of superpixels generated;

[0038] The deviation between the actual number of superpixels and the expected number of superpixels is calculated; the smaller the deviation, the higher the region confidence.

[0039] Furthermore, the step of constructing a skin color template based on skin color features within the initial ROI, and using the skin color template to perform target tracking in subsequent frame images, includes:

[0040] Generate a rectangular mask within the initial ROI that retains only the hand area, and calculate the skin color histogram of the pixels within the rectangular mask in the HSV color space as the skin color template;

[0041] For subsequent frames, the entire frame image is converted to the HSV color space, and a back projection map is calculated based on the skin color histogram.

[0042] The back-projected image is input into the CamShift tracking algorithm to iteratively calculate the probability density peak, thereby dynamically updating the position and size of the hand ROI.

[0043] Furthermore, the method of determining the finger extension state by utilizing the relative positional relationship between key points on the hand includes:

[0044] The 21 key points are divided into three categories: fingertip, knuckle, and palm base.

[0045] For the thumb, whether it is extended is determined by whether the difference between the normalized x-coordinate of the key point of its fingertip and the key point of the corresponding palm base is greater than the first threshold.

[0046] For any finger from the index finger to the little finger, determine whether it is extended based on whether the difference between the normalized y-coordinate of its fingertip key point and the corresponding phalanx key point is greater than the second threshold.

[0047] Further, the first threshold is a value between 0.015 and 0.025, and the second threshold is a value between 0.025 and 0.035.

[0048] Furthermore, the output gesture recognition result also includes determining at least one of the following gestures:

[0049] Clenched fist gesture: The normalized y-coordinates of all fingertip key points are greater than the normalized y-coordinates of the corresponding knuckle key points, and the normalized x-coordinates of each fingertip key point are within the preset range of the normalized x-coordinates of the corresponding palm root key points.

[0050] OK gesture: The normalized Euclidean distance between the key points of the thumb tip and the key points of the index finger tip is less than the third threshold, and the other fingers meet the judgment conditions of the fist gesture.

[0051] Furthermore, it also includes support for simultaneous recognition of both hands:

[0052] The maximum number of inspection hands is set to two;

[0053] Based on the detected key points of the hands of the two hands, the initial ROI determination, superpixel segmentation, skin color template construction and tracking, and gesture recognition steps are performed independently respectively;

[0054] Specifically, when the dynamically updated ROIs of the two hands overlap, a re-detection is triggered.

[0055] Furthermore, the method is executed entirely by the central processing unit (CPU) without the involvement of the graphics processing unit (GPU), and by processing only the image data within the initial ROI and the dynamically updated hand ROI, the memory usage per frame is limited to less than 200KB.

[0056] Furthermore, the superpixel segmentation can also employ the SEEDS superpixel segmentation or LSC superpixel segmentation algorithm; the target tracking can also employ the Mean Shift or KCF tracking algorithm.

[0057] Another aspect of this invention proposes a real-time gesture recognition system based on multi-feature fusion, comprising:

[0058] The image acquisition module is used to acquire image sequences containing the hand;

[0059] The skeleton detection module is used to detect the hand skeleton in the current frame image, extract key points of the hand, and determine the initial region of interest (ROI) of the hand based on the key points of the hand.

[0060] The superpixel segmentation module is used to perform superpixel segmentation within the initial ROI, generate a boundary mask for the hand region, and calculate the region confidence based on the deviation between the superpixel segmentation result and the initial ROI to help verify the validity of the hand region.

[0061] The skin color tracking module is used to construct a skin color template based on the skin color features within the initial ROI, and to use the skin color template to perform target tracking on subsequent frame images in order to dynamically update the hand ROI;

[0062] The gesture classification module is used to determine the finger extension state by utilizing the relative positional relationship between key points on the hand and output the gesture recognition result.

[0063] Furthermore, the system also includes:

[0064] The calibration module is configured to perform at least one of the following operations: when the region confidence is lower than a preset confidence threshold, triggering a readjustment of the initial ROI; when the area difference between the region covered by the boundary mask and the initial ROI exceeds a preset ratio threshold, triggering a readjustment of the initial ROI; when the deviation between the center position of the dynamically updated hand ROI and the center position of the hand skeleton in the current frame exceeds a preset distance threshold, re-initializing the target tracking region of the skin color template using the hand key points of the current frame.

[0065] Furthermore, the skeleton detection module uses the MediaPipe Hands framework to extract the coordinates of 21 hand key points, and generates an initial ROI by calculating the minimum bounding rectangle of all hand key points and extending its boundary outward by a preset pixel value.

[0066] Furthermore, the skin color tracking module is specifically used to: generate a rectangular mask that retains only the hand region within the initial ROI; calculate the skin color histogram of the pixels within the rectangular mask in the HSV color space as the skin color template; for subsequent frames, convert the entire frame image to the HSV color space; calculate the back projection map based on the skin color histogram; input the back projection map into the CamShift tracking algorithm; iteratively calculate the probability density peak value to dynamically update the position and size of the hand ROI.

[0067] Furthermore, the system is deployed on a conventional terminal device without a graphics processing unit (GPU), which includes a central processing unit (CPU) and a monocular RGB camera.

[0068] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the real-time gesture recognition method based on multi-feature fusion as described above.

[0069] Compared with the prior art, the present invention has the following advantages:

[0070] (1) Significantly improved robustness in complex scenes: Through closed-loop calibration of the three features of "skeleton-contour-skin color", the recognition accuracy in typical complex scenes such as partial occlusion, background with similar skin color, and irregular hand shape is better than the existing single feature or deep learning scheme. The accuracy reaches 99.52% in ideal environment, 94.17% in partial occlusion, 95.0% in background with similar skin color, and 93.65% in hand orientation deviation.

[0071] (2) Lightweight design and real-time performance guarantee: It only relies on CPU and ordinary monocular RGB camera, without the need for GPU or dedicated chip, reducing hardware cost by more than 80%; on ordinary PC, the average processing time per frame is 78.5ms and the frame rate is 12.74FPS, which meets the needs of continuous human eye interaction.

[0072] (3) Low cost and high generalization: No need to collect exclusive training data for different users. The accuracy of adaptation to users with different skin color, hand size and age is over 92%, and it can be used out of the box.

[0073] (4) Dynamic Adaptation and Error Compensation: Through regional confidence feedback and tracking center deviation detection, the ROI and tracking model are autonomously calibrated, avoiding the overall collapse caused by the failure of a single module, and significantly improving the stability of the system during long-term operation. Attached Figure Description

[0074] Figure 1 This is one of the flowcharts for a real-time gesture recognition method based on multi-feature fusion according to an embodiment of the present invention;

[0075] Figure 2 This is a second flowchart of a real-time gesture recognition method based on multi-feature fusion according to an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of superpixel segmentation provided in an embodiment of the present invention;

[0077] Figure 4 This is an example of a back projection diagram provided in an embodiment of the present invention;

[0078] Figure 5 This is the Hand Landmarks hand key point detection model provided in the embodiments of the present invention;

[0079] Figure 6 This is a schematic diagram of the architecture of a real-time gesture recognition system based on multi-feature fusion according to an embodiment of the present invention. Detailed Implementation

[0080] For ease of understanding, the following explanations are provided for some of the terms used in the specific embodiments of this invention:

[0081] 1. MediaPipe Hands: A lightweight hand pose estimation framework developed by Google, based on a deep learning model (model size approximately 2MB). It can detect 21 key points of the hand in real time and output 3D coordinates (x / y normalized to 0-1, z is the relative palm root depth). It supports CPU / GPU operation, with a key point detection accuracy of ≥95% in complex backgrounds and a single frame processing time of ≤30ms (CPU).

[0082] 2. Superpixel segmentation: A region partitioning technique in computer vision that divides an image or ROI into multiple consecutive pixel blocks (called "superpixels") that are "similar in color, texture, and brightness". Each superpixel block is a processing unit, which can reduce redundant calculations in pixel-level processing while preserving the region boundary information of the image. Commonly used algorithms include SLICO, SEEDS, LSC, etc.

[0083] 3. SLICO algorithm: short for "Simple Linear Iterative Clustering with Orthogonality constraint", is one of the mainstream superpixel segmentation algorithms. It initializes cluster centers in the LAB color space (arranged at intervals of region_size) and iteratively optimizes the position and range of the cluster centers, so that the superpixel blocks have both "color similarity" and "shape regularity". Its characteristics are fast segmentation speed (single frame time ≤ 20ms for 10 iterations) and high boundary accuracy (overlap rate with manually labeled boundaries ≥ 85%).

[0084] 4. CamShift Algorithm: Short for "Continuously Adaptive Mean Shift", it is a dynamic target tracking algorithm based on color histograms. The core logic is: ① Calculate the color histogram of the target region in the initial frame; ② Generate a target probability map through back projection in subsequent frames; ③ Use the Mean Shift algorithm to find the density peak in the probability map and determine the new position of the target; ④ Adaptively adjust the tracking window size to adapt to target scaling. The processing time per frame is ≤25ms (CPU), making it suitable for tracking dynamic targets such as hands.

[0085] 5. ROI (Region of Interest): A specific area in the image that needs to be processed. In this invention, it is the hand region. By extracting the ROI, the scope of subsequent processing can be reduced from the entire image to the hand area, which greatly reduces the amount of computation, improves the speed of the algorithm, and avoids the interference of background pixels on feature extraction.

[0086] 6. Normalized coordinates: Normalizing image pixel coordinates ( , Converting coordinates to relative coordinates in the 0-1 interval, the conversion formula is as follows: , It can eliminate the influence of different camera resolutions (such as 640×480, 1280×720) and different device screen sizes on the judgment of key point positions, making the algorithm parameters (such as the stretch threshold) universal.

[0087] 7. LAB color space: A device-independent color space consisting of three channels: ① L channel (brightness, 0-100, 0 is black, 100 is white); ② A channel (red-green difference, -128 to 127, negative values ​​are green, positive values ​​are red); ③ B channel (yellow-blue difference, -128 to 127, negative values ​​are blue, positive values ​​are yellow). Its characteristic is that the color difference is closer to human eye perception, making it suitable for scenarios that require precise color differentiation, such as superpixel segmentation and image enhancement.

[0088] 8. HSV Color Space: A color space composed of three channels: Hue, Saturation, and Value. The ranges of each channel are: ①H (0-180, representing the color type, such as 0 for red, 60 for yellow, and 120 for green); ②S (0-255, representing the color purity, 0 for gray and 255 for pure color); ③V (0-255, representing the brightness, 0 for black and 255 for the brightest). It is characterized by its strong robustness to changes in lighting (the brightness of the V channel can be adjusted independently), making it suitable for scenarios such as skin color recognition and target tracking.

[0089] 9. Back-Projection Map: A probability image generated based on a color histogram. The gray value (0-255) of each pixel in the image represents the degree of matching between that pixel and the target color (such as skin color). The higher the gray value, the higher the matching degree. It can transform the problem of finding the target into the problem of finding high gray-scale regions. It is the core input of tracking algorithms such as CamShift and Mean Shift.

[0090] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0091] like Figure 1 , Figure 2 As shown, a real-time gesture recognition method based on multi-feature fusion includes:

[0092] S1: Image Acquisition and Preprocessing

[0093] A sequence of RGB images containing the hand (e.g., 1280×720 resolution) was acquired using a standard monocular RGB camera. Each frame was horizontally flipped to eliminate mirror effects and converted to RGB format to meet the input requirements of the subsequent skeleton detection model.

[0094] S2: Hand skeleton detection and initial ROI generation

[0095] This embodiment uses the MediaPipe Hands hand landmark detection model to extract the 3D normalized coordinates of 21 key points on the hand. By judging the relative positional relationship between the fingertips and joints (e.g., whether the index fingertip is higher than the knuckle), the number of outstretched fingers is initially identified. If no hand is detected, the message "No hand detected" is returned and the system waits for the next frame. Once a hand is detected, the minimum (x_min, y_min) and maximum (x_max, y_max) values ​​of all key points in the x and y directions are calculated to form a minimum bounding rectangle from (x_min, y_min) to (x_max, y_max). Each boundary of this rectangle is then expanded outward by 20 pixels to obtain the initial region of interest (ROI) of the hand from (x_min-20, y_min-20) to (x_max+20, y_max+20), thus reducing the image range for subsequent processing.

[0096] S3: Superpixel segmentation and region confidence calculation

[0097] First, the initial ROI obtained by S2 is converted from the BGR color space to the LAB color space to enhance color discrimination. Then, the SLICO superpixel segmentation interface (cv2.ximgproc.createSuperpixelSLIC) of OpenCV (e.g., OpenCV version 4.12.0) is called, setting the region size (region_size) to 20 and the number of iterations to 10, to perform clustering segmentation, so that each pixel is assigned a corresponding superpixel label. Pixels with labels different from their neighboring pixels are identified as boundary pixels, thereby extracting the boundaries between superpixel blocks to form a binary boundary mask in which boundary pixels are white (255) and non-boundary pixels are black (0). The white pixels (boundaries) in this boundary mask are marked as green ([0,255,0]) on the original ROI to generate a hand region image with contours, such as... Figure 3 As shown, it is used for subsequent visual verification and boundary calibration.

[0098] Simultaneously, by calculating the deviation between the actual number of superpixels and the expected number, a region confidence score (reflecting the integrity and accuracy of the hand region) is generated. This helps verify the effectiveness of the hand region and reduces the impact of interference from non-hand regions on the recognition results.

[0099] As one possible implementation, in this embodiment, the region_size parameter of the SLICO algorithm is set to 20, meaning that each superpixel block is expected to cover an average of 20 × 20 = 400 pixels. Based on this, the expected number of superpixels N_expected is calculated according to the area ROI_Area of ​​the initial ROI:

[0100] N_expected = ROI_Area / 400

[0101] After performing SLICO segmentation on the initial ROI, the actual number of superpixel blocks generated, N_actual, is counted. The deviation ratio is calculated.

[0102] Deviation ratio = |N_actual - N_expected| / N_expected

[0103] The regional confidence level C is defined as follows:

[0104] C = 1 - Deviation Ratio

[0105] When C is lower than the preset threshold (e.g., 0.8, i.e., the deviation ratio exceeds 20%), it is determined that the ROI may contain a large amount of background or incomplete hand area. At this time, S2 is triggered to re-extract the skeleton key points and adjust the ROI.

[0106] As one possible implementation, after generating the superpixel boundary mask, the number of non-zero pixels in the boundary mask is counted as the superpixel coverage area. The difference rate between this area and the initial ROI area is calculated. When the difference rate exceeds a preset threshold (20% in this embodiment), it is determined that there is a significant deviation between the initial ROI and the actual hand contour, triggering the following calibration operations: If the superpixel coverage area is less than 80% of the ROI area, it indicates that the initial ROI contains too much background area, so the bounding rectangle of the hand region is recalculated based on the maximum connected component of the superpixel boundary mask, and the ROI boundary is shrunk; if the superpixel coverage area is close to or equal to the ROI area, but the region confidence is low, it indicates that the hand region may exceed the range of the initial ROI, so the hand skeleton detection step is triggered to re-extract key points and expand the ROI.

[0107] S4: Skin Color Histogram Modeling and CamShift Dynamic Tracking

[0108] Upon initial detection of a hand, a rectangular mask for the hand region is generated using S2 keypoints, retaining only the pixels of the hand region. In the HSV color space, a 2D skin color histogram of the hand region (180 bins in the H channel and 256 bins in the S channel) is calculated as a skin color template to ensure focus on the hand's color features.

[0109] For each subsequent frame, the entire image is converted to the HSV color space, and backprojection is performed based on the skin color histogram to obtain a backprojected image (pixels with high matching degree have high brightness, such as...). Figure 4(As shown). Subsequently, this back-projected image is input into the CamShift algorithm to iteratively find the probability density peak, thereby dynamically updating the position, size, and rotation angle of the hand's ROI to achieve adaptive tracking of hand movement, scaling, and rotation. As one possible implementation, the CamShift termination criterion is set to 10 iterations.

[0110] As one possible implementation, after updating the hand ROI in S4, the center coordinates of the ROI are calculated and compared with the center coordinates of the hand skeleton extracted in the current frame S2. If the Euclidean distance between the two exceeds a preset distance threshold (5 pixels in this embodiment), tracking drift is determined to have occurred. At this time, the CamShift tracking window is forcibly reinitialized using the skeleton keypoint positions of the current frame, that is, the skin color histogram modeling process in S4 is re-executed to eliminate accumulated errors and ensure long-term tracking stability.

[0111] S5: Judging Finger Extension Status and Classifying Gestures

[0112] Based on the coordinates of 21 key hand points obtained by S2 (e.g. Figure 5 As shown in the figure, the finger extension state is determined after normalization.

[0113] Key point classification: The 21 key points are divided into fingertip category (key points 4, 8, 12, 16, 20), knuckle category (key points 3, 7, 11, 15, 19) and palm base category (key points 2, 5, 9, 13, 17).

[0114] It is worth noting that in this embodiment, the 21 key hand points are divided into three levels according to their functions: (1) direct judgment layer, including fingertip key points (4, 8, 12, 16, 20) and knuckle key points (3, 7, 11, 15, 19), used for judging the finger extension state; (2) spatial positioning layer, including wrist root key point (0) and palm root key point (2, 5, 9, 13, 17), used to determine the range of hand ROI and the center position of the skeleton; (3) intermediate reference layer, including thumb metacarpophalangeal joint key point (1) and metacarpophalangeal joint key points from index finger to little finger (6, 10, 14, 18), as connection reference points between fingers and palm, participating in the guarantee of ROI integrity. In the finger extension judgment, the fingertip-knuckle relative relationship with the best discrimination is adopted; in ROI generation and tracking drift calibration, the spatial coordinate information of all key points is comprehensively utilized.

[0115] Thumb identification: The normalized x-coordinate of the fingertip key point 4 is compared with the normalized x-coordinate of the palm root key point 2. If the difference is greater than the first threshold (0.02 in this embodiment), the thumb is determined to be extended. Since the thumb's movement direction is different from the other four fingers, the x-coordinate difference is used for identification.

[0116] Judging from index finger to little finger: For each finger, compare the normalized y-coordinate of the fingertip key point (8 / 12 / 16 / 20) with the normalized y-coordinate of the corresponding knuckle key point (7 / 11 / 15 / 19). If the fingertip y-coordinate is less than the knuckle y-coordinate minus the second threshold (0.03 in this embodiment), it is judged as extension.

[0117] The number of extended fingers is counted, and the predicted gesture category ("1" to "5") is output for each frame. The first threshold ranges from 0.015 to 0.025, and the second threshold ranges from 0.025 to 0.035.

[0118] Specifically, gestures are categorized as follows:

[0119] “1”: Only the index finger is extended;

[0120] “2”: Extend your index and middle fingers;

[0121] “3”: Extend your index, middle, and ring fingers;

[0122] “4”: Extend your index finger, middle finger, ring finger, and little finger;

[0123] “5”: All five fingers fully extended.

[0124] It is worth noting that the process for determining the first and second thresholds is as follows:

[0125] Through testing with 2000 sets of samples (covering different hand sizes and resolutions), the distribution of key point coordinate differences under "extended / bent" states was statistically analyzed based on normalized coordinates, ultimately yielding the following results:

[0126] 1) The first threshold (thumb x-direction threshold) is preferably 0.02: Under this threshold, the accuracy of thumb extension judgment is 97.2% (when the threshold is <0.015, the false positive rate increases to 12% and when it is >0.025, the false negative rate increases to 10%).

[0127] 2) The second threshold (the threshold in the y-direction from index finger to little finger) is preferably 0.03: Under this threshold, the accuracy of finger extension judgment is 96.8% (the false judgment rate is 11% when the threshold is <0.025 and the false negative rate is 9% when the threshold is >0.035).

[0128] Normalized coordinates eliminate the influence of different camera resolutions and different hand sizes, making the threshold universal and eliminating the need for adjustments for specific devices or users.

[0129] Through the above steps S1 to S5, gesture recognition in real-time video streams is achieved. Since this invention only processes the ROI region, the computational load of superpixel segmentation and CamShift tracking is reduced by 90%; the number of SLICO iterations is set to 10 (instead of the higher 15), balancing segmentation accuracy (95%) and time (18.7ms); the CamShift termination criterion is set to 10 iterations to avoid overcomputation; only keypoint coordinates and histogram data are retained, and intermediate images (such as back-projection maps) are not stored, resulting in a memory footprint of <200KB; through "reduced computational range + parameter optimization + data simplification," real-time processing on a regular CPU is achieved, overcoming the contradiction between lightweight design and robustness.

[0130] This invention solves the robustness problems of traditional vision methods in cases of physical occlusion, background color interference, inaccurate gestures, and finger occlusion by using MediaPipe Hands for high-precision hand key point detection, superpixel segmentation contour optimization, and CamShift back projection skin color tracking algorithm. At the same time, it achieves real-time processing on ordinary CPUs.

[0131] Based on the above embodiments, such as Figure 6 As shown, this invention also proposes a real-time gesture recognition system based on multi-feature fusion, comprising:

[0132] The image acquisition module is used to acquire image sequences containing the hand;

[0133] The skeleton detection module is used to detect the hand skeleton in the current frame image, extract key points of the hand, and determine the initial region of interest (ROI) of the hand based on the key points of the hand.

[0134] The superpixel segmentation module is used to perform superpixel segmentation within the initial ROI, generate a boundary mask for the hand region, and calculate the region confidence based on the deviation between the superpixel segmentation result and the initial ROI to help verify the validity of the hand region.

[0135] The skin color tracking module is used to construct a skin color template based on the skin color features within the initial ROI, and to use the skin color template to perform target tracking on subsequent frame images in order to dynamically update the hand ROI;

[0136] The gesture classification module is used to determine the finger extension state by utilizing the relative positional relationship between key points on the hand and output the gesture recognition result.

[0137] Furthermore, the system also includes:

[0138] The calibration module is configured to perform at least one of the following operations: when the region confidence is lower than a preset confidence threshold, triggering a readjustment of the initial ROI; when the area difference between the region covered by the boundary mask and the initial ROI exceeds a preset ratio threshold, triggering a readjustment of the initial ROI; when the deviation between the center position of the dynamically updated hand ROI and the center position of the hand skeleton in the current frame exceeds a preset distance threshold, re-initializing the target tracking region of the skin color template using the hand key points of the current frame.

[0139] Furthermore, the system also includes:

[0140] The image preprocessing module is used to horizontally flip RGB images (1280×720 resolution) captured by ordinary cameras (to avoid mirroring effects) and convert them into RGB format (MediaPipe input requirement).

[0141] Furthermore, the skeleton detection module is specifically used for:

[0142] Based on the MediaPipe Hands framework, the hand keypoint detection model extracts the coordinates of 21 hand keypoints in real time. By determining the relative positional relationship between the fingertips and knuckles (e.g., whether the index finger tip is higher than the knuckle), it initially identifies the number of outstretched fingers. As a fundamental module for gesture recognition, it directly provides the raw prediction results of the number of fingers, ensuring the dominance of core features.

[0143] The input is a preprocessed RGB image. The module performs the following steps:

[0144] 1) Key point detection: Call the MediaPipe Hands interface to output the 3D coordinates of 21 hand key points. If no hand is detected, return the message "No hand detected".

[0145] 1) ROI extraction: Based on the x and y coordinates of the key points, calculate the minimum bounding rectangle of the hand from (x_min-20, y_min-20) to (x_max+20, y_max+20) as the ROI (Region of Interest) of the hand, reducing the amount of subsequent processing of the background area.

[0146] Furthermore, the superpixel segmentation module is specifically used for:

[0147] The SLICO superpixel segmentation algorithm is used to segment the detected hand region (ROI). The region confidence score (reflecting the completeness and accuracy of the hand region) is generated by calculating the deviation between the actual and expected number of superpixels. This helps verify the validity of the hand region and reduces the impact of interference from non-hand regions on the recognition results.

[0148] The following steps are taken to perform superpixel segmentation on the hand ROI:

[0149] 1) Color space conversion: Convert the ROI of the hand from BGR to LAB color space to enhance color differentiation.

[0150] 2) SLICO segmentation: Call the OpenCV cv2.ximgproc.createSuperpixelSLIC interface, set the region size to 20, iterate 10 times, and generate a superpixel boundary mask.

[0151] 3) Contour overlay: Mark the white pixels (boundaries) in the boundary mask as green ([0,255,0]) on the original ROI to generate a hand region image with contours for subsequent visual verification and boundary calibration.

[0152] Furthermore, the skin color tracking module is specifically used for:

[0153] A backprojection algorithm based on skin color histograms, combined with the CamShift tracking algorithm, tracks the hand region in real time. Color matching score (reflecting the consistency between the current region and skin color features) is generated by statistically analyzing the proportion of pixels with high matching scores. This enhances the stability of skin color features and improves tracking robustness in complex backgrounds.

[0154] This module works in conjunction with the skeleton detection module and the superpixel segmentation module to achieve dynamic hand tracking and precise skin color region selection. The core process is as follows:

[0155] 1) Skin color histogram initialization: When the hand is detected for the first time, the minimum bounding box of the hand is calculated based on the 21 key points extracted by MediaPipe Hands and a rectangular mask is generated (only the pixels of the hand area are retained). The skin color histogram in HSV space is calculated as the "skin color template" for subsequent tracking to ensure that the focus is on the color features of the hand.

[0156] 2) Back projection and enhancement: After each frame of image is converted to HSV (more suitable for color analysis), it is substituted into the skin color histogram to generate a back projection map (pixels with high matching degree have high brightness), so that the skin color area is displayed as bright and the background is displayed as dark.

[0157] 3) Dynamic ROI Update: The back-projected image is input into the CamShift algorithm to automatically update the position, size, and rotation angle of the hand ROI (adapting to hand movement, scaling, and rotation). Simultaneously, the superpixel segmentation module enhances the contour of the hand region within the ROI (green boundary) to aid in verifying the accuracy of the tracking area and prevent ROI shift due to background interference. If the hand moves, the ROI automatically follows, avoiding the loss of keypoint detection.

[0158] Furthermore, the gesture classification module is specifically used for:

[0159] Based on the skeleton features output by MediaPipe, the finger extension state is determined and counted. The steps are as follows:

[0160] 1) Judging by finger extension:

[0161] a) Thumb: Determine if the x-coordinate of the fingertip (4) is greater than the x-coordinate of the palm root (2) plus 0.02 (because the thumb extends in a different direction than other fingers, the horizontal direction needs to be determined separately);

[0162] b) Index finger to little finger: Determine whether the y-coordinate of the fingertip (8 / 12 / 16 / 20) is less than the y-coordinate of the knuckle (7 / 11 / 15 / 19) minus 0.03 (judged by vertical extension).

[0163] 2) Gesture Classification:

[0164] “1”: Only the index finger is extended;

[0165] “2”: Extend your index and middle fingers;

[0166] “3”: Extend your index, middle, and ring fingers;

[0167] “4”: Extend your index finger, middle finger, ring finger, and little finger;

[0168] “5”: All five fingers fully extended.

[0169] 3) Output results: Output the predicted gesture category (1~5) for each frame.

[0170] Furthermore, the system is deployed on a conventional terminal device without a graphics processing unit (GPU), which includes a central processing unit (CPU) and a monocular RGB camera.

[0171] This invention employs a three-feature closed-loop calibration approach—"skeleton-contour-skin color"—to achieve superior recognition accuracy compared to existing single-feature or deep learning solutions in typical complex scenarios such as partial occlusion, backgrounds with similar skin color, and irregular hand shapes. Through region confidence feedback and tracking center deviation detection, it achieves autonomous calibration of the ROI and tracking model, preventing overall system failure due to single-module malfunction and significantly improving system stability during long-term operation.

[0172] This invention strictly follows the sequence of "skeleton extraction → contour optimization → skin color tracking," with each module providing a baseline for the next and each module correcting errors for the previous one.

[0173] 1) The skeleton detection module (based on MediaPipe Hands) first determines the approximate area of ​​the hand (ROI), avoiding superpixel segmentation of the entire image (reducing computation by 80%).

[0174] 2) The superpixel segmentation module optimizes the hand boundary, providing CamShift with a more accurate skin color area (reducing background pixel interference and improving tracking accuracy by 15%).

[0175] 3) CamShift tracks and corrects keypoint drift in MediaPipe Hands (when the hand moves quickly, the keypoints in MediaPipe Hands are prone to shifting; CamShift uses skin tone-stabilized tracking to calibrate the shift error).

[0176] Complementary features: Skeleton features (structural information), contour features (boundary information), and skin color features (pixel information) cover three layers of features: macro, meso, and micro, solving the limitations of single features (such as skeletons having no boundaries, contours having no structure, and skin color having no shape).

[0177] The dynamic ROI linkage update and error calibration logic of this invention is as follows:

[0178] (1) Initial ROI generation

[0179] The initial ROI is generated by expanding the minimum bounding rectangle of the 21 hand key points extracted by MediaPipe Hands outward by a preset pixel value, ensuring that the initial ROI completely covers the hand area.

[0180] (2) ROI boundary calibration based on superpixel segmentation

[0181] After performing superpixel segmentation within the initial ROI, the accuracy of the ROI is evaluated and calibration is triggered through two complementary mechanisms:

[0182] Region confidence mechanism: The region confidence score is calculated based on the deviation between the actual number of superpixels and the expected number of superpixels estimated based on the ROI area. When the region confidence score is lower than a preset threshold, it indicates that the ROI may be mixed with background or the hand area may be incomplete, triggering the skeleton detection module to re-extract key points and adjust the ROI.

[0183] Area difference mechanism: The effective area covered by the superpixel boundary mask is statistically analyzed. When the difference rate between this area and the initial ROI area exceeds a preset ratio threshold (e.g., 20%), it indicates that there is a significant deviation between the initial ROI and the actual hand contour, triggering ROI boundary adjustment or skeleton re-examination.

[0184] The two mechanisms described above evaluate the quality of ROI from the two dimensions of regularity and coverage of superpixel segmentation, respectively, forming a complementary relationship that ensures that subsequent processing always focuses on the precise hand region.

[0185] (3) Tracking drift calibration based on skeleton feedback

[0186] The CamShift algorithm dynamically tracks subsequent frames based on the skin color histogram, continuously updating the hand's ROI. Simultaneously, after tracking each frame, it calculates the Euclidean distance between the center of the CamShift tracking window and the center of the MediaPipe Hands skeleton keypoints in the current frame. When this distance exceeds a preset threshold, tracking drift is considered to have occurred, and the skin color model and tracking window are reinitialized using the skeleton keypoint positions of the current frame. This mechanism uses skeleton features as anchor points to periodically correct accumulated errors in skin color tracking, preventing hand loss due to error accumulation during long-term tracking.

[0187] (4) Overall effect of dynamic ROI adjustment

[0188] Through the coordinated work of initial ROI generation, superpixel boundary calibration, and tracking drift calibration, this invention achieves frame-by-frame dynamic optimization of the hand ROI, ensuring that each module always processes the accurate hand area, reducing background interference, and improving overall recognition efficiency and accuracy.

[0189] By dynamically adjusting the ROI, we ensure that each module processes the precise hand area, reducing background interference and improving overall efficiency and accuracy.

[0190] To verify the effectiveness of this invention, the following experimental setup was conducted:

[0191] 1. Complex scenarios

[0192] By fusing three features—skeleton, contour, and skin tone—the accuracy rate outperformed existing solutions in four typical complex scenarios. Specific comparative data is as follows (Test environment: standard PC with a Core i5-12400F processor, 1280×720 resolution, 20 testers, 50 samples per person):

[0193] Table 1. Accuracy Comparison in Four Typical Complex Scenarios

[0194]

[0195] As shown in Table 1, the present invention does not require a GPU, and its accuracy in complex scenarios exceeds that of deep learning solutions, and is 15% to 25% higher than that of single feature solutions.

[0196] 2. Terminal compatibility and real-time performance

[0197] This invention only requires a regular CPU (Core i5 or above) + 8GB of memory, without the need for a GPU or dedicated chip, reducing hardware costs by 80% (compared to deep learning solutions that require a GTX 1650 GPU, costing approximately 800 yuan).

[0198] With the test device having an Intel(R) Core(TM) Ultra 9275HX CPU, 16 GB of memory, and no GPU acceleration (simulating a typical consumer terminal), the real-time performance metrics are shown in Table 2.

[0199] Table 2 Real-time performance indicators

[0200]

[0201] In terms of frame rate performance, the average frame rate of this invention is 12.74 FPS, the minimum frame rate is 9.8 FPS (initial histogram initialization), and the maximum frame rate is 15.2 FPS (during stable tracking), which is far higher than the 5 FPS threshold required for continuous human eye interaction, with no obvious stuttering.

[0202] 3. Cost and generalizability

[0203] Hardware cost: Only a regular monocular RGB camera is required (such as a built-in laptop camera, costing 0 yuan; external camera ≤ 100 yuan), with no additional sensor or computing power hardware overhead, reducing the cost by 90% compared to EMG sensor-based solutions.

[0204] Regarding the generalization test (20 testers, covering different dimensions), the specific results are shown in Table 3.

[0205] Table 3 Generalization test results

[0206]

[0207] This invention eliminates the need to collect dedicated training data for different users, making it ready to use out of the box and adaptable to diverse user needs in consumer scenarios.

[0208] 4. Robustness and real-time compatibility

[0209] Existing solutions often suffer from a contradiction: high robustness comes at the cost of poor real-time performance, and good real-time performance comes at the cost of low robustness. This invention achieves a balance through the collaborative operation of three modules:

[0210] Real-time performance guarantee: No complex model calculations are required. Only the coordinates of 21 key points (approximately 252 bytes) and the HS histogram (180×256×4 bytes = 184320 bytes) are stored, with a memory usage of <200KB, avoiding memory overflow.

[0211] Robustness guarantee: The three modules are mutually calibrated (e.g., MediaPipe is used to correct CamShift offset, and ROI is re-extracted when superpixel boundaries are abnormal), forming an error compensation mechanism to avoid the failure of a single module causing the overall recognition to collapse.

[0212] 5. Dynamic Adaptation Aspect

[0213] It supports real-time updates of the skin tone HSV range (based on the pixel distribution of the current hand area), adapting to recognition needs under different lighting and skin tone conditions. It also provides visualization windows for superpixel segmentation and back projection, intuitively displaying intermediate processing results and facilitating quick identification of problems in areas such as region extraction and color tracking.

[0214] Experimental results show that the accuracy of this invention is close to 100% under ideal conditions, 93% in partially occluded scenes, 95% in backgrounds with similar skin tones, and 92% in scenes with hand pointing deviation / irregular finger shape. The average processing time per frame is 78.52ms, and the frame rate is 12.74FPS, which meets the real-time and accurate interaction requirements of consumer terminals.

[0215] As one possible implementation, this embodiment, in addition to the above-mentioned gesture classification, also supports the judgment of fist gestures and OK gestures.

[0216] Clenched fist gesture: The normalized y-coordinate of all fingertip key points is greater than the normalized y-coordinate of the corresponding knuckle key point plus 0.02, and the normalized x-coordinate of each fingertip key point is within the range of ±0.03 of the normalized x-coordinate of the corresponding palm root key point.

[0217] OK gesture: The normalized Euclidean distance between the key points of the thumb tip and the key points of the index finger tip is less than the third threshold (0.05 in this embodiment), and the other fingers (middle finger, ring finger, little finger) meet the above judgment conditions for the fist gesture.

[0218] The specific gesture recognition method is as follows:

[0219] 1. Rules for judging fist gesture: ① The normalized y-coordinate of all fingertip key points (4,8,12,16,20) is greater than the normalized y-coordinate of the corresponding knuckle key points (3,7,11,15,19) + 0.02 (fingers are bent and closed); ② The normalized x-coordinate of all fingertip key points is within the range of ±0.03 of the normalized x-coordinate of the corresponding palm base key points (2,5,9,13,17) (hand is closed); ③ The mean confidence score of key points is ≥0.7.

[0220] 2. OK gesture judgment rules: ① The normalized Euclidean distance between the tip of the thumb (4) and the tip of the index finger (8) is <0.05 (when the two fingers are brought together to form a circle). The Euclidean distance formula is: ② The remaining fingertips (12, 16, 20) satisfy the bending condition of "clenching a fist" (normalized y-coordinate > normalized y-coordinate of the knuckle + 0.02); ③ The mean confidence level of the key points is ≥ 0.7.

[0221] Through the above design, the gesture coverage has been expanded from 5 categories to 7 categories, adapting to more interaction scenarios (such as "OK" triggering a confirmation operation and "clenched fist" triggering a cancellation operation); the performance impact is small: the single frame processing time increases by ≤5ms (only distance calculation and condition judgment are added), and the frame rate on ordinary PCs is still ≥12FPS; the accuracy is stable: the accuracy of "clenched fist" and "OK" gestures is high, basically on par with the accuracy of the original digital gestures (≥93.65%).

[0222] As one possible implementation, based on the above embodiments, this embodiment also supports simultaneous recognition of both hands (adapting to VR / AR two-handed interaction scenarios), specifically including:

[0223] Adjust the "max_num_hands" parameter of MediaPipe from 1 to 2, and add "hand differentiation" and "parallel processing" logic to achieve independent hand gesture recognition.

[0224] Complete implementation:

[0225] 1. Hand differentiation mechanism: ① Position differentiation: determined by the x-coordinate of the ROI of both hands (x < image width / 2 for left hand, x > image width / 2 for right hand); ② Depth differentiation: if the positions overlap (e.g., hands crossed), determined by the z-value of MediaPipe Hands keypoints (the hand with the smaller z-value is closer to the camera); ③ Labeling rules: assigned “L” and “R” labels to the left and right hands respectively;

[0226] 2. Parallel processing flow: ① After image preprocessing, MediaPipe Hands simultaneously outputs 21 key points for both hands (each marked with "L" and "R"); ② Each hand independently executes the "ROI extraction → superpixel segmentation → CamShift tracking → gesture classification" process, sharing image data but not computing resources; ③ Conflict handling: If the overlap area of ​​the ROIs of both hands is greater than 20%, re-detection is triggered to avoid mutual interference;

[0227] 3. Output: Synchronously output the combination of "left hand gesture - right hand gesture".

[0228] The above design expands the application scenarios: it adapts to two-handed collaborative operation in VR / AR (such as "left hand makes a 2, right hand makes a 3" to achieve object scaling) and combined control of smart homes (such as "both hands make a fist" to turn off all devices); and the performance is controllable: the average frame time is 92.3ms and the frame rate is 10.8FPS when processing with both hands, which still meets the real-time interaction requirements (no stuttering felt by the human eye); and there are no additional hardware requirements: only the MediaPipe Hands parameters and the added differentiation logic need to be adjusted, without the need to add cameras or computing power, and the cost remains unchanged.

[0229] Based on the above embodiments, the present invention also proposes a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the real-time gesture recognition method based on multi-feature fusion as described above.

[0230] It should be understood that the above embodiments are merely illustrative examples of the present invention. Those skilled in the art can make equivalent substitutions or reasonable modifications to some technical features without departing from the core concept of the present invention.

[0231] (a) Equivalent replacement at the module level

[0232] The core technical concept of this invention lies in achieving robust real-time gesture recognition in complex scenarios through the temporal progressive extraction and closed-loop feedback calibration of three types of features: "skeleton features, contour features, and skin color features." Any technical means that can provide similar features and integrate with this closed-loop calibration architecture can be used as an alternative.

[0233] Specifically:

[0234] The skeleton detection module extracts key hand points to provide structural features and initial spatial extent. Besides MediaPipe Hands, any lightweight model capable of outputting hand key point coordinates in real time (such as regression models based on backbone networks like MobileNet and EfficientNet) can be used.

[0235] The superpixel segmentation module generates a boundary mask within the hand ROI to provide contour features and region confidence. Besides the SLICO algorithm, any algorithm capable of dividing an image region into compact sub-regions with color or texture similarity and extracting region boundaries (such as graph-based segmentation algorithms or clustering-based segmentation algorithms) is applicable.

[0236] The skin color tracking module is designed to continuously track the hand region based on a skin color template to provide temporal continuity. Besides CamShift, any tracking algorithm that uses density estimation or filtering prediction based on a target appearance model (such as mean-shift algorithms or correlation-filtering algorithms) can be used.

[0237] The color space conversion (LAB, HSV) can be replaced with other color spaces that are insensitive to light or have perceived uniformity (such as YCrCb, HSL, etc.) depending on the application scenario.

[0238] (ii) Adaptive adjustments at the parameter level

[0239] The first threshold (0.015-0.025) and the second threshold (0.025-0.035) for finger extension judgment are preferred ranges determined based on normalized coordinate statistics. For input images of different resolutions or hands of different sizes, adaptive adjustments can be made within this range, or new thresholds can be determined through calibration with a small number of samples based on actual application scenarios. These all fall under the technical concept of this invention for finger extension judgment based on the relative difference of normalized coordinates.

[0240] (III) Reasonable expansion at the functional level

[0241] The closed-loop calibration architecture of this invention has inherent scalability. For example, expanding the gesture categories from numerical gestures "1-5" to more static gestures such as clenched fists and OK only requires adding corresponding conditional rules to the finger extension judgment. The core skeleton-contour-skin color three-feature coordination and calibration mechanism remain unchanged, thus constituting a direct application extension of this invention.

[0242] Similarly, when extending single-hand recognition to simultaneous two-hand recognition, it is only necessary to adjust the maximum number of hands detected by the skeleton detection module to two, and to independently instantiate a set of ROI extraction, superpixel segmentation, skin color tracking, and calibration processes for each hand, without interference between the two hands. When the ROIs of the two hands overlap, the region confidence detection in the calibration mechanism is naturally triggered for readjustment, demonstrating the versatility of the closed-loop calibration architecture of this invention.

[0243] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time gesture recognition method based on multi-feature fusion, characterized in that, include: Obtain an image sequence containing hands; Perform hand skeleton detection on the current frame image, extract hand key points, and determine the initial region of interest (ROI) of the hand based on the hand key points; Superpixel segmentation is performed within the initial ROI to generate a boundary mask for the hand region. Based on the deviation between the superpixel segmentation result and the initial ROI, the region confidence is calculated to help verify the validity of the hand region. A skin color template is constructed based on the skin color features within the initial ROI, and the skin color template is used to perform target tracking on subsequent frame images to dynamically update the hand ROI; By utilizing the relative positional relationships between key points on the hand, the finger extension state is determined, and the gesture recognition result is output. 2.The real-time gesture recognition method based on multi-feature fusion according to claim 1, characterized in that, The method further includes at least one of the following calibration steps: When the confidence level of a region falls below a preset confidence threshold, the initial ROI is readjusted. When the area difference between the region covered by the boundary mask and the initial ROI exceeds a preset ratio threshold, the initial ROI is readjusted. When the deviation between the center position of the dynamically updated hand ROI and the center position of the hand skeleton in the current frame exceeds a preset distance threshold, the target tracking area of ​​the skin color template is reinitialized using the hand key points in the current frame. 3.The real-time gesture recognition method based on multi-feature fusion according to claim 1, characterized in that, The hand skeleton detection uses the MediaPipe Hands framework to extract the two-dimensional or three-dimensional coordinates of 21 key hand points. 4.The real-time gesture recognition method based on multi-feature fusion according to claim 3, characterized in that, The determination of the initial region of interest (ROI) for the hand based on hand key points includes: Calculate the minimum bounding rectangle of all hand keypoints in the image coordinate system; The initial ROI is generated by expanding the boundary of the minimum enclosing rectangle outward by a preset pixel value.

5. The real-time gesture recognition method based on multi-feature fusion according to claim 1, characterized in that, The process of performing superpixel segmentation within the initial ROI includes: Convert the initial ROI from the RGB color space to the LAB color space; In the LAB color space, the SLICO algorithm is used for superpixel segmentation. Superpixel blocks are generated through iterative clustering, and the boundaries between superpixel blocks are extracted to form the boundary mask. 6.The real-time gesture recognition method based on multi-feature fusion according to claim 1, characterized in that, The deviation between the superpixel segmentation result and the initial ROI is used to calculate the region confidence score, including: Calculate the expected number of superpixels based on the area of ​​the initial ROI; Count the actual number of superpixels generated; The deviation between the actual number of superpixels and the expected number of superpixels is calculated; the smaller the deviation, the higher the region confidence. 7.The real-time gesture recognition method based on multi-feature fusion according to claim 1, characterized in that, The step of constructing a skin color template based on skin color features within the initial ROI, and using the skin color template to perform target tracking in subsequent frame images, includes: Generate a rectangular mask within the initial ROI that retains only the hand area, and calculate the skin color histogram of the pixels within the rectangular mask in the HSV color space as the skin color template; For subsequent frames, the entire frame image is converted to the HSV color space, and a back projection map is calculated based on the skin color histogram. The back-projected image is input into the CamShift tracking algorithm to iteratively calculate the probability density peak, thereby dynamically updating the position and size of the hand ROI. 8.The real-time gesture recognition method based on multi-feature fusion according to claim 3, characterized in that, The method of determining finger extension state by utilizing the relative positional relationship between key points on the hand includes: The 21 key points are divided into three categories: fingertip, knuckle, and palm base. For the thumb, whether it is extended is determined by whether the difference between the normalized x-coordinate of the key point of its fingertip and the key point of the corresponding palm base is greater than the first threshold. For any finger from the index finger to the little finger, determine whether it is extended based on whether the difference between the normalized y-coordinate of its fingertip key point and the corresponding phalanx key point is greater than the second threshold.

9. A real-time gesture recognition system based on multi-feature fusion, characterized in that, include: The image acquisition module is used to acquire image sequences containing the hand; The skeleton detection module is used to detect the hand skeleton in the current frame image, extract key points of the hand, and determine the initial region of interest (ROI) of the hand based on the key points of the hand. The superpixel segmentation module is used to perform superpixel segmentation within the initial ROI, generate a boundary mask for the hand region, and calculate the region confidence based on the deviation between the superpixel segmentation result and the initial ROI to help verify the validity of the hand region. The skin color tracking module is used to construct a skin color template based on the skin color features within the initial ROI, and to use the skin color template to perform target tracking on subsequent frame images in order to dynamically update the hand ROI; The gesture classification module is used to determine the finger extension state by utilizing the relative positional relationship between key points on the hand and output the gesture recognition result.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the real-time gesture recognition method based on multi-feature fusion as described in any one of claims 1 to 8.

Images