A real-time fingertip detection method and device based on a lightweight model

Abstract

The application relates to the technical field of fingertip detection, and discloses a real-time fingertip detection method and equipment based on a lightweight model, which comprises the following steps: collecting hand image data and constructing a hand skeleton constraint relationship database; performing hierarchical feature extraction and feature hierarchical adaptive fusion through a mixed density encoder network to generate a comprehensive feature map; performing feature mapping fusion to output initial fingertip position coordinate data; adopting a model lightweight technology to optimize the network structure of the mixed density encoder network to generate a lightweight fingertip detection model; performing interframe difference detection and adaptive confidence threshold adjustment to output a real-time updated fingertip accurate position coordinate set, thereby effectively suppressing the jitter phenomenon of single-frame detection results, providing a smooth and continuous fingertip trajectory, and improving the naturalness and comfort of human-computer interaction.

Description

Technical Field

[0001] This application relates to the field of fingertip detection technology, and in particular to a real-time fingertip detection method and device based on a lightweight model. Background Technology

[0002] Traditional fingertip detection methods are mostly based on deep learning technology, relying on complex network structures and large amounts of computing resources, making it difficult to achieve real-time performance in scenarios with limited computing resources, such as mobile devices and embedded systems. Furthermore, these methods often simplify the fingertip detection problem to a simple keypoint detection task, ignoring the unique biomechanical constraints of the hand and the special texture features of the fingertip region, resulting in insufficient detection accuracy in complex environments.

[0003] Existing lightweight fingertip detection methods primarily reduce computational complexity by simplifying network structures or decreasing model parameters. However, this often oversimplifies the model's expressive power, leading to a significant drop in detection accuracy, especially in complex scenarios such as multi-finger overlap and partial occlusion. Furthermore, these methods generally lack the ability to effectively extract fingertip texture features, making it difficult to distinguish similar fingers and accurately locate fingertip positions. Additionally, there is an inherent contradiction between lightweight models and real-time performance; maintaining high detection accuracy while ensuring real-time inference capabilities remains a pressing technical challenge. Summary of the Invention

[0004] This application provides a real-time fingertip detection method and device based on a lightweight model, which effectively suppresses the jitter phenomenon of single-frame detection results, provides smooth and continuous fingertip trajectories, and improves the naturalness and comfort of human-computer interaction.

[0005] The first aspect of this application provides a real-time fingertip detection method based on a lightweight model, the real-time fingertip detection method based on a lightweight model includes: Collect hand image data and construct a database of hand skeletal constraint relationships; The hand image data is input into a preset hybrid density encoder network for hierarchical feature extraction and adaptive feature fusion to generate a comprehensive feature map; The integrated feature map is fused with the hand skeleton constraint relationship database through feature mapping to output the initial fingertip position coordinate data; The network structure of the hybrid density encoder network is optimized using model lightweighting techniques to generate a lightweight fingertip detection model. Based on the lightweight fingertip detection model and the initial fingertip position coordinate data, inter-frame differential detection and adaptive confidence threshold adjustment are performed to output a set of real-time updated precise fingertip position coordinates.

[0006] A second aspect of this application provides a real-time fingertip detection device based on a lightweight model, the real-time fingertip detection device based on the lightweight model comprising: The acquisition module is used to acquire hand image data and build a database of hand skeletal constraint relationships; The hierarchical feature extraction module is used to input the hand image data into a preset hybrid density encoder network for hierarchical feature extraction and feature hierarchical adaptive fusion to generate a comprehensive feature map; The feature mapping fusion module is used to perform feature mapping fusion between the comprehensive feature map and the hand skeleton constraint relationship database, and output the initial fingertip position coordinate data; The network structure optimization module is used to optimize the network structure of the hybrid density encoder network using model lightweighting technology to generate a lightweight fingertip detection model. The output module is used to perform inter-frame difference detection and adaptive confidence threshold adjustment based on the lightweight fingertip detection model and the initial fingertip position coordinate data, and output a set of real-time updated precise fingertip position coordinates.

[0007] Compared with existing technologies, this application has the following advantages: By introducing a hand skeleton constraint relation database and a biomechanical constraint model, combined with a "fingertip-constraint" dual loss function, the accuracy of fingertip detection is significantly improved, especially in complex scenarios such as multi-finger overlap and partial occlusion, effectively solving the problem of insufficient accuracy of traditional methods. Based on the improved MobileNetV3 backbone network, depthwise separable convolution and structural pruning, and weight quantization, the model lightweighting techniques significantly reduce the network computational complexity and parameter count, enabling the system to run smoothly on mobile devices and embedded platforms. Through the design of a hybrid density encoder network and asymmetric convolution kernels (1×3 and 3×1), the directional features of fingertip texture are extracted efficiently, improving the feature representation capability of the fingertip region while avoiding redundant feature calculations. By adopting a feature-level adaptive fusion mechanism and channel attention gateway technology, feature-level weights are dynamically allocated for scenarios of different complexities, allowing low-level features to focus on fingertip texture details and high-level features to focus on the overall gesture, balancing the expression of local details and global structure. By employing inter-frame differential detection technology and an adaptive confidence threshold mechanism, an intelligent inference strategy based on motion intensity is implemented. A lightweight tracking algorithm is used for stationary states, significantly reducing unnecessary computational overhead and ensuring the stability of detection results. Combining a multi-frame temporal integration strategy and exponentially weighted averaging, jitter in single-frame detection results is effectively suppressed, providing smooth and continuous fingertip trajectories and enhancing the naturalness and comfort of human-computer interaction. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0010] Figure 1 This is a flowchart illustrating the real-time fingertip detection method based on a lightweight model provided in an embodiment of the present invention. Figure 2 This is a schematic block diagram of the structure of a real-time fingertip detection device based on a lightweight model provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should also be further understood that the term "and / or" as used in this application specification and the appended claims refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations. See also Figure 1One embodiment of the real-time fingertip detection method based on a lightweight model in this application includes: Step 100: Collect hand image data and construct a database of hand skeletal constraint relationships; It is understood that the executing entity of this application can be a real-time fingertip detection device based on a lightweight model, or it can be a terminal or a server; the specific implementation is not limited here. This application's embodiments use a server as an example for illustration.

[0015] Specifically, large-scale image data of the hand under different postures, angles, and lighting conditions is acquired using image acquisition equipment. Key point annotation is performed on these hand images, identifying the positions of all key joints of the palm and five fingers in each image, including the metacarpophalangeal joints, proximal interphalangeal joints, distal interphalangeal joints of each finger, and the wrist center point, generating a set of high-quality hand skeletal structure data. Motion attributes are defined for each joint in the above structural data, that is, rotational degrees of freedom parameters conforming to human physiological structure are assigned. The thumb, due to its unique structure and high flexibility, is assigned three rotational degrees of freedom, while the joints of the other fingers are assigned two degrees of freedom. This step generates a dataset of hand joint rotational degrees of freedom, providing a foundation for subsequent constraint modeling. Based on human anatomical reference data, the angular range of motion of each joint is limited. For example, the thumb and other fingers have specific flexion and extension angle ranges at the interphalangeal or metacarpophalangeal joints. These constraint definitions constitute joint activity boundary data to prevent the network from outputting abnormal postures beyond the physiologically permissible range. Meanwhile, to improve the rationality and coherence of posture prediction, cooperative motion rules between adjacent joints are defined. For example, when the distal joint of a finger bends, the bending state of its proximal joint should respond in a certain regular manner. Such angular relationships caused by tendon linkage are summarized as joint angle-dependent data. By analyzing natural motion sequences in a large number of labeled samples, joint linkage patterns are extracted and mathematical expression templates are established. The above-mentioned multi-source data, including hand skeletal structure, joint degrees of freedom, angle boundaries, and joint linkage relationships, are integrated to construct a hand skeletal constraint relationship database for the neural network learning process. This database is expressed in the form of graph structure, vector set, or tensor, and directly participates in feature fusion or posture correction as a structural prior in the subsequent model training and inference stages, thereby effectively reducing the model's dependence on redundant data.

[0016] Step 200: Input the hand image data into the preset hybrid density encoder network for hierarchical feature extraction and feature hierarchical adaptive fusion to generate a comprehensive feature map; Specifically, hand image data is input into a pre-defined hybrid density encoder network, which uses MobileNetV3 as its backbone extraction architecture, significantly reducing the number of model parameters and computational complexity while maintaining high expressive power. After entering the MobileNetV3 backbone network, the image data undergoes initial depthwise separable convolution and channel fusion operations to extract initial feature maps containing global spatial structure and basic texture information. The initial feature maps are then input into a fingertip texture specialization module designed specifically for fingertip detection tasks. In this module, three parallel branches perform in-depth mining of directional details in the image. The first branch uses horizontally elongated convolutional kernels to extract texture changes in the horizontal direction, the second branch uses vertical convolutional kernels to extract fingertip details in the vertical direction, and the third branch uses standard point convolution to maintain the original texture integrity of the input features. The outputs of the three are merged into a set of structured fingertip texture feature maps through channel concatenation operations, effectively enhancing the model's ability to perceive fine-grained textures. The fingertip texture feature map is input into the inverted residual structure module of the network for high-dimensional representation enhancement. This module sequentially includes a channel expansion layer, a depthwise convolutional layer, and a channel compression layer. The expansion layer performs a linear mapping on the input channels to increase the representation dimension. The depthwise convolutional layer enhances the spatial sensitivity of the features through multi-scale convolution operations in local regions, while the projection layer compresses the number of channels back to the design specifications to ensure the network's lightweight characteristics. Simultaneously, to accommodate the differences in nonlinear representation of features at different levels, the network uses the ReLU6 activation function in the shallow structure to improve training stability, while employing the more complex Swish activation function in the deep structure to enhance the network's expressive power and nonlinear fitting strength. Through this step, an enhanced feature map is obtained. This enhanced feature map is then input into the channel compression layer of the hybrid density encoder network for dimensionality reduction. By reducing the number of channels, computational resources are optimally allocated, and multi-scale feature outputs are generated while maintaining information integrity, forming a multi-level feature dataset. By constructing a feature-level adaptive fusion mechanism, the above feature data is dynamically weighted and fused. This mechanism introduces a channel attention gateway, which automatically allocates fusion weights according to the contribution of different feature levels to the task. During the fusion process, a texture enhancement submodule and a global context encoder are applied to adapt to the needs of low-level local textures and high-level gesture semantic understanding. Finally, while maintaining the resolution of details, a comprehensive feature map containing directional textures of the fingertip area, full palm structure information and pose priors is constructed.

[0017] The multi-level feature data was divided into four feature level sets based on its semantic depth and spatial resolution: Level 1, Level 2, Level 3, and Level 4 features. Level 1 features have the highest spatial resolution but the lowest semantic abstraction, primarily containing rich texture and edge detail information. Level 4 features have the lowest resolution but the highest semantic expressiveness, reflecting the overall spatial organization and posture of the gesture. After the hierarchical division, two differentiated feature enhancement strategies were adopted for the task division of different levels. Level 1 and Level 2 features were input into the fingertip texture enhancement module for local detail enhancement. This module consists of two parallel branches: one branch introduces the classic Sobel edge operator to extract texture boundaries and directional gradient information, capturing local detail changes in the fingertip region; the other branch maintains the original state of the input features for structural completion and texture restoration. Subsequently, the outputs of the two branches are fused through an attention mechanism to form texture-enhanced feature data that retains the original structure while enhancing edge expressiveness. Simultaneously, the third and fourth level features are input into the gesture understanding module to complete the unified modeling of high-level semantics. This module adopts a global context encoder structure, which can capture the pose, movement trend, and collaborative relationships between cross-finger regions of the entire hand at lower resolutions. It then forms semantically complete gesture understanding feature data through feature compression and a fully connected attention mechanism. Dynamic weight allocation is applied to the texture enhancement feature data and gesture understanding feature data, introducing a channel attention gateway mechanism. Global average pooling and a nonlinear mapping function are used to weight the response of each level. The weight allocation references the complexity of the image itself and considers the perceptual contribution of each feature level in a specific scene, resulting in a set of task-adaptive weighted multi-scale features. To unify the feature representation at different resolutions and ensure the spatial integrity and accuracy of the final feature map, depthwise separable deconvolution is used to spatially reconstruct the features at each level. Simultaneously, pixel shuffling is used to rearrange the upsampled feature map, saving computational resources while avoiding the blurring problem in texture restoration inherent in traditional interpolation methods, ultimately yielding a comprehensive feature map.

[0018] Step 300: Perform feature mapping and fusion between the comprehensive feature map and the hand skeleton constraint relationship database to output the initial fingertip position coordinate data; It should be noted that the spatial transformation operation on the comprehensive feature map is performed. This process uses convolution and upsampling mechanisms to transform the fused multi-scale features into a candidate fingertip heatmap. Specifically, it generates the response intensity distribution for each fingertip location in the image coordinate space, providing a set of heatmap regions with high-resolution localization capabilities to represent the potential fingertip locations. Simultaneously, to ensure the network prediction results conform to the physiological structural constraints of the human hand, the pre-established biomechanical constraint matrix in the hand skeletal constraint relation database is morphologically adjusted to be compatible with the feature map in terms of data structure and dimension. This matrix is ​​then transformed into a constraint feature map through tensor channel expansion, preserving its original structural dependency information while possessing the expressive power of image tensor format. The candidate fingertip heatmap and this constraint feature map are concatenated along the channel dimension to form a constraint-enhanced fingertip prediction map that contains both semantic texture features and structural constraint information. During the training phase, to guide the network in optimizing the representation quality of the predicted map, a dual loss function is constructed that simultaneously considers localization accuracy and anatomical consistency. The loss function includes fingertip localization loss, which measures the difference between the predicted position and the true coordinates; joint angle constraint loss, which penalizes postures where joint angles exceed physiological limits; and bone length constraint loss, which standardizes the relative distance between adjacent joints to maintain stable human proportions. A smoothness constraint loss is also introduced to prevent drastic jitter or discontinuous jumps in fingertip predictions across consecutive frames. These four losses together constitute a composite evaluation index for structural rationality and accuracy consistency. Guided by the loss function, peak detection is performed on the constrained fingertip prediction map. This involves locating the region with the strongest response in each channel to identify the most probable positions of the five fingertips in the current image, and outputting this set of position data as initial fingertip position coordinates in image coordinate form.

[0019] Step 400: Optimize the network structure of the hybrid density encoder network using model lightweighting technology to generate a lightweight fingertip detection model; Specifically, structural pruning is performed on the hybrid density encoder network to significantly reduce the number of model parameters and computational load. This pruning operation is based on the L1 norm. It assesses the channel-level importance of the weight tensors of all convolutional layers in the MobileNetV3 backbone network, fingertip texture specialization module, and inverted residual structure. The average absolute value of the output channels of each convolutional kernel is calculated and used as the quantification of channel contribution. Channels with small weights and weak information expression capabilities are then filtered out based on a set contribution threshold, resulting in the removal of the corresponding convolutional kernels. This leads to a simplified network architecture with less redundancy, significantly reducing memory and computational overhead while maintaining the integrity of the core perceptual path. To compress model storage size and improve operational efficiency, weight quantization is performed on the model parameters in the simplified network structure, mapping the original 32-bit floating-point representation of convolutional weights to 8-bit integer representations. This process comprises two stages: In the quantization calibration stage, the distribution of activation values ​​from representative input data is statistically analyzed to determine the maximum and minimum ranges of weights and activation values ​​for each layer, and the scaling ratio and offset are calculated. Then, in the parameter transformation stage, all floating-point weights are range-scaled and mapped to integers to obtain an integer-based model for practical deployment. Since direct quantization leads to a decrease in model accuracy, a quantization-aware training mechanism is introduced. During training, the forward inference path of quantization is simulated, using integer approximations for feature transformation, while continuous floating-point gradients are still used for updates during backpropagation. This allows for targeted compensation and correction of errors caused by quantization during training, resulting in a quantized model with restored accuracy. Using a teacher-student structure knowledge distillation technique, a pre-trained high-precision teacher model is used as a performance reference. By minimizing the distribution difference between the output features of the two models, the learning direction of the student model is guided, ensuring that it retains lightweight features while closely approximating the discrimination boundary and response characteristics of the teacher model in the fingertip detection task. This results in a student model that is structurally simple and reliable in accuracy. To reduce runtime computation latency and data access overhead, operator fusion processing is performed on adjacent convolutional layers and normalization layers in the student model. For example, batch normalization and convolutional layers before and after are merged into a unified computing node. While maintaining the consistency of the network structure, the overhead of intermediate storage and synchronization operations is reduced, the overall inference speed and deployment efficiency are optimized, and a lightweight fingertip detection model is finally constructed.

[0020] Step 500: Based on the lightweight fingertip detection model and the initial fingertip position coordinate data, perform inter-frame difference detection and adaptive confidence threshold adjustment, and output a set of real-time updated precise fingertip position coordinates.

[0021] Specifically, the continuously acquired video frames are compared frame by frame, and the change information between the current frame and the previous frame is obtained by using pixel-level difference calculation. By calculating the intensity difference at each pixel position of the two frames and constructing an inter-frame difference map, the global change intensity of the difference map is statistically analyzed to obtain a numerical motion intensity index, which is used to measure the actual degree of hand movement in the current frame. The motion intensity index is compared with a preset threshold. When the index exceeds the threshold, it means that there is significant motion in the current video frame, which may indicate a new gesture change or fingertip occlusion. At this time, the system triggers the complete lightweight fingertip detection model to re-perform feature extraction and pose estimation operations on the current frame, obtaining a set of high-precision fingertip position results based on the current image information. When the motion intensity index is below the threshold, it means that the image difference between the current frame and the previous frame is very small. The system does not need to repeat the calculation and inference process. Instead, it adopts a lightweight tracking strategy based on the Lucas-Kanade optical flow method, using only the initial fingertip position coordinates of the previous frame as the starting point, and using the pixel gradient distribution within the local window to predict the new position of the fingertip in the current frame, obtaining a set of fingertip position update data with lower computational cost but stronger continuity. Meanwhile, to enhance the system's adaptability to scenes with varying visual complexity, image entropy calculation is performed on the current frame to analyze the complexity of the grayscale distribution of the image content. Based on this analysis, the confidence threshold is dynamically adjusted. The more complex the scene, the higher the image entropy value, and the higher the confidence requirement is, thus reducing the false recognition rate. In simpler scenes, the judgment threshold is appropriately relaxed, ultimately resulting in an adaptive confidence threshold for the current image. Based on the complete inference or tracking results of the current frame, combined with the fingertip position data cached from the previous two frames, a short-time series containing three time nodes is constructed. The fingertip coordinates in this series are then integrated using an exponentially weighted average algorithm, assigning higher weights to results from more recent frames and lower weights to results from more distant frames. This achieves temporal smoothing and anomaly suppression of the fingertip position, especially effective in improving detection stability under conditions of drastic motion changes or image quality fluctuations. A set of real-time updated, precise fingertip position coordinates is output.

[0022] In this embodiment, by introducing a hand skeleton constraint relationship database and a biomechanical constraint model, combined with a "fingertip-constraint" dual loss function, the accuracy of fingertip detection is significantly improved, especially in complex scenarios such as multi-finger overlap and partial occlusion, effectively solving the problem of insufficient accuracy in traditional methods. Based on the improved MobileNetV3 backbone network, depthwise separable convolution and structural pruning, and weight quantization, the model lightweighting techniques significantly reduce the network computational complexity and parameter count, enabling the system to run smoothly on mobile devices and embedded platforms. Through a hybrid density encoder network and asymmetric convolution kernel (1×3 and 3×1) design, the directional features of fingertip texture are extracted efficiently, improving the feature representation capability of the fingertip region while avoiding redundant feature calculations. A feature-level adaptive fusion mechanism and channel attention gateway technology are adopted to dynamically allocate feature-level weights for scenarios of different complexities, allowing low-level features to focus on fingertip texture details and high-level features to focus on the overall gesture, balancing the expression of local details and global structure. By employing inter-frame differential detection technology and an adaptive confidence threshold mechanism, an intelligent inference strategy based on motion intensity is implemented. A lightweight tracking algorithm is used for stationary states, significantly reducing unnecessary computational overhead and ensuring the stability of detection results. Combining a multi-frame temporal integration strategy and exponentially weighted averaging, jitter in single-frame detection results is effectively suppressed, providing smooth and continuous fingertip trajectories and enhancing the naturalness and comfort of human-computer interaction.

[0023] In one specific embodiment, the process of performing step 100 may specifically include the following steps: Collect hand image data and annotate the joints in the hand image data to obtain hand skeletal structure data; Define the degrees of freedom for each joint in the hand skeletal structure data to obtain a dataset of joint rotational degrees of freedom. Based on human anatomical data, the angle limit range of the joint rotational degree of freedom dataset is set to obtain joint activity boundary data; Based on the joint movement boundary data, the angular relationship between adjacent joints is calculated to obtain joint angle dependency data; A database of hand skeletal constraint relationships was constructed based on hand skeletal structure data, joint rotational degrees of freedom dataset, and joint angle dependence data.

[0024] Specifically, the process involves acquiring hand images. This requires collecting large-scale, high-resolution sequences of hand images from multiple angles, poses, and environmental conditions to ensure the complete and clear hand structure and coverage of various common movement variations. After acquisition, specialized annotation tools are used to meticulously annotate key skeletal nodes in the image data. Annotations include the metacarpophalangeal joints, proximal interphalangeal joints, distal interphalangeal joints, fingertip nodes, and key structural points of the palm and wrist for each finger. This ensures that each frame contains complete information on the positions of all 22 joint points, forming a hand skeletal structure dataset with spatial coordinates. Based on the clear hand skeletal structure, degrees of freedom are defined for these joints. Each joint is assigned a reasonable rotational dimension according to the actual human skeletal movement mechanism. The thumb, due to its unique anatomical structure, has three rotational degrees of freedom, enabling complex movements such as adduction, abduction, and flexion / extension. Each joint of the other fingers is assigned two degrees of freedom, corresponding to flexion / extension and internal / external rotation, respectively. These joints are then mapped one-to-one with their corresponding degrees of freedom, forming a joint rotational degree of freedom dataset. To make hand modeling biologically meaningful, based on medical literature and clinical measurement data, strict angular ranges of motion were set for each direction of motion involved in the above-mentioned degree of freedom definition. For example, the flexion and extension angle of the interphalangeal joint of the thumb was set to 0 to 90 degrees, and the metacarpophalangeal joint was set to 0 to 60 degrees. The range of motion of the interphalangeal joints of the index finger to the little finger was 0 to 110 degrees, and the range of motion of the metacarpophalangeal joint was 0 to 90 degrees. Corresponding upper and lower limits were established within the system to form a set of angle constraint data covering all joints, constituting a joint activity boundary dataset. To improve the completeness and accuracy of hand skeleton modeling, the natural linkage relationships between adjacent joints are modeled. In actual human hand movement, the distal and proximal joints of the same finger do not move completely independently, but are jointly influenced by tendon and muscle tension. Therefore, by analyzing large-scale real motion data, the cooperative change law between adjacent joints is summarized. That is, when a joint bends, its neighboring joints will bend synchronously according to a fixed proportion or a limited range of deviation. Thus, the correlation constraint function between adjacent angles is derived, and these laws are summarized into a highly structured and constrained joint angle dependency dataset. This allows the model to not only focus on the position matching of individual nodes when performing position prediction tasks, but also maintain the logical consistency of the overall posture structure in accordance with human mechanisms. By integrating hand skeleton structure data, joint rotational degrees of freedom data, and joint angle dependency data, a hand skeleton constraint relationship database is constructed in terms of structural organization using graph neural network modeling. Each joint point is regarded as a node in the graph, and each pair of constrained connected skeletal units constitutes an edge in the graph. The node attributes contain its degree of freedom definition, and the edge attributes record the angle dependency relationship and length constraint conditions.In addition, the database includes a standard length definition for each set of skeletal vectors to ensure that subsequent neural networks can optimize errors in a way that minimizes structural violations when handling hand pose recovery and fingertip prediction tasks.

[0025] In one specific embodiment, the process of performing step 200 may specifically include the following steps: Hand image data is input into the MobileNetV3 backbone network of the hybrid density encoder network for feature extraction to obtain an initial feature map; The initial feature map is input into the fingertip texture specialization module of the hybrid density encoder network for directional feature extraction, resulting in a fingertip texture feature map. The fingertip texture specialization module contains three parallel branches: the first branch uses a 1×3 convolution kernel to extract the horizontal texture, the second branch uses a 3×1 convolution kernel to extract the vertical texture, and the third branch uses a 1×1 convolution to preserve the original features. The fingertip texture feature map is input into the inverted residual structure of the hybrid density encoder network for feature enhancement, resulting in an enhanced feature map. The inverted residual structure includes an extension layer, a deep convolutional layer, and a projection layer. The shallow part of the network uses the ReLU6 activation function, and the deep part uses the Swish activation function. The enhanced feature map is input into the channel compression layer of the hybrid density encoder network to reduce its dimensionality, resulting in multi-level feature data; Adaptive fusion of feature levels is performed on multi-level feature data to generate a comprehensive feature map.

[0026] Specifically, hand image data is input into the MobileNetV3 backbone network of the hybrid density encoder network. This backbone network is designed for low-power, high-efficiency computing scenarios. While maintaining structural depth, it significantly reduces redundant parameters through depthwise separable convolutions and inverted residual connections, enabling stable inference speed and modeling capabilities on mobile devices or edge processing platforms. In this structure, the input image undergoes initial dimensionality reduction and channel expansion through standard convolutional layers. Subsequently, local texture features, spatial boundary information, and basic structural patterns are extracted in multiple depthwise separable convolutional modules to form a set of initial feature maps with moderate semantic depth and high resolution. The initial feature maps are then input into a specially designed fingertip texture specialization module in the hybrid density encoder network to enhance the network's sensitivity to fingertip edges and local texture changes. This module consists of three parallel convolutional branches, each capturing the most discriminative texture features from different directions and scales. The first branch uses a horizontally extended 1×3 convolutional kernel to perceive boundary abrupt changes and morphological extensions in the horizontal direction of the fingertip, suitable for capturing elongated structural information such as fingertip texture and fingertip contour. The second branch uses vertically arranged 3×1 convolutional kernels to enhance the network's ability to perceive edge changes in the vertical direction, suitable for extracting response patterns in the knuckle compression area or the direction of illumination gradient. The third branch uses a 1×1 convolutional kernel as a preservation path to retain the original texture and spatial information in the input feature map, providing a comparative benchmark and structural completion capability for subsequent multi-scale information fusion. After each of these three directional branches completes feature encoding, their outputs are concatenated along the feature dimension through channel concatenation operations to form a set of fingertip texture feature maps with directional sensitivity, boundary prominence, and original fidelity, improving the model's discriminative ability and localization accuracy in local region recognition tasks. The aforementioned fingertip texture feature map is input into the inverted residual structure module inside the hybrid density encoder network for deep feature enhancement. This structure integrates three major processing stages: convolutional channel expansion, deep convolution extraction, and projection compression. The expansion layer expands the number of feature channels in high-dimensional space to many times that of the original input, thereby improving the network's ability to encode multi-channel redundant representations. Next, the deep convolutional layer performs local spatial convolution operations on each channel to fully extract cross-pixel patterns and micro-directional gradients in the feature map. Finally, the projection layer compresses the number of channels back to the set dimension, ensuring that the representation after feature enhancement maintains expressive power while controlling computational load and memory consumption.To enhance the model's ability to express nonlinearity at different levels, the inverted residual structure introduces the ReLU6 activation function in the shallow layers of the network. This function, through truncation of negative values ​​and nonlinear transformation, exhibits good feature preservation capabilities in edge detection and low-order texture processing. In deeper modules, the Swish activation function is used. This activation function is continuous, differentiable, and flexible, making it more suitable for high-order abstraction of complex semantic patterns, thus enabling the network to progressively enhance its expressive power throughout the entire depth space. After multi-channel, multi-directional, and multi-scale enhancement processing, the enhanced feature map is input to the channel compression layer of the hybrid density encoder network. This layer performs rapid channel dimension compression through 1×1 convolution operations, reducing the model's memory requirements and computational load, while avoiding gradient propagation barriers and overfitting risks caused by high channel numbers. The compressed feature results are divided into several output levels, corresponding to intermediate feature representations at different stages in the network. These features differ in spatial resolution, semantic depth, and structural abstraction capabilities; therefore, they are unified and integrated through a feature level adaptive fusion mechanism. To complete the final feature fusion process, a hierarchical fusion module incorporating an attention mechanism is employed. Each resolution branch of the multi-level feature data is input into a specific channel weight evaluation path. Channel-level statistical information is extracted through global average pooling, and then dynamic weight allocation coefficients are generated via two fully connected layers and activation functions. These weights are applied to the feature maps of each layer, ensuring that the fusion process prioritizes feature channels with complete structure, clear texture, or rich semantics. The fusion process employs three steps—feature upsampling, channel concatenation, and convolutional reshaping—to achieve spatial unification, dimensional alignment, and information fusion. Simultaneously, a texture enhancement module is added for low-level features, and a gesture understanding module is introduced for high-level features, ensuring that the entire feature set maintains detailed information while possessing global structural consistency. The final comprehensive feature map shows improvements in spatial localization capability, semantic coverage, and structural integrity.

[0027] In one specific embodiment, the process of performing feature-level adaptive fusion on multi-level feature data to generate a comprehensive feature map may specifically include the following steps: The multi-level feature data is divided into four feature level sets with different resolutions. The feature level sets include first-level features, second-level features, third-level features, and fourth-level features. The fingertip texture enhancement module is used to enhance the details of the first-level and second-level features in the feature hierarchy set to obtain texture enhancement feature data. The fingertip texture enhancement module contains two branches: one branch applies the Sobel operator to extract gradient information, and the other branch retains the original features. The gesture understanding module performs global feature extraction on the third-level and fourth-level features in the feature hierarchy set to obtain gesture understanding feature data. The gesture understanding module uses a global context encoder to capture the overall hand posture. Dynamic weight allocation is performed on texture enhancement feature data and gesture understanding feature data to obtain weighted multi-scale features. By using depthwise separable deconvolution and pixel shuffling techniques, weighted multi-scale features are processed in a unified manner to obtain a comprehensive feature map.

[0028] Specifically, the multi-level feature data is divided into four independent feature level sets according to their resolution from high to low using a structured approach: first-level features, second-level features, third-level features, and fourth-level features. The first-level features come from the network's Stage 2 stage, have the highest spatial resolution but weaker semantic abstraction ability, and are used to preserve texture edges and local image details. The second-level features are located below it, still have a certain resolution, and have slightly enhanced semantic depth. The third and fourth-level features come from Stage 4 and Stage 5, respectively, with gradually decreasing resolution but richer semantic expressiveness and global structural information. Therefore, the four levels form a multi-scale feature structure with increasing spatial resolution and semantic depth. To enhance the model's ability to understand detailed textures in low-level features, a fingertip texture enhancement module is introduced on the first and second level features. This module is designed specifically for fine-grained texture analysis and directional edge extraction. It contains two complementary branch paths. One path uses the classic Sobel operator to calculate the gradient of the input features and extracts the edge responses in the horizontal and vertical directions, thereby enhancing the directional contours and boundary abrupt changes of the fingertip region. The other path directly retains the original input features, ensuring that existing spatial structure information is not lost or the original texture hierarchy is not destroyed during the enhancement process. After the two paths are processed in parallel, the outputs are concatenated and information is fused through 1×1 convolution to generate a set of texture enhancement feature data that combines edge sensitivity and structural integrity. Meanwhile, for the third and fourth level features with higher semantic depth, a gesture understanding module is designed to extract the global semantic information they contain. This module embeds a global context encoder mechanism and uses methods such as global average pooling, nonlinear mapping and broadcast weighting to capture the long-distance dependencies and spatial cooperation patterns between regions in the feature map. This allows macroscopic information such as gesture structure, finger distance and palm posture to be abstracted and extracted from low-resolution features and encoded in a compact way to form a set of gesture understanding feature data that includes the overall posture distribution, relative positional relationship and hand shape semantics. Dynamic weight allocation is performed on texture enhancement feature data and gesture understanding feature data. A channel attention gateway is constructed to assign adaptive fusion weights to each set of feature branches. This attention mechanism obtains the channel-level response by performing global average pooling on the input features, and then generates a set of normalized weight vectors through a multilayer perceptron and activation function, and applies them to the feature channels of each layer to realize the reweighting of information channels. This enables the model to automatically adjust the fusion strategy according to image content, gesture complexity and feature contribution during the fusion process, enhance the response of key features, suppress redundant channel interference, and finally obtain a set of weighted multi-scale features with semantic bias, texture integrity and structural stability.To ensure spatial consistency and a unified output format for the weighted multi-scale features, a depthwise separable deconvolution operation is employed to upsample all low-resolution features. Compared to traditional transposed convolution or bilinear interpolation, depthwise separable deconvolution offers better feature preservation and spatial restoration accuracy while maintaining computational efficiency. Furthermore, a pixel shuffling technique is introduced to rearrange the channel information in the convolution output according to a fixed mapping method, thus avoiding the blurring or boundary distortion caused by traditional interpolation and ensuring that each feature level is restored to a uniform spatial resolution. All upsampled feature maps are then concatenated and fused along the channel dimension, and channel compression and semantic integration are achieved through 1×1 convolution to form a comprehensive feature map.

[0029] In one specific embodiment, the process of performing step 300 may specifically include the following steps: A feature space transformation is performed on the comprehensive feature map to obtain the candidate fingertip heatmap; The biomechanical constraint matrix in the hand bone constraint relation database is converted into a feature map compatible format to obtain the constraint feature map; Perform channel connection operations on the candidate fingertip heatmap and constraint feature map to generate a constraint-enhanced fingertip prediction map; The constraint-enhanced fingertip point prediction map calculates the dual loss values ​​of fingertip and constraints, which include fingertip localization loss, joint angle constraint loss, bone length constraint loss and smoothness constraint loss. Peak detection is performed on the constraint-enhanced fingertip point prediction map based on the dual loss values ​​of fingertip and constraint to obtain the initial fingertip position coordinate data.

[0030] Specifically, a feature space transformation is performed on the comprehensive feature map. This involves a series of spatial transformation processes consisting of convolution, upsampling, and normalization operations, mapping the multi-scale fused comprehensive feature map onto the fingertip response space. This outputs a candidate fingertip heatmap, which contains five channels, each corresponding to a response region of one of the five fingertips. High-response regions in each channel represent the most likely positional distribution of that fingertip. This process decodes visual features into probabilistic heatmaps using the general spatial perception capability of convolutional channels, thus completing the mapping from high-dimensional encoding to explicit positional distribution. Simultaneously, to improve the structural biorhythm of the prediction results, a biomechanical constraint matrix from the hand skeleton constraint relation database is invoked. This matrix, in its original form, is an abstract constraint graph structure and parameter mapping relationship, representing the angular constraints, degrees of freedom range, and dynamic dependencies between adjacent joints in the hand. To enable these structural priors to participate in the prediction process in the neural network, they are tensorized. This involves encoding each joint and its connections into a tensor form consistent with the image space dimension, and mapping angle and length constraints to channel features in discrete space, generating a constraint feature map whose format matches the candidate fingertip heatmap. This feature map is spatially consistent with the image coordinate system and expresses different structural guidance information in the channel dimension, ensuring that the subsequent neural network can perceive biological constraints and automatically avoid unreasonable postures during prediction inference. Channel concatenation operations are performed between the candidate fingertip heatmap and the constraint feature map to construct a set of constraint-enhanced fingertip prediction maps. These prediction maps contain both visual response information from image semantic analysis and kinematic constraint information from the skeletal structure level, giving the model dual expressive capabilities. On the one hand, it can extract potential fingertip regions from image texture; on the other hand, it is guided by the physical structure of the hand during spatial judgment, thereby improving prediction stability and structural rationality. To train the structure-enhanced prediction map and ensure its robustness and high consistency across different image and gesture scenarios, a dual loss function system based on fingertip and constraint is defined to constrain the target optimization direction during the prediction process.The loss system includes: a fingertip localization loss, which measures the distance error between the predicted fingertip position and the actual labeled position. In its implementation, a smooth L1 loss function is used to provide smooth gradient optimization under small errors and stronger robustness against abnormal predictions under large errors. Secondly, it includes a joint angle constraint loss, which mainly limits whether the predicted joint angles fall within the biomechanically defined range of motion. If a joint's predicted angle exceeds the maximum or falls below the minimum, this loss will generate a penalty feedback, guiding the network to correct the posture. A bone length constraint loss further standardizes the distance ratio between adjacent joints during prediction, ensuring that the length of each bone segment remains within the statistically significant physiological range, thus avoiding nonlinear deformation caused by image occlusion or blurring. A smoothness constraint loss is introduced to ensure that the changes in the finger joint chain in the temporal or spatial continuous frames present a natural transition. This loss minimizes the second-order difference between adjacent nodes, suppressing abrupt changes or oscillations, especially showing a stabilizing effect in video sequence inference. These four types of losses work together to form a multi-dimensional control mechanism for the fingertip's spatial position, structural rationality, and dynamic continuity. After training and constructing the prediction map, the final peak detection operation is performed on the constrained enhanced fingertip prediction map. This involves identifying the pixel with the highest thermal value in each fingertip channel, which is the most likely location of the fingertip in the image. During peak detection, a confidence threshold filtering mechanism is used to eliminate background noise and false response regions, ensuring that only biomechanically feasible coordinates with strong thermal responses are retained as initial prediction results. Non-maximum suppression and other methods are used to further enhance the spatial exclusivity of the fingertip response region. The final output initial fingertip position coordinate data has accuracy in coordinate space, conforms to anatomical rules in structural logic, and can directly enter the subsequent temporal tracking, adaptive fusion, and coordinate calibration stages, forming a complete structural modeling and prediction process from feature map transformation to fingertip localization.

[0031] In one specific embodiment, the process of performing step 400 may specifically include the following steps: Based on the L1 norm, structural pruning is performed on the MobileNetV3 backbone network, fingertip texture specialization module, and convolutional layers in the inverted residual structure of the hybrid density encoder network to obtain a simplified network structure with reduced parameters. The structural pruning process removes filter channels whose contribution is lower than a preset threshold. The model parameters in the simplified network structure are weighted and quantized to obtain an integer model. The quantization operation is simulated in the forward propagation, and the continuous gradient is used in the back propagation to perform precision compensation on the integer model, so as to obtain the precision-restored quantized model. Knowledge distillation is performed on the quantized model for accuracy recovery based on the teacher model to obtain the student model. Then, operator fusion is performed on adjacent convolutional layers in the student model to obtain a lightweight fingertip detection model.

[0032] Specifically, targeting key modules in the hybrid density encoder network, including the MobileNetV3 backbone network, the fingertip texture specialization module, and all convolutional layers contained in the inverted residual structure, a detailed analysis of each channel in these convolutional layers is performed using an L1 norm-based parameter importance evaluation method. During actual execution, the average absolute value of the weights corresponding to each convolutional kernel's output channel is used as its contribution evaluation index. This value is then compared with a preset global pruning threshold. Channels below the threshold are considered redundant channels with low contribution to the overall feature representation, and the system performs a removal operation, structurally removing these channels and their corresponding convolutional kernels to avoid ineffective computational paths continuously consuming computational resources. This pruning process not only significantly reduces the number of model parameters but also further reduces feature map size and gradient backpropagation overhead by simplifying the number of channels, resulting in a simplified network structure that maintains structural coherence but optimizes computational paths. The model parameters in the simplified network structure are weighted, converting floating-point weights into integer weights with a smaller storage format to adapt to devices with limited computing resources, such as low-power chips and embedded platforms. A fixed-point quantization strategy is adopted to compress all convolutional weights and bias parameters in the simplified network structure from 32-bit floating-point numbers to 8-bit integer representations. This process is divided into two stages: the first stage is the quantization calibration stage, which uses a set of representative hand image input data to infer each layer of the model, records the maximum and minimum intervals of activation values ​​and weight distributions of each layer, and calculates the quantization scaling factor and zero-point offset accordingly; the second stage is the quantization conversion stage, which multiplies each floating-point weight by the scaling factor, subtracts the zero point, and then performs rounding to generate the final integer weight matrix. This integerized model can be efficiently executed using fixed-point arithmetic units during deployment, improving inference speed and reducing storage bandwidth requirements. A quantization-aware training mechanism is introduced. During the training phase, the network is precision compensated by simulating quantization operations. During forward propagation, all calculations are simulated according to the quantized integer model, that is, integer multiplication and addition operations are used to simulate the low-precision inference process in the actual deployment environment; while during backpropagation, floating-point continuous gradients are used for error correction to maintain the stability of gradient flow and avoid training stagnation caused by discrete jumps, resulting in a set of quantized models with restored precision. The model maintains an integer representation in its structure, gradually adapts to the computational perturbations caused by low-order weights during training, and minimizes the prediction error through loss function optimization, thereby achieving a balance between inference efficiency and detection accuracy.To enhance the discrimination and generalization capabilities of the quantization model in practical detection tasks, a knowledge distillation mechanism is introduced. This involves constructing a teacher-student network structure. A higher-performance, larger-parameter standard model is pre-loaded into the teacher model as a guide, while the quantization model, achieving accuracy recovery, serves as the student model. During training, the system does not directly use labels as the sole optimization objective for the student model. Instead, it minimizes the distribution difference between the student and teacher model outputs. This allows the student network to not only mimic label prediction but also learn the internal representation structure formed by the teacher network between the feature extraction and output layers, thus achieving an expressive capability closer to that of a high-precision network. Specifically, the logits vectors generated by the student and teacher networks at the output layers are input into the KL divergence loss function. A temperature coefficient is introduced to soften the distribution, enhancing sensitivity to detailed features and reducing overfitting risk, ultimately leading to parameter convergence under this guidance. After knowledge distillation optimization, operator fusion is performed on the student model's structure. For computationally intensive paths in convolutional neural networks, especially the residual structure, the fingertip texture module, and the combined structures between convolutional layers and batch normalization layers in deep convolutional layers, the system merges these continuous linear transformations. By pre-integrating normalization parameters into the convolutional kernel weights and biases, seamless operator connections are achieved during the inference phase, eliminating intermediate interruptions and storage processes. This reduces the number of intermediate tensor writes, accesses, and synchronization waits during inference, improving overall throughput and latency control. A highly optimized, lightweight fingertip detection model is thus constructed.

[0033] In one specific embodiment, the process of performing step 500 may specifically include the following steps: Pixel-level difference calculations are performed on continuous video frames acquired in real time to obtain inter-frame difference maps and motion intensity indices; Motion judgment processing is performed on video frames based on motion intensity index. When the motion intensity index exceeds a preset threshold, a lightweight fingertip detection model is triggered to perform complete inference to obtain the fingertip position of the current frame. When the motion intensity index is lower than the preset threshold, lightweight tracking processing is performed on the initial fingertip position coordinate data based on Lucas-Kanade optical flow method to obtain updated fingertip position data. The scene complexity is calculated based on the image entropy value of the current video frame, and the confidence threshold is dynamically adjusted based on the scene complexity to obtain an adaptive confidence threshold. Based on the fingertip position in the current frame and the updated fingertip position data, the fingertip position data for three consecutive frames is determined. Then, based on an adaptive confidence threshold, the fingertip position data of the three consecutive frames is processed by exponential weighted averaging, and the real-time updated precise fingertip position coordinate set is output.

[0034] Specifically, a frame-by-frame pixel-level difference operation is performed on the real-time acquired video frame sequence. The gray-level difference between each current frame and the previous frame is calculated point by point to construct an inter-frame difference map. This difference map reflects the local pixel intensity changes of the image in the temporal dimension. By normalizing the sum of the absolute values ​​of all pixels in the entire difference map, a motion intensity index is calculated. This index can quantify the overall change of the current frame compared to the previous frame. The larger the value, the greater the change in image content, accompanied by violent hand movements or sudden changes in posture. The motion intensity index is compared with a preset motion judgment threshold. When the index exceeds the threshold, it indicates that there is a significant movement or posture adjustment in the current frame. Traditional tracking algorithms are difficult to adapt to such rapid changes. At this time, the system triggers the complete lightweight fingertip detection model to perform a forward inference process. It uses a hybrid density encoder network to re-extract image features, generate a comprehensive feature map, fuse skeletal constraints, and predict the fingertip coordinates in the current frame, thereby obtaining accurate identification of the fingertip position under the new posture. When the motion intensity index is below the threshold, it indicates that the overall image changes little and the hand position or fingertip state remains basically stable in the time dimension. At this time, the more efficient Lucas-Kanade optical flow method is used for lightweight fingertip position tracking. Specifically, the five fingertip positions detected in the previous frame are used as initial feature points. In the current frame, the optimal matching position is found by calculating the gray-level gradient and brightness conservative constraints within the local window of the image, and the updated fingertip position data of the current frame is obtained. Meanwhile, to improve the system's adaptability and recognition accuracy in complex scenarios, image entropy analysis is performed on the current video frame. Entropy is an important indicator of the complexity of image information distribution. The higher the entropy value, the more drastic the grayscale changes and the richer the content in the image. Factors such as occlusion, reflection, and texture interference can affect the detection accuracy. Based on this, the system dynamically adjusts the confidence threshold strategy. When the image entropy value is significantly higher than the median entropy value in the training data, the system automatically raises the judgment standard, that is, raises the minimum confidence threshold required for the thermal peak in the fingertip response image, in order to avoid misidentification of non-fingertip areas in complex backgrounds. When the image entropy value is low, the system appropriately relaxes the confidence standard to improve the pass rate of the fingertip response, thereby realizing an adaptive confidence mechanism of "careful judgment in complex scenarios and rapid pass in simple scenarios", so that the confidence threshold dynamically floats between different frames. The prediction results undergo time-series stabilization processing. The corresponding fingertip coordinates are extracted from the current frame, the previous frame, and the two frames prior, forming three consecutive frames of fingertip position data. This three-frame data is then used as a short-time window input to an exponentially weighted averaging module for temporal fusion. This processing mechanism applies an exponential decay coefficient to the trajectory of each fingertip point in reverse chronological order. Data closer to the current frame has a higher weight, while historical data further away has a lower weight. This preserves the immediate response of the latest input to the current pose while utilizing historical information to improve the smoothness of the results and avoid position jumps caused by occasional detection errors.Furthermore, the exponential weighted averaging process is controlled by the aforementioned adaptive confidence threshold. Only results with a confidence level higher than the dynamic threshold are included in the weighted calculation, thereby ensuring that the fusion result has sufficient credibility and avoiding the offset or drift problem in temporal fusion caused by low-quality input. It outputs a set of spatially accurate, temporally stable, and structurally reasonable real-time updated precise fingertip position coordinates.

[0035] The above describes the real-time fingertip detection method based on a lightweight model in the embodiments of this application. The following describes the real-time fingertip detection device based on a lightweight model in the embodiments of this application. Please refer to [link / reference]. Figure 2 One embodiment of the real-time fingertip detection device based on a lightweight model in this application includes: The acquisition module 11 is used to acquire hand image data and construct a database of hand skeletal constraint relationships; The hierarchical feature extraction module 12 is used to input hand image data into a preset hybrid density encoder network for hierarchical feature extraction and adaptive fusion of feature levels to generate a comprehensive feature map; The feature mapping fusion module 13 is used to perform feature mapping fusion with the comprehensive feature map and the hand skeleton constraint relationship database to output the initial fingertip position coordinate data; The network structure optimization module 14 is used to optimize the network structure of the hybrid density encoder network using model lightweighting technology to generate a lightweight fingertip detection model. Output module 15 is used to perform inter-frame difference detection and adaptive confidence threshold adjustment based on the lightweight fingertip detection model and initial fingertip position coordinate data, and output a set of real-time updated precise fingertip position coordinates.

[0036] Through the collaborative efforts of the aforementioned components, and by introducing a hand skeleton constraint database and a biomechanical constraint model, combined with a dual "fingertip-constraint" loss function, the accuracy of fingertip detection is significantly improved, particularly in complex scenarios such as multi-finger overlap and partial occlusion, effectively addressing the accuracy limitations of traditional methods. Based on an improved MobileNetV3 backbone network, depthwise separable convolutions, structural pruning, and weight quantization, lightweight model techniques significantly reduce network computational complexity and parameter count, enabling the system to run smoothly on mobile devices and embedded platforms. Through a hybrid density encoder network and asymmetric convolution kernel (1×3 and 3×1) design, directional features of fingertip textures are efficiently extracted, enhancing the feature representation capability of the fingertip region while avoiding redundant feature computation. Employing a feature-level adaptive fusion mechanism and channel attention gateway technology, feature-level weights are dynamically allocated for scenarios of varying complexity, allowing low-level features to focus on fingertip texture details and high-level features to focus on the overall gesture, balancing the expression of local details and global structure. By employing inter-frame differential detection technology and an adaptive confidence threshold mechanism, an intelligent inference strategy based on motion intensity is implemented. A lightweight tracking algorithm is used for stationary states, significantly reducing unnecessary computational overhead and ensuring the stability of detection results. Combining a multi-frame temporal integration strategy and exponentially weighted averaging, jitter in single-frame detection results is effectively suppressed, providing smooth and continuous fingertip trajectories and enhancing the naturalness and comfort of human-computer interaction.

[0037] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0038] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0039] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A real-time fingertip detection method based on a lightweight model, characterized in that, include: The process involves collecting hand image data and constructing a hand skeletal constraint relationship database. Specifically, this includes: collecting hand image data and annotating the joint points in the hand image data to obtain hand skeletal structure data; defining the degrees of freedom for each joint point in the hand skeletal structure data to obtain a joint rotational degrees of freedom dataset; setting angle limits for the joint rotational degrees of freedom dataset based on human anatomical data to obtain joint movement boundary data; calculating the angle relationships between adjacent joints based on the joint movement boundary data to obtain joint angle dependency data; and constructing a hand skeletal constraint relationship database based on the hand skeletal structure data, the joint rotational degrees of freedom dataset, and the joint angle dependency data. The hand image data is input into a preset hybrid density encoder network for hierarchical feature extraction and adaptive feature fusion to generate a comprehensive feature map. Specifically, this includes: inputting the hand image data into the MobileNetV3 backbone network of the hybrid density encoder network for feature extraction to obtain an initial feature map; inputting the initial feature map into the fingertip texture specialization module of the hybrid density encoder network for directional feature extraction to obtain a fingertip texture feature map. The fingertip texture specialization module contains three parallel branches: the first branch uses a 1×3 convolutional kernel to extract horizontal texture, the second branch uses a 3×1 convolutional kernel to extract vertical texture, and the third branch... The original features are preserved using 1×1 convolutions, and the outputs of the three methods are merged into a structured fingertip texture feature map through channel concatenation. This fingertip texture feature map is then input into the inverted residual structure of the hybrid density encoder network for feature enhancement, resulting in an enhanced feature map. The inverted residual structure includes an extension layer, a deep convolutional layer, and a projection layer, where the shallow layers use the ReLU6 activation function and the deep layers use the Swish activation function. This enhanced feature map is then input into the channel compression layer of the hybrid density encoder network for dimensionality reduction, resulting in multi-level feature data. Finally, the multi-level feature data undergoes adaptive feature-level fusion to generate a comprehensive feature map. The process involves fusing the comprehensive feature map with the hand skeleton constraint relationship database using feature mapping to output initial fingertip position coordinate data. Specifically, this includes: performing feature space transformation on the comprehensive feature map to obtain candidate fingertip heatmaps; converting the biomechanical constraint matrix in the hand skeleton constraint relationship database into a feature map-compatible format to obtain a constraint feature map. The original form of this biomechanical constraint matrix is ​​an abstract constraint map structure and parameter mapping relationship, representing the angle constraints, degree of freedom range, and dynamic dependencies between adjacent joints of the hand; performing channel connection operations on the candidate fingertip heatmap and the constraint feature map to generate a constraint-enhanced fingertip prediction map; calculating a dual loss value for the fingertip and constraints based on the constraint-enhanced fingertip prediction map, which includes fingertip positioning loss, joint angle constraint loss, bone length constraint loss, and smoothness constraint loss; and performing peak detection on the constraint-enhanced fingertip prediction map based on the dual loss values ​​for the fingertip and constraints to obtain initial fingertip position coordinate data. The network structure of the hybrid density encoder network is optimized using model lightweighting techniques to generate a lightweight fingertip detection model. Based on the lightweight fingertip detection model and the initial fingertip position coordinate data, inter-frame difference detection and adaptive confidence threshold adjustment are performed to output a real-time updated set of precise fingertip position coordinates. Specifically, this includes: performing pixel-level difference calculations on continuously acquired video frames in real time to obtain an inter-frame difference map and a motion intensity index; performing motion judgment processing on the video frames based on the motion intensity index; when the motion intensity index exceeds a preset threshold, triggering the lightweight fingertip detection model to perform complete inference to obtain the fingertip position of the current frame; when the motion intensity index is lower than the preset threshold, performing lightweight tracking processing on the initial fingertip position coordinate data based on the Lucas-Kanade optical flow method to obtain updated fingertip position data; calculating the scene complexity based on the image entropy value of the current video frame, and dynamically adjusting the confidence threshold based on the scene complexity to obtain an adaptive confidence threshold; determining the fingertip position data for three consecutive frames based on the current frame fingertip position and the updated fingertip position data, and performing an exponentially weighted average processing on the fingertip position data of the three consecutive frames based on the adaptive confidence threshold to output a real-time updated set of precise fingertip position coordinates.

2. The real-time fingertip detection method based on a lightweight model according to claim 1, characterized in that, The step of performing feature-level adaptive fusion on the multi-level feature data to generate a comprehensive feature map includes: The multi-level feature data is divided into four feature level sets with different resolutions, and the feature level sets include first-level features, second-level features, third-level features and fourth-level features; The fingertip texture enhancement module is used to enhance the details of the first-level features and the second-level features in the feature hierarchy set to obtain texture enhancement feature data. The fingertip texture enhancement module contains two branches: one branch applies the Sobel operator to extract gradient information, and the other branch retains the original features. A gesture understanding module is used to extract global features from the third-level and fourth-level features in the feature hierarchy set to obtain gesture understanding feature data. The gesture understanding module uses a global context encoder to capture the overall hand posture. Dynamic weight allocation processing is performed on the texture enhancement feature data and the gesture understanding feature data to obtain weighted multi-scale features; The weighted multi-scale features are processed by depthwise separable deconvolution and pixel shuffling techniques to obtain a comprehensive feature map.

3. The real-time fingertip detection method based on a lightweight model according to claim 1, characterized in that, The process of optimizing the hybrid density encoder network using model lightweighting techniques to generate a lightweight fingertip detection model includes: Based on the L1 norm, structural pruning is performed on the MobileNetV3 backbone network, the fingertip texture specialization module, and the convolutional layers in the inverted residual structure of the hybrid density encoder network to obtain a simplified network structure with reduced parameters. The structural pruning process removes filter channels whose contribution is lower than a preset threshold. The model parameters in the simplified network structure are weighted and quantized to obtain an integer model. The quantization operation is simulated in the forward propagation, and the continuous gradient is used in the back propagation to perform precision compensation on the integer model, so as to obtain the precision-restored quantization model. Based on the teacher model, knowledge distillation is performed on the quantized model for accuracy recovery to obtain the student model. Then, operator fusion is performed on adjacent convolutional layers in the student model to obtain a lightweight fingertip detection model.

4. A real-time fingertip detection device based on a lightweight model, characterized in that, For performing the real-time fingertip detection method based on a lightweight model as described in any one of claims 1-3, the real-time fingertip detection device based on the lightweight model comprises: The acquisition module is used to acquire hand image data and construct a hand skeleton constraint relationship database. Specifically, it includes: acquiring hand image data and annotating the hand image data with joint points to obtain hand skeleton structure data; defining the degrees of freedom for each joint in the hand skeleton structure data to obtain a joint rotational degrees of freedom dataset; setting angle limits on the joint rotational degrees of freedom dataset based on human anatomy data to obtain joint movement boundary data; performing angle relationship calculations between adjacent joints based on the joint movement boundary data to obtain joint angle dependency data; and constructing a hand skeleton constraint relationship database based on the hand skeleton structure data, the joint rotational degrees of freedom dataset, and the joint angle dependency data. The hierarchical feature extraction module is used to input the hand image data into a preset hybrid density encoder network for hierarchical feature extraction and adaptive feature fusion to generate a comprehensive feature map. Specifically, it includes: inputting the hand image data into the MobileNetV3 backbone network of the hybrid density encoder network for feature extraction to obtain an initial feature map; inputting the initial feature map into the fingertip texture specialization module of the hybrid density encoder network for directional feature extraction to obtain a fingertip texture feature map. The fingertip texture specialization module contains three parallel branches: the first branch uses a 1×3 convolutional kernel to extract horizontal texture, and the second branch uses a 3×1 convolutional kernel to extract vertical texture. The third branch uses 1×1 convolution to preserve the original features. The outputs of the three branches are merged into a set of structured fingertip texture feature maps through channel concatenation operations. The fingertip texture feature maps are then input into the inverted residual structure of the hybrid density encoder network for feature enhancement, resulting in an enhanced feature map. The inverted residual structure includes an extension layer, a deep convolutional layer, and a projection layer, where the shallow part of the network uses the ReLU6 activation function and the deep part uses the Swish activation function. The enhanced feature map is then input into the channel compression layer of the hybrid density encoder network for dimensionality reduction, resulting in multi-level feature data. The multi-level feature data is then subjected to feature-level adaptive fusion to generate a comprehensive feature map. The feature mapping fusion module is used to perform feature mapping fusion between the comprehensive feature map and the hand bone constraint relationship database, and output initial fingertip position coordinate data. Specifically, it includes: performing feature space transformation on the comprehensive feature map to obtain a candidate fingertip heatmap; converting the biomechanical constraint matrix in the hand bone constraint relationship database into a feature map-compatible format to obtain a constraint feature map. The original form of this biomechanical constraint matrix is ​​an abstract constraint map structure and parameter mapping relationship, representing the angle restrictions, degree of freedom range, and dynamic dependencies between adjacent joints of the hand; performing channel connection operations on the candidate fingertip heatmap and the constraint feature map to generate a constraint-enhanced fingertip prediction map; calculating a dual loss value for the fingertip and constraints based on the constraint-enhanced fingertip prediction map, where the dual loss value includes fingertip positioning loss, joint angle constraint loss, bone length constraint loss, and smoothness constraint loss; and performing peak detection on the constraint-enhanced fingertip prediction map based on the dual loss value for the fingertip and constraints to obtain initial fingertip position coordinate data. The network structure optimization module is used to optimize the network structure of the hybrid density encoder network using model lightweighting technology to generate a lightweight fingertip detection model. The output module is used to perform inter-frame difference detection and adaptive confidence threshold adjustment based on the lightweight fingertip detection model and the initial fingertip position coordinate data, and output a real-time updated set of precise fingertip position coordinates. Specifically, it includes: performing pixel-level difference calculation on continuously acquired video frames in real time to obtain an inter-frame difference map and a motion intensity index; performing motion judgment processing on the video frames based on the motion intensity index; when the motion intensity index exceeds a preset threshold, triggering the lightweight fingertip detection model to perform complete inference to obtain the fingertip position of the current frame; when the motion intensity index is below a preset threshold... When the value is reached, the initial fingertip position coordinate data is subjected to lightweight tracking processing based on the Lucas-Kanade optical flow method to obtain fingertip position update data; the scene complexity is calculated based on the image entropy value of the current video frame, and the confidence threshold is dynamically adjusted based on the scene complexity to obtain an adaptive confidence threshold; based on the fingertip position of the current frame and the fingertip position update data, the fingertip position data of three consecutive frames are determined, and the fingertip position data of the three consecutive frames is subjected to exponential weighted averaging processing based on the adaptive confidence threshold to output a real-time updated set of precise fingertip position coordinates.

Images