Hand pose detection knowledge distillation method, electronic device, and storage medium

Abstract

This application discloses a knowledge distillation method, electronic device, and storage medium for hand pose detection, relating to the field of knowledge distillation technology. The method includes: synchronously inputting hand detection training images into a teacher model and a student model to obtain first coordinate information, first depth information, second coordinate information, and second depth information; determining a first distillation loss value based on the difference between the first and second coordinate information, and determining a second distillation loss value based on the difference between the first and second depth information; comparing the actual skeleton proportions determined based on the second coordinate information and second depth information with a preset constraint threshold range to determine a third distillation loss value; determining the total loss value, and iteratively optimizing the student model based on the total loss value. This application incorporates a third distillation loss based on skeleton proportion constraints, enabling the student model to maintain high-precision UV coordinate and depth detection capabilities while outputting a reasonable hand shape that conforms to human physiological structure.

Description

Technical Field

[0001] This application relates to the field of knowledge distillation technology, and in particular to a method, electronic device and storage medium for knowledge distillation of hand posture detection. Background Technology

[0002] In the field of hand pose detection in computer vision, 2.5D hand pose detection aims to simultaneously output the two-dimensional UV coordinates (a two-dimensional texture mapping coordinate system that uses the U (horizontal) and V (vertical) axes to locate image pixels) and their pixel-level depth information of hand key points. It is one of the core technologies for realizing natural human-computer interaction on mobile terminals, embedded devices, and other edge devices. To adapt to the limited computing power and storage resources of edge devices, the industry currently typically adopts knowledge distillation technology to transfer the knowledge contained in complex teacher models to lightweight student models, thereby seeking a balance between model size and detection accuracy.

[0003] However, most existing knowledge distillation schemes focus on the transfer of accuracy in a single dimension, that is, independently optimizing the predictive ability of the student model in keypoint UV coordinates or pixel-level depth. Although this separate supervision approach can make the student model approach the performance of the teacher model in a single metric, the direct consequence is that although the hand keypoints output by the student model have high accuracy in local coordinates or depth values, the combined hand often results in deformed hand shapes that do not conform to the real hand shape, such as abnormal finger length and disproportionate joints, which seriously affects its usability in interactive applications. Summary of the Invention

[0004] The main purpose of this application is to provide a knowledge distillation method, electronic device and storage medium for hand posture detection, which aims to solve the technical problem that it is difficult to simultaneously achieve the detection accuracy and hand shape realism of current edge-side student models.

[0005] To achieve the above objectives, this application proposes a hand posture detection knowledge distillation method, which includes: Acquire hand detection training images and synchronously input the hand detection training images into the teacher model and the student model to obtain the first coordinate information and first depth information of each key point output by the teacher model, and the second coordinate information and second depth information of each key point output by the student model; The first distillation loss value is determined based on the difference between the first coordinate information and the second coordinate information, and the second distillation loss value is determined based on the difference between the first depth information and the second depth information; The actual bone ratio is determined based on the second coordinate information and the second depth information. The actual bone ratio is compared with a preset constraint threshold range to obtain a comparison result. The third distillation loss value is determined based on the comparison result. The total loss value is determined based on the first distillation loss value, the second distillation loss value, and the third distillation loss value, and the student model is iteratively optimized based on the total loss value.

[0006] In one embodiment, the step of comparing the actual bone proportion with a preset constraint threshold range to obtain the comparison result includes: By comparing the proportion of any dimension in the actual skeleton proportion with the constraint threshold range of the corresponding dimension, comparison results under multiple dimensions are obtained. The actual skeletal proportions include the length ratio of each phalanx to the metacarpal bone, and the distance ratio between each first finger and second finger or palm root and between each third finger and fourth finger or palm root. The length ratio of any phalanx to the metacarpal bone, and the distance ratio between any first finger and second finger or palm root and between any third finger and fourth finger or palm root are all proportions in one dimension.

[0007] In one embodiment, the step of determining the third distillation loss value based on the comparison result includes: If the comparison result of any dimension is such that the proportion of the dimension exceeds the constraint threshold range of the dimension, then the deviation value of the proportion of the dimension from the threshold range of the dimension is determined. The comparison results of each dimension are iterated, and the third distillation loss value is determined according to each deviation value and the corresponding preset penalty coefficient.

[0008] In one embodiment, the step of determining the total loss value based on the first distillation loss value, the second distillation loss value, and the third distillation loss value includes: Based on the first coordinate information, the first depth information, the second coordinate information, the second depth information, the actual bone ratio, and the constraint threshold range, the degree of error influence of each error source factor in the hand detection training image is evaluated; The first weight of the first distillation loss value, the second weight of the second distillation loss value, and the third weight of the third distillation loss value are determined according to the degree of influence of the error. The total loss value is determined based on the first distillation loss value and the first weight, the second distillation loss value and the second weight, and the third distillation loss and the third weight.

[0009] In one embodiment, the error sources include coordinate drift, insufficient depth accuracy, and hand deformity. The step of evaluating the error impact of each error source in the hand detection training image based on the first coordinate information, the first depth information, the second coordinate information, the second depth information, the actual bone proportion, and the constraint threshold range includes: By comparing the first coordinate information and the second coordinate information, the average coordinate error value is obtained; By comparing the first depth information and the second depth information, the average depth error value is obtained; By comparing the actual bone ratio with the constraint threshold range, the average deviation value of the ratio is obtained; By comparing the average coordinate error value, the average depth error value, and the average scale deviation value, the degree of error influence of each error source factor in the hand detection training image is obtained; The magnitude of the average coordinate error relative to the average depth error and the average proportional deviation is positively correlated with the degree of influence of the coordinate drift factor. The magnitude of the average depth error relative to the average coordinate error and the average proportional deviation is positively correlated with the degree of error influence of the insufficient depth accuracy factor. The magnitude of the proportional average deviation relative to the coordinate average error and the depth average error is positively correlated with the degree of error influence of the hand deformity factor.

[0010] In one embodiment, before the step of comparing the actual bone proportion with a preset constraint threshold range to obtain the comparison result, the method further includes: Obtain a hand detection image dataset, and extract the key point coordinate information and reference depth information of each sample image in the hand detection image dataset; For any sample image, based on the key point coordinate information and reference depth information of the sample image, determine the relative proportions of the hand bones in each dimension of the sample image; After traversing each sample image, the constraint threshold range for each dimension is determined based on the maximum and minimum values ​​of the relative proportions in the same dimension.

[0011] In one embodiment, the step of determining the constraint threshold range for each dimension based on the maximum and minimum values ​​of the relative proportions within the same dimension includes: Based on the preset first margin and the maximum value among the relative proportions in the same dimension, the upper limit of the constraint threshold range in each dimension is determined; Based on the preset second margin and the minimum value among the relative proportions in the same dimension, the lower limit of the constraint threshold range in each dimension is determined.

[0012] In one embodiment, after the step of iteratively optimizing the student model based on the total loss value, the method further includes: A hand image to be detected is acquired and input into an iteratively optimized student model to obtain the inference coordinate information and inference depth information to be output through the iteratively optimized student model. The constraint threshold range is used to perform a proportional verification of the inference coordinate information and the inference depth information; If the ratio verification of the inference coordinate information and the inference depth information fails, the inference coordinate information and the inference depth information are linearly corrected until the linearly corrected inference coordinate information and inference depth information pass the ratio verification and are output.

[0013] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the hand posture detection knowledge distillation method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the hand posture detection knowledge distillation method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application first acquires hand detection training images and simultaneously inputs them into a teacher model and a student model, obtaining the first coordinate information and first depth information of each keypoint output by the teacher model, and the second coordinate information and second depth information of each keypoint output by the student model. Next, a first distillation loss value is determined based on the difference between the first and second coordinate information to optimize coordinate accuracy, and a second distillation loss value is determined based on the difference between the first and second depth information to optimize depth accuracy. Simultaneously, the actual skeleton proportion is calculated based on the second coordinate information and second depth information, and compared with a preset constraint threshold range. A third distillation loss value is determined based on the comparison result to apply physiological structural constraints. Finally, the total loss value is determined by combining the first, second, and third distillation loss values, and the student model is iteratively optimized accordingly.

[0016] Therefore, by simultaneously introducing a two-dimensional precision distillation loss for coordinates and depth, the student model is ensured to learn the high-precision knowledge of the teacher model in keypoint localization and depth prediction. Building upon this, a third distillation loss based on skeletal proportion constraints is incorporated to directly impose physiological constraints on the hand shape structure output by the student model. This forces the student model to simultaneously consider both accuracy learning and structural regularity during the optimization process. Thus, the construction of the total loss value deeply couples the accuracy target with the structural constraints, enabling the iteratively optimized student model to maintain high-precision UV coordinates and depth detection capabilities while outputting a reasonable hand shape that conforms to human physiological structure. This effectively resolves the inherent contradiction between accuracy and realism in edge-side lightweight hand pose detection. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the knowledge distillation method for hand posture detection provided in this application (Example 1). Figure 2 This is a flowchart illustrating Embodiment 2 of the hand posture detection knowledge distillation method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the hand posture detection knowledge distillation method of this application; Figure 4 This is a schematic diagram of the overall process of the hand posture detection knowledge distillation method provided in Embodiment 3 of this application; Figure 5 This is a detailed flowchart illustrating the skeletal constraint distillation and dynamic weight allocation of the hand posture detection knowledge distillation method provided in Embodiment 3 of this application. Figure 6 This is a schematic diagram of the hardware operating environment of the hand posture detection knowledge distillation method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: A hand detection training image is acquired, and the hand detection training image is synchronously input into a teacher model and a student model to obtain the first coordinate information and first depth information of each key point output by the teacher model, and the second coordinate information and second depth information of each key point output by the student model; a first distillation loss value is determined based on the difference between the first coordinate information and the second coordinate information, and a second distillation loss value is determined based on the difference between the first depth information and the second depth information; the actual skeleton ratio is determined based on the second coordinate information and the second depth information, and the actual skeleton ratio is compared with a preset constraint threshold range to obtain a comparison result, and a third distillation loss value is determined based on the comparison result; a total loss value is determined based on the first distillation loss value, the second distillation loss value, and the third distillation loss value, and the student model is iteratively optimized based on the total loss value.

[0024] Existing knowledge distillation schemes mostly focus on single-dimensional accuracy transfer, that is, independently optimizing the student model's prediction ability in keypoint UV coordinates or pixel-level depth. This separate supervision approach, while enabling the student model to approach the teacher model's performance on a single metric, has a direct consequence: although the hand keypoints output by the student model have high accuracy in local coordinates or depth values, the combined results often show deformed hand shapes that do not conform to real hand shapes, such as abnormal finger lengths and disproportionate joints, which seriously affect its usability in interactive applications.

[0025] This application provides a solution that simultaneously introduces a two-dimensional precision distillation loss for coordinates and depth to ensure that the student model learns high-precision knowledge from the teacher model in keypoint localization and depth prediction. Building upon this, a third distillation loss based on skeletal proportion constraints is incorporated to directly impose physiological constraints on the hand shape structure output by the student model, thereby forcing the student model to simultaneously consider both accuracy learning and structural regularity during optimization. Thus, the construction of the total loss value deeply couples the accuracy target with the structural constraints, enabling the iteratively optimized student model to maintain high-precision UV coordinate and depth detection capabilities while outputting a reasonable hand shape that conforms to human physiological structure. This effectively resolves the inherent contradiction between accuracy and realism in edge-side lightweight hand pose detection.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses an electronic device as an example to illustrate this embodiment and the subsequent embodiments.

[0027] Based on this, embodiments of this application provide a method for knowledge distillation of hand posture detection, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the hand posture detection knowledge distillation method of this application.

[0028] In this embodiment, the hand posture detection knowledge distillation method includes steps S10~S40: Step S10: Obtain hand detection training images and synchronously input the hand detection training images into the teacher model and student model to obtain the first coordinate information and first depth information of each key point output by the teacher model, and the second coordinate information and second depth information of each key point output by the student model. It should be noted that the hand detection training images are the input data used in the training phase of the knowledge distillation model, specifically a set of image samples containing hand regions. These images are usually derived from a pre-constructed and labeled hand detection dataset, which contains images of real or synthetic hands in different poses, backgrounds, and lighting conditions, aiming to provide diverse visual feature inputs of hands for the training of student models.

[0029] The key point is the predefined location points that need to be identified and located in the hand pose detection task. These points usually correspond to parts of the hand with clear physiological significance, such as the fingertips, interphalangeal joints, metacarpophalangeal joints, and wrist joints, which are used together to digitally represent the overall spatial pose and local structure of the hand.

[0030] The first coordinate information consists of the coordinate values ​​of each keypoint on the two-dimensional plane of the image, output by the teacher model after forward inference on the hand detection training image. This coordinate information is usually represented by pixel positions, such as UV coordinates, and represents the teacher model's prediction of the precise location of the keypoints, serving as the learning target for the student model in the coordinate prediction dimension. The first depth information consists of the pixel-level depth values ​​corresponding to each keypoint, output by the teacher model after inference on the same hand detection training image. This depth information represents the relative or absolute distance of the keypoints along the direction perpendicular to the image plane, reflecting the teacher model's depth perception ability of the three-dimensional hand structure, and serving as the learning target for the student model in the depth prediction dimension.

[0031] The second coordinate information consists of the predicted coordinates of each keypoint output by the student model under training after inference on the same hand detection training image. This information represents the student model's estimation of the keypoint positions under its current parameter state and will be continuously optimized during training by comparing it with the first coordinate information. The second depth information consists of the predicted depth values ​​corresponding to each keypoint output by the same student model. This information represents the student model's current estimation of the depth dimension and is another core predictive output that needs to be improved through distillation during training.

[0032] Step S20: Determine the first distillation loss value based on the difference between the first coordinate information and the second coordinate information, and determine the second distillation loss value based on the difference between the first depth information and the second depth information; It should be noted that the first distillation loss value is a scalar value, obtained by calculating the difference between the second and first coordinate information. This loss value specifically measures the deviation between the student model and the teacher model in keypoint coordinate prediction. Its purpose is to reduce this deviation through backpropagation during training, thereby driving the student model to learn the high-precision knowledge of coordinate localization from the teacher model.

[0033] The second distillation loss value is also a scalar value, obtained by calculating the difference between the second depth information and the first depth information. This loss value specifically measures the deviation between the student model and the teacher model in keypoint depth prediction, aiming to guide the student model to absorb the teacher model's knowledge in depth estimation, thereby improving its own depth prediction accuracy.

[0034] Step S30: Determine the actual bone ratio based on the second coordinate information and the second depth information, compare the actual bone ratio with a preset constraint threshold range to obtain a comparison result, and determine the third distillation loss value based on the comparison result; It should be noted that the actual skeletal proportions refer to a series of proportional measurements of the hand's skeletal structure derived from the second coordinate and second depth information currently output by the student model, using a predetermined geometric calculation formula. These proportions reflect the relative lengths or distances between various skeletal parts (such as between finger joints) in the hand pose predicted by the student model, and are used to assess the physiological plausibility of the predicted hand shape.

[0035] The constraint threshold range is a predefined numerical interval, with its upper and lower limits being reasonable ranges of bone proportions that conform to the physiological structure of the human hand, derived from a large amount of real hand data. This range serves as an objective standard to determine whether the actual bone proportions are within acceptable normal deformation, thus introducing structural constraints into the knowledge distillation process.

[0036] The third distillation loss value is also a scalar value, determined by comparing the calculated actual bone proportions with a constraint threshold range. If the actual proportions exceed the threshold range, a penalty loss value is generated based on the degree of deviation; if they are within the range, the loss value can be zero. This loss value is introduced to constrain the student model's predictions of deformed hand shapes that do not conform to physiological structure, thereby ensuring that the optimized model has the ability to maintain the realism of the hand shape.

[0037] Step S40: Determine the total loss value based on the first distillation loss value, the second distillation loss value, and the third distillation loss value, and iteratively optimize the student model based on the total loss value.

[0038] Understandably, existing knowledge distillation schemes in hand pose detection focus only on the single-dimensional accuracy transfer of keypoint coordinates or depth information, often resulting in deformed hand shapes with skeletal disproportion in the output of student models. Therefore, this embodiment acquires hand detection training images and simultaneously inputs them into the teacher model and student model, obtaining the first coordinate information and first depth information of each keypoint output by the teacher model, and the second coordinate information and second depth information of each keypoint output by the student model. Then, based on the difference between the first and second coordinate information, a first distillation loss value is determined to optimize coordinate accuracy, and based on the difference between the first and second depth information, a second distillation loss value is determined to optimize depth accuracy. Simultaneously, based on the second coordinate information and second depth information, the actual skeletal proportion is determined. By comparing the actual skeletal proportion with a preset constraint threshold range, a comparison result is obtained, and a third distillation loss value is determined to apply physiological structural constraints. Finally, the first, second, and third distillation loss values ​​are combined to determine the total loss value, and the student model is iteratively optimized based on the total loss value. This embodiment effectively avoids the problem of balancing hand shape realism and detection accuracy due to the lack of multi-dimensional collaborative optimization. It enables the student model to output a reasonable hand shape that conforms to human physiological structure while maintaining high-precision coordinates and depth prediction capabilities, effectively solving the inherent contradiction between accuracy and realism in end-side lightweight hand posture detection.

[0039] For example, firstly, hand detection training images are obtained from a standard dataset labeled with hand keypoints. Then, these hand detection training images are simultaneously input into a high-precision teacher model with frozen parameters and a lightweight student model with parameters to be optimized. The teacher model outputs first coordinate information (represented by pixel coordinates) and first depth information (represented by relative depth values) for each keypoint, while the student model outputs second coordinate information and second depth information for the corresponding keypoints. Next, the difference between the first and second coordinate information is calculated using an L1 loss function as the first distillation loss value, and the difference between the first and second depth information is calculated using the same loss function as the second distillation loss value. Simultaneously, the second coordinate information and second depth information are used to calculate the second distillation loss value. The system calculates the actual bone ratio between key hand bones using a predefined geometric formula and compares this ratio with a constraint threshold range statistically generated from a large amount of real hand data. If the actual bone ratio falls within the threshold range, the third distillation loss value is zero; otherwise, it is determined by multiplying the excess distance by a fixed coefficient. Then, the first, second, and third distillation loss values ​​are added together with preset fixed weights to obtain the total loss value. Finally, the Adam optimizer is used to backpropagate and update the parameters of the student model based on the total loss value. The above steps are iteratively executed until the loss of the student model converges, thus obtaining an end-side hand posture detection model that combines high accuracy and reasonable hand shape.

[0040] This embodiment provides a knowledge distillation method for hand pose detection. By simultaneously introducing a two-dimensional precision distillation loss of coordinates and depth, it ensures that the student model learns high-precision knowledge from the teacher model in keypoint localization and depth prediction. Building upon this, a third distillation loss based on skeletal proportion constraints is incorporated to directly impose physiological constraints on the hand shape structure output by the student model. This forces the student model to simultaneously consider both accuracy learning and structural regularity during the optimization process. Thus, the construction of the total loss value deeply couples the accuracy target with the structural constraints, enabling the iteratively optimized student model to maintain high-precision UV coordinate and depth detection capabilities while outputting a reasonable hand shape that conforms to human physiological structure. This effectively resolves the inherent contradiction between accuracy and realism in edge-based lightweight hand pose detection.

[0041] In one feasible implementation, the step of comparing the actual bone ratio with a preset constraint threshold range in step S30 to obtain the comparison result may include step S31: Step S31: Compare the proportion of any dimension in the actual bone proportion with the constraint threshold range of the corresponding dimension to obtain the comparison results under multiple dimensions; The actual skeletal proportions include the length ratio of each phalanx to the metacarpal bone, and the distance ratio between each first finger and second finger or palm root and between each third finger and fourth finger or palm root. The length ratio of any phalanx to the metacarpal bone, and the distance ratio between any first finger and second finger or palm root and between any third finger and fourth finger or palm root are all proportions in one dimension.

[0042] It should be noted that the length ratio of each phalanx to the metacarpal bone refers to the ratio obtained by dividing the three-dimensional pixel length of a particular phalanx (such as the proximal, middle, or distal phalanx of the index finger) by the three-dimensional pixel length of its connected metacarpal bone (i.e., the metacarpal bone corresponding to the palm portion) in a skeletal model formed by key points of the hand. This ratio is used to quantify the relative length relationship between the finger segment and the basic part of the palm in a specific finger, and is one of the important dimensions for assessing whether the local skeletal structure of the hand conforms to physiological norms.

[0043] The ratio of the distance between the first finger and the second finger or the base of the palm to the distance between the third finger and the fourth finger or the base of the palm refers to the following: First, calculate the three-dimensional Euclidean distance from a designated key point (e.g., fingertip) of the first finger to a designated key point of the second finger or to a key point of the base of the palm, as the numerator; simultaneously calculate the three-dimensional Euclidean distance from a designated key point of the third finger to a designated key point of the fourth finger or to a key point of the base of the palm, as the denominator; divide the numerator by the denominator to obtain the ratio. This ratio is used to quantify the coordination of the relative spatial distances between different fingers of the hand, or between fingers and the base of the palm, and is another key dimension for evaluating the rationality of hand shape from the perspective of overall spatial structure.

[0044] In this context, the first and second fingers are two different fingers, and the third and fourth fingers are also two different fingers. The first and third fingers can be the same finger or different fingers. If the first and third fingers are the same finger, then the second and fourth fingers are different fingers; if the first and third fingers are different fingers, then the second and fourth fingers can be the same finger or different fingers.

[0045] Understandably, due to the complexity of the hand's physiological structure, a single or general proportional constraint cannot comprehensively and accurately depict the true proportional relationships of the hand's bones, potentially leading to detection failures for certain types of deformed hands, such as excessively long phalanges or abnormal relative positions between specific fingers. Therefore, this implementation further defines the actual bone proportions as two main dimensions: the length ratio between each phalanx and metacarpal bone, and the distance ratio between each first finger and second finger or palm base, and the distance between each third finger and fourth finger or palm base. A corresponding constraint threshold range is set for each independent dimension for comparison. This avoids structural constraint blind spots caused by incomplete or vague constraint dimensions, preventing the coverage of all key anatomical proportional relationships. This achieves a more refined and comprehensive multi-dimensional quantitative assessment and constraint of the hand's physiological rationality, ensuring that the optimized student model's output hand shape conforms to the structural characteristics of a real hand across multiple key anatomical dimensions, significantly improving the level of hand shape realism.

[0046] For example, in the process of determining the actual bone proportion based on the second coordinate information and the second depth information, the proportions of multiple independent dimensions are specifically calculated, including the first length ratio of the length of the proximal phalanx of the index finger to the length of the metacarpal bone of the index finger, and the first distance ratio of the three-dimensional pixel distance between the key points of the thumb tip and the key points of the index finger tip to the three-dimensional pixel distance between the key points of the little finger tip and the key points of the palm root. Subsequently, when comparing the actual bone proportion with the preset constraint threshold range, the first length ratio is compared with the threshold range of "length ratio of the proximal phalanx of the index finger to the metacarpal bone" obtained in advance based on training data, such as [0.15, 0.25], and the first distance ratio is independently compared with the threshold range of "thumb-index finger distance to little finger-palm root distance" obtained in advance, such as [0.8, 1.20], so as to obtain independent comparison results of multiple specific dimensions such as the length proportion dimension and the distance proportion dimension.

[0047] In one feasible implementation, the step of determining the third distillation loss value based on the comparison result in step S30 may include steps S32-S33: Step S32: If the comparison result of any dimension is that the proportion of the dimension exceeds the constraint threshold range of the dimension, determine the deviation value between the proportion of the dimension and the threshold range of the dimension. It should be noted that the deviation value refers to the quantified distance by which the calculated proportion value in a specific dimension of the actual bone ratio exceeds the boundary (upper or lower limit) of the preset constraint threshold range for that dimension. For example, if the actual calculated length ratio of a phalanx to a metacarpal is 0.28, while the upper limit of the constraint threshold range for that dimension is 0.25, then the deviation value is 0.03. This value accurately measures the extent to which the predicted proportion deviates from the reasonable physiological range, providing a specific numerical basis for subsequent loss calculations.

[0048] The penalty coefficient is a pre-defined positive weighting factor for each bone proportion dimension. When the actual proportion of a certain dimension exceeds its constraint threshold, the penalty coefficient is multiplied by the deviation value calculated for that dimension, together constituting the contribution of that dimension to the overall structural constraint loss. This coefficient determines the severity of the penalty for different types or degrees of bone proportion mismatch during model training, and its value can be adjusted based on prior knowledge or experience to balance the intensity between structural constraints and accuracy optimization.

[0049] Step S33: Iterate through the comparison results of each dimension and determine the third distillation loss value based on each deviation value and the corresponding preset penalty coefficient.

[0050] Understandably, simply judging whether the proportion exceeds the threshold range cannot quantify the severity of the deformity, which may lead to indiscriminate constraints on slight deviations and severe deformities during training, thus affecting the efficiency and accuracy of model optimization. Therefore, this embodiment further quantifies the degree of deformity by first calculating the specific deviation value between the proportion and the threshold boundary when the proportion in each dimension exceeds its corresponding constraint threshold range. Then, it iterates through all dimensions, multiplies the deviation value of each dimension by the penalty coefficient set in advance for that dimension, and synthesizes them to obtain the third distillation loss value. This avoids the problem of rigid constraint strength caused by the lack of quantitative evaluation of the degree of excess, and realizes differentiated and refined penalties for different degrees of skeletal proportion deformities. This allows the model training process to adaptively adjust the strength of structural constraints according to the severity of the deformity, thereby guiding the student model to converge towards an output hand shape that conforms to the physiological proportions of all dimensions more efficiently and accurately.

[0051] For example, when determining the third distillation loss value, if the actual ratio of the dimension "ratio of index finger proximal phalanx to metacarpal length" (0.28) exceeds its upper threshold range of 0.25, then the deviation value of this dimension is calculated to be 0.03; simultaneously, if the actual ratio of the dimension "ratio of thumb-index finger distance to little finger-palmar distance" (0.7) is lower than its lower threshold range of 0.8, then the deviation value of this dimension is calculated to be 0.1; subsequently, the deviation value of the first dimension (0.03) is multiplied by a preset penalty coefficient of 1.5 to obtain the penalty loss contributed by this dimension (0.045), and the deviation value of the second dimension (0.1) is multiplied by a preset penalty coefficient of 1.0 to obtain the penalty loss contributed by this dimension (0.1); finally, the penalty losses obtained after traversing all dimensions are added together, for example, 0.045 and 0.1 are summed to obtain the third distillation loss value of 0.145.

[0052] In one feasible implementation, before the step of comparing the actual bone ratio with a preset constraint threshold range in step S30 to obtain the comparison result, steps S01 to S03 may also be included: Step S01: Obtain the hand detection image dataset and extract the key point coordinate information and reference depth information of each sample image in the hand detection image dataset; It should be noted that the hand detection image dataset is a pre-built collection of images used for model training or analysis, or it can be derived from the full sample data output by the teacher's model. This collection contains multiple images of hand regions with different hand poses, backgrounds, and lighting conditions, and each image typically includes ground truth annotations of corresponding hand keypoints, providing a data foundation for subsequent feature extraction and statistical regularity analysis.

[0053] Keypoint coordinate information refers to the positional data of key anatomical points of the hand on the two-dimensional plane of the image, obtained from sample images in the hand detection image dataset. This information is usually represented in pixel coordinates, such as UV coordinates, which clearly defines the precise location of each predefined keypoint, such as each fingertip or joint, in the image.

[0054] Reference depth information refers to the depth values ​​obtained from sample images of the same hand detection image dataset, corresponding to the keypoint coordinate information mentioned above. This information represents the distance of each keypoint along the camera's optical axis and can be a relative depth relative to a reference point (such as the wrist joint) or an absolute depth value, providing third-dimensional information about the hand structure.

[0055] Step S02: For any sample image, based on the key point coordinate information and reference depth information of the sample image, determine the relative proportions of the hand bones in each dimension of the sample image. It should be noted that relative proportion refers to the proportional value describing the length or distance relationship between different parts of the hand skeleton, obtained by using a predefined geometric calculation formula based on the extracted keypoint coordinate information and reference depth information of any sample image. For example, the ratio of the three-dimensional length of a certain phalanx to the three-dimensional length of the adjacent metacarpal bone is a specific relative proportion used to quantify the intrinsic physiological structural characteristics of the hand.

[0056] Step S03: After traversing each sample image, determine the constraint threshold range for each dimension based on the maximum and minimum values ​​of the relative proportions in the same dimension.

[0057] It is understandable that, since the preset constraint threshold range serves as the core basis for structural constraints, its rationality directly determines the final effect of knowledge distillation. If the generation method of this range is subjective or based on incomplete sample statistics, it will lead to deviations in the constraint standard itself, failing to accurately reflect the reasonable variation range of the real hand physiological structure. Therefore, this embodiment first acquires a hand detection image dataset containing diverse samples, extracts the key point coordinate information and baseline depth information of each sample, and then calculates the relative proportion of bones in multiple dimensions for each sample. Finally, it determines the constraint threshold range for each dimension based on the maximum and minimum values ​​of the relative proportions of all samples in the same dimension. This effectively avoids the problem of unreasonable thresholds and potential mis-exclusion of real and reasonable hand shape variations caused by the constraint threshold range setting relying on subjective experience or using biased statistical methods such as quantile filtering. It achieves the objective generation of constraint threshold ranges based on the full distribution of real data, covering the vast majority of reasonable hand shape structures, providing a reliable and complete objective standard for subsequent bone proportion constraint distillation, thereby ensuring the correctness of the optimization direction for the realism of hand posture from the source.

[0058] For example, a labeled hand detection image dataset is obtained, containing thousands of hand images covering different gestures, viewpoints, and lighting conditions. Then, using a preprocessing script or tool, the two-dimensional pixel coordinates of all predefined hand keypoints are extracted from each sample image in the dataset, and pixel-level reference depth information corresponding to each keypoint is simultaneously extracted with the wrist joint as the reference point. Then, for any sample image, based on its keypoint coordinates and reference depth information, the following parameters are calculated using three-dimensional spatial distance formulas: the ratio of the three-dimensional pixel length of the proximal phalanx of the index finger to the three-dimensional pixel length of the metacarpal bone of the index finger, and the distance from the tip of the thumb to the tip of the index finger. The data includes multiple dimensions such as the relative proportions of the three-dimensional pixel distance and the ratio of the three-dimensional pixel distance from the palm base to the tip of the little finger. After traversing all sample images in the dataset, for the dimension of "the ratio of the length of the proximal phalanx of the index finger to the length of the metacarpal bone", the maximum value (e.g., 0.24) and minimum value (e.g., 0.16) of the relative proportions calculated from all samples are found. These maximum and minimum values ​​are directly determined as the upper and lower limits of the constraint threshold range for this dimension, thus forming an objective threshold range, such as [0.16, 0.24]. This process is repeated for all other preset bone proportion dimensions, and finally a complete set of constraint threshold ranges based on full data statistics is generated.

[0059] In one feasible implementation, the step of determining the constraint threshold range for each dimension based on the maximum and minimum values ​​of the relative proportions under the same dimension in step S03 may include steps S031-S032: Step S031: Based on the preset first margin and the maximum value among the relative proportions in the same dimension, determine the upper limit of the constraint threshold range in each dimension; Step S032: Based on the preset second margin and the minimum value among the relative proportions in the same dimension, determine the lower limit of the constraint threshold range in each dimension.

[0060] It should be noted that the first margin and the second margin can be the same or different, and this implementation does not specify the value of the margin.

[0061] Understandably, directly using the maximum and minimum values ​​obtained from sample statistics as threshold boundaries is too strict, failing to consider the inherent slight fluctuations in model prediction and the existence of extreme but reasonable rare hand shape samples in the dataset. This may lead to unnecessary constraint penalties on predictions that are actually near reasonable boundaries during training, thus affecting the stability and generalization ability of model optimization. Therefore, this implementation further determines the upper limit by pre-setting a positive first margin (e.g., 5%) for the maximum value of each relative proportion in the same dimension, and a positive second margin (e.g., 5%) for the minimum value, thereby appropriately expanding the original statistical extreme value range to generate the final constraint threshold range. This implementation avoids the over-constraint problem that may be caused by overly rigid threshold boundary settings and a lack of fault tolerance space, achieving a reasonable softening of the constraint threshold range based on objective statistics, enabling it to accommodate reasonable fluctuations in model prediction and cover a wider range of real hand shape physiological variations, thereby improving the robustness of structural constraints and the stability of model training during knowledge distillation.

[0062] For example, when determining the constraint threshold range based on the maximum and minimum values ​​of each relative proportion under the same dimension, for the dimension of "the ratio of the length of the proximal phalanx of the index finger to the length of the metacarpal bone", a first margin of 5% and a second margin of 5% are set; if the statistically obtained maximum value of this dimension is 0.24, then the upper limit is calculated as 0.24 × (1 + 5%) = 0.252; if the minimum value is 0.16, then the lower limit is calculated as 0.16 × (1 - 5%) = 0.152; finally, the constraint threshold range of this dimension is determined to be [0.152, 0.252]; for each other bone proportion dimension, the same preset margin is used, and the upper and lower limits are calculated independently based on the statistical maximum and minimum values ​​of each dimension, thereby generating a set of constraint threshold ranges with uniform safety margins.

[0063] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The step of determining the total loss value based on the first distillation loss value, the second distillation loss value, and the third distillation loss value in step S40 may include steps S41 to S43: Step S41: Based on the first coordinate information, the first depth information, the second coordinate information, the second depth information, the actual bone ratio, and the constraint threshold range, evaluate the degree of error influence of each error source factor in the hand detection training image; It should be noted that the degree of error impact of each error source factor refers to the relative contribution or severity level of different error types (such as coordinate prediction bias, depth prediction bias, and bone proportion misalignment) to the overall prediction inaccuracy of the image, quantified and evaluated during the knowledge distillation training process for each hand detection training image. This is achieved by comprehensively analyzing the differences between the second coordinate information, second depth information, and the actual bone proportions calculated based on this information currently output by the student model, and the first coordinate information, first depth information, and preset constraint threshold range provided by the teacher model. This evaluation result is a multi-dimensional metric used to characterize which error sources are the main optimization bottlenecks under the current training samples, thus providing a basis for subsequent dynamic allocation of loss weights.

[0064] Step S42: Determine the first weight of the first distillation loss value, the second weight of the second distillation loss value, and the third weight of the third distillation loss value according to the degree of influence of the error. It should be noted that the sum of the first weight, the second weight, and the third weight is one.

[0065] Step S43: Determine the total loss value based on the first distillation loss value and the first weight, the second distillation loss value and the second weight, and the third distillation loss and the third weight.

[0066] Understandably, since the main sources of error in different training samples, such as coordinate drift, inaccurate depth or hand shape deformity, and their severity vary significantly, it is difficult to use fixed loss weights to perform differentiated and precise optimization for the error characteristics of different samples. This results in a lack of targeting in the training process and limits the overall convergence efficiency and the final performance of the model. Therefore, this embodiment further evaluates the error impact of each error source factor in the current hand detection training image based on the first coordinate information, the first depth information, the second coordinate information, the second depth information, the actual bone ratio, and the constraint threshold range. Then, based on the evaluation results, it dynamically determines the first, second, and third weights corresponding to the first, second, and third distillation loss values. Finally, it obtains the total loss value by weighted summation based on these dynamic weights. This avoids the problem of insufficient training targeting caused by the inability to adapt to the heterogeneity of error distribution among samples due to the use of fixed loss weights. It realizes the adaptive adjustment and optimization focus based on the specific error composition of each training sample, making the model training process more targeted, thereby significantly improving the efficiency and effect of knowledge distillation. Ultimately, it enables the student model to achieve the dual goals of high accuracy and high hand shape realism more evenly and robustly in complex and ever-changing scenarios.

[0067] For example, for each hand detection training image, after obtaining the first distillation loss value, the second distillation loss value, and the third distillation loss value, a preset weight generation network is used to evaluate the error influence of each error source factor. The weight generation network takes the first distillation loss value, the second distillation loss value, and the third distillation loss value as input features, and after processing through a fully connected layer and a softmax activation function, outputs three normalized weight values, which are determined as the first weight, the second weight, and the third weight, respectively, and their sum is 1. Subsequently, the total loss value is calculated according to the formula that the total loss value is equal to the first weight multiplied by the first distillation loss value, plus the second weight multiplied by the second distillation loss value, plus the third weight multiplied by the third distillation loss value. Finally, the dynamically calculated total loss value is used to optimize the student model for this iteration, thereby achieving adaptive allocation of optimization focus according to the magnitude of each loss component of the current sample.

[0068] In this embodiment, by dynamically determining the first, second, and third weights corresponding to the first, second, and third distillation loss values ​​based on the degree of influence of each error source factor in the current hand detection training image, and then calculating the total loss value based on these dynamic weights, this approach avoids the problem of training lacking specificity and optimization efficiency caused by dynamic changes in the training process due to the inability to adapt to the main error causes in different training samples, such as coordinate drift, inaccurate depth, or hand shape deformity, caused by using fixed weights to fuse multi-dimensional losses. This enables adaptive adjustment of the optimization focus based on the specific error composition of each sample, allowing the model training resources to be more accurately focused on the most prominent error type. As a result, the model is guided to improve coordinate accuracy, depth accuracy, and hand shape realism in a more efficient and balanced manner, ultimately obtaining a better overall end-side hand pose detection model.

[0069] In one feasible implementation, the error sources include coordinate drift, insufficient depth accuracy, and hand shape deformity. Step S41 may include steps S411-S414: Step S411: Compare the first coordinate information and the second coordinate information to obtain the average coordinate error value; The coordinate average error value refers to a scalar value obtained by calculating the difference between the second coordinate information output by the student model and the first coordinate information output by the teacher model at each key point for the current hand detection training image, such as by using absolute error or Euclidean distance, and taking the arithmetic mean of the difference results of all key points. This value quantifies the average deviation of the student model in predicting the overall key point position on the current sample and is used to assess the severity of coordinate drift factors.

[0070] Step S412: Compare the first depth information and the second depth information to obtain the average depth error value; The depth mean error value refers to a scalar value obtained by calculating the difference between the second depth information output by the student model and the first depth information output by the teacher model at various key points for the same hand detection training image, such as the absolute difference of the depth values, and taking the arithmetic mean of the depth difference results of all key points. This value quantifies the average deviation of the student model in the overall depth prediction on the current sample and is used to assess the severity of the factors that are insufficient in depth accuracy.

[0071] Step S413: Compare the actual bone ratio with the constraint threshold range to obtain the average deviation value of the ratio; The proportional average deviation value refers to a scalar value obtained by first comparing the actual bone proportions calculated based on the student model output with the corresponding constraint threshold ranges in each dimension for the same hand detection training image. For each dimension that exceeds the threshold range, the specific deviation value is calculated, which is the difference between the proportional value and the nearest threshold boundary. Then, the arithmetic mean of the deviations of all dimensions (or all preset dimensions) that need to calculate the deviation value is taken. This value quantifies the average degree of overall bone structure abnormality of the hand shape predicted by the student model on the current sample and is used to assess the severity of hand deformity factors.

[0072] Step S414: Compare the average coordinate error value, the average depth error value, and the average scale deviation value to obtain the degree of error influence of each error source factor in the hand detection training image; The magnitude of the average coordinate error relative to the average depth error and the average proportional deviation is positively correlated with the degree of influence of the coordinate drift factor. The magnitude of the average depth error relative to the average coordinate error and the average proportional deviation is positively correlated with the degree of error influence of the insufficient depth accuracy factor. The magnitude of the proportional average deviation relative to the coordinate average error and the depth average error is positively correlated with the degree of error influence of the hand deformity factor.

[0073] It is understandable that if the evaluation method for the degree of error impact of each error source factor is too subjective or vague, the dynamic weight generation itself will be unstable or inaccurate, and may even introduce noise, weakening the training effect. Therefore, this implementation method clarifies that: first, the average coordinate error value is obtained by comparing the first coordinate information and the second coordinate information; the average depth error value is obtained by comparing the first depth information and the second depth information; and the average deviation value of the proportion is obtained by comparing the actual bone proportion and the constraint threshold range. Then, the relative magnitude of these three specific scalar values ​​is directly compared to objectively determine the degree of error impact of coordinate drift factor, insufficient depth accuracy factor, and hand shape deformity factor, that is, the larger the value, the greater the degree of impact. This scheme avoids the problem of dynamic weight strategy failure or random fluctuation caused by the lack of clear and calculable objective standards for evaluating the degree of error impact. It realizes reliable dynamic adjustment of weights based on intuitive and stable numerical comparison, thereby ensuring the accuracy of multi-dimensional loss fusion and the efficiency of the training process. This allows the student model to perform targeted reinforcement learning based on the most significant error type of each training sample, and the optimization path is clearer and more reasonable.

[0074] For example, for each hand detection training image, firstly, the L1 distances of all keypoints between the second coordinate information output by the student model and the first coordinate information output by the teacher model are calculated, and the arithmetic mean of these distance values ​​is taken to obtain the coordinate average error value (e.g., 1.5 pixels); simultaneously, the L1 distances of all keypoints between the second depth information and the first depth information are calculated, and the arithmetic mean is taken to obtain the depth average error value (e.g., 0.8 depth units); next, the actual skeleton proportions are calculated based on the second coordinate information and the second depth information, and the actual proportion values ​​of multiple dimensions are compared with the constraint threshold range of the corresponding dimensions. For each dimension that exceeds the range, its deviation value is calculated, and the arithmetic mean of all calculated deviation values ​​is taken to obtain the proportion average deviation value (e.g., 0.05); subsequently, the three scalar values ​​(1.5, ...) are directly compared. The relationship between the magnitudes of 0.8 and 0.05 shows that the average coordinate error value of 1.5 is the largest, followed by the average depth error value of 0.8, and the average proportional deviation value of 0.05 is the smallest. Therefore, it is determined that the coordinate drift factor has the highest impact on the current sample, followed by the insufficient depth accuracy factor, and the hand shape deformity factor has the lowest impact. Finally, based on this determination result, the weights are adaptively allocated: the first weight of the first distillation loss value is increased to 0.6, the second weight of the second distillation loss value is decreased to 0.3, and the third weight of the third distillation loss value is decreased to 0.1, thereby achieving enhanced optimization for the current main source of error (coordinate drift).

[0075] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S40, steps S100 to S300 may also be included: Step S100: Obtain the hand image to be detected, and input the hand image to be detected into the iteratively optimized student model, so as to obtain the inference coordinate information and inference depth information to be output through the iteratively optimized student model; It should be noted that the hand image to be detected refers to the input image containing the hand region to be identified, which is collected or acquired in real-time by edge devices such as AR / VR glasses and smartphone cameras in a real-world application scenario after the knowledge distillation training is completed and deployed. This image is unlabeled raw data, which serves as direct input to the optimized student model to trigger a complete hand pose inference process.

[0076] Inference coordinate information refers to the predicted coordinates of each hand keypoint on the two-dimensional plane of the image, which are directly inferred and output by the student model after the hand image to be detected is input into the iteratively optimized model. This information is the model's final estimate of the position of the hand keypoints in the current input image, usually represented in UV coordinate form, and can be directly provided to subsequent applications, such as gesture recognition modules. It is one of the core components of a complete 2.5D hand pose output.

[0077] Inference depth information refers to the depth prediction values ​​corresponding to each keypoint, which are output synchronously by the same student model when inferring the same hand image to be detected. This information represents the model's estimate of the current hand pose distribution along the depth direction in three-dimensional space and is another core component of the complete 2.5D hand pose output.

[0078] Step S200: Use the constraint threshold range to perform a proportional verification on the inference coordinate information and the inference depth information; Step S300: If the ratio verification of the inference coordinate information and the inference depth information fails, the inference coordinate information and the inference depth information are linearly corrected until the linearly corrected inference coordinate information and inference depth information pass the ratio verification and are output.

[0079] It should be noted that the inference coordinate information and the inference depth information are output only after the ratio verification of the inference coordinate information and the inference depth information passes.

[0080] It is understandable that, even after knowledge distillation and iterative optimization, student models may still produce slightly abnormal skeletal proportions in actual deployment on the device, due to fluctuations in input image quality, rare gestures, or inherent errors in the model itself. Directly using these abnormal results would affect the reliability and stability of the final interactive experience. Therefore, this embodiment further adopts a technical solution that involves acquiring the hand image to be detected during the model deployment phase, optimizing the model output inference coordinates and inference depth information, and then actively calling the constraint threshold range pre-stored during the training phase to perform real-time proportional verification. If the verification fails, the output information is immediately linearly corrected until it passes. This avoids the problem that structural constraints from the training phase fail during the inference phase, which could lead to the device still outputting deformed hand shapes that do not conform to physiological structure. This achieves seamless extension of the structural rationality assurance capability obtained during the training phase to the actual application scenario, ensuring that the hand posture output by the device strictly conforms to physiological constraints under any circumstances, thereby significantly improving the robustness, reliability, and user experience of the final product.

[0081] For example, when the optimized student model is deployed on the AR glasses, the system captures a real-time image of the user's hand using the built-in camera and inputs it into the model. After inference, the model outputs the inference coordinates (represented by pixel UV coordinates) and inference depth information (represented by pixel-level absolute depth values) of all key points of the hand. The system then calls the constraint threshold range generated and pre-stored during the training phase, calculates the actual skeletal proportions of the current hand shape based on the inference coordinates and depth information, such as calculating the length ratio of the proximal phalanx of the index finger to the metacarpal bone, and verifies it against the threshold range of the corresponding dimension. If the length ratio is found to be 0.26, exceeding the upper limit threshold of 0.252, the ratio verification is deemed to have failed, and a linear correction logic is initiated: based on the ratio of the excess value of 0.008 to the total excess tolerance, the depth information of the relevant key points constituting the ratio is finely adjusted in reverse. For example, the depth value of the key point at the distal end of the proximal phalanx of the index finger is increased by a small increment, and the ratio is recalculated until it falls within the threshold range. Finally, the corrected inference coordinate information and inference depth information that have passed the ratio verification are output to the gesture interaction application module to complete a real-time, stable hand posture detection that conforms to physiological structure.

[0082] Furthermore, based on the edge-side inference process constructed in this embodiment, this embodiment can also solidify the skeletal point format conversion logic and coordinate system unification logic in the unified inference interface. Specifically, after the student model outputs inference coordinate information and inference depth information that have passed proportional verification (or correction), the interface does not directly output the original data. Instead, according to preset mapping rules, it automatically converts the key point sequence and coordinate values ​​into the standardized skeletal point data format and coordinate system required by the current target interaction scene (such as a specific AR SDK or gesture control library), such as converting from the image pixel coordinate system to the normalized device screen coordinate system or world coordinate system. This allows the same optimized model to be seamlessly integrated directly by calling this unified interface without the need for secondary development or format adaptation for different downstream applications. This avoids the technical problems of high engineering complexity, long development cycle, and high implementation cost caused by the need to develop and maintain multiple customized output modules for the same model due to different edge-side interaction platform standards. It achieves the engineering advantages of "one-time training, one-stop deployment, and universal applicability", greatly reducing the technical threshold and integration cost of integrating high-precision and high-realism hand posture detection models into diverse terminal products.

[0083] In this embodiment, during the edge model deployment phase, the hand image to be detected is first acquired and input into the iteratively optimized student model to obtain inference coordinate information and inference depth information. Then, the constraint threshold range pre-stored during the training phase is called to perform real-time proportional verification of the inference information. If the verification fails, the inference coordinate information and inference depth information are linearly corrected until they pass the proportional verification before being output. This avoids the problem that relying solely on optimization during the training phase cannot completely eliminate the occasional abnormal hand shape output caused by input perturbation or inherent errors during model inference. It realizes real-time and automated structural compliance assurance of the model output results in the final step in the actual edge application environment, ensuring that the hand posture data finally delivered to the application layer strictly conforms to physiological constraints under any circumstances, thereby greatly improving the output stability, reliability and user experience of the edge hand interaction system.

[0084] For example, to help understand the implementation process of the hand posture detection knowledge distillation method obtained in this embodiment combined with the above embodiment one, please refer to... Figure 4 , Figure 4 A simplified flowchart of a knowledge distillation method for hand posture detection is provided, specifically: Figure 4This demonstrates a complete closed-loop process from model initialization to final deployment: the process begins with the initialization of the teacher model (a high-precision 2.5D hand detection model) and the student model (a lightweight 2.5D hand detection model on the edge); then data preprocessing is performed, including loading the hand detection training set, extracting the UV coordinates and pixel-level wrist joint depth of key points, and generating a bone proportion constraint threshold based on a global extremum with a preset margin (e.g., 5%), i.e., generating the constraint threshold range; next, a two-dimensional knowledge distillation stage is entered, which simultaneously performs accuracy distillation (ensuring that the output of the student model in the UV coordinate and depth branches is aligned with the teacher model) and structural distillation (verifying the bone proportions output by the student model based on the generated thresholds), correspondingly determining the schemes for the first, second, and third distillation loss values; during the distillation process, the system calculates the loss values ​​based on UV error, depth... The system dynamically generates adaptive weights α1, α2, and α3, i.e., the first weight, the second weight, and the third weight, based on sample error features such as degree error and hand shape deformity. Using these weights, the L1 loss of UV (first distillation loss value), the L1 loss of depth (second distillation loss value), and the bone proportion penalty loss (third distillation loss value) are weighted and fused to obtain the total composite loss (total loss value). Then, the system enters the training iteration and optimization stage, using backpropagation and the AdamW optimizer to update the student model parameters, and evaluates the accuracy and bone compliance through the validation set. This cycle continues until the model converges, i.e., the student model is iteratively optimized based on the total loss value. Finally, the converged model enters the edge deployment and inference stage, where real-time bone proportion constraint verification and anomaly correction are performed, and a lightweight edge model that can be directly applied is output.

[0085] Further, please refer to Figure 5The core computational details of the distillation process are elaborated: the left side of the flowchart displays the prediction results output by the student model and the teacher model respectively; in the precision distillation part, the coordinate information output by the teacher model and the student model is reshaped (e.g., converted to 21×2 format) and the L1 loss of UV is calculated, as well as the depth L1 loss is calculated based on the depth information of both; in the structural distillation part, the bone pixel ratio is calculated based on the student model output and verified against a pre-stored constraint threshold. If the ratio is within the threshold, the constraint loss is 0; if it exceeds the threshold, the constraint loss is equal to the product of the deviation value and the penalty coefficient; the input of the dynamic weight generation module is "sample error". The "Difference Feature" outputs either "Global Alpha (Stage-based)" or "Sample-level Alpha (PCR-driven)" depending on whether it is a pre-set per-sample weight. This means that the error impact of each error source factor is evaluated and the dynamic weights are determined accordingly. Global Alpha means that all samples share a set of dynamic weights in the same stage, while sample-level Alpha means that a set of dynamic weights is used for a single sample. Finally, all losses are weighted and fused according to the generated dynamic weights to output the total composite loss. This fully connects the application of skeletal proportion constraints with the adaptive fusion of multi-dimensional losses at the micro-computational level.

[0086] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the hand posture detection knowledge distillation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0087] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the hand posture detection knowledge distillation method in Embodiment 1 above.

[0088] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0089] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0090] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0091] The electronic device provided in this application employs the hand pose detection knowledge distillation method described in the above embodiments, which can solve the technical problem that it is difficult to simultaneously achieve both detection accuracy and hand shape realism in current edge-side student models. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the hand pose detection knowledge distillation method described in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0092] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0094] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the hand posture detection knowledge distillation method in the above embodiments.

[0095] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0096] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0097] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following: it acquires a hand detection training image and synchronously inputs the hand detection training image into a teacher model and a student model, obtaining first coordinate information and first depth information of each key point output by the teacher model, and second coordinate information and second depth information of each key point output by the student model; it determines a first distillation loss value based on the difference between the first coordinate information and the second coordinate information, and determines a second distillation loss value based on the difference between the first depth information and the second depth information; it determines the actual skeleton ratio based on the second coordinate information and the second depth information, compares the actual skeleton ratio with a preset constraint threshold range, obtains a comparison result, and determines a third distillation loss value based on the comparison result; it determines a total loss value based on the first distillation loss value, the second distillation loss value, and the third distillation loss value, and iteratively optimizes the student model based on the total loss value.

[0098] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0100] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0101] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described hand pose detection knowledge distillation method. This solves the technical problem that current edge-side student models struggle to simultaneously achieve both detection accuracy and hand shape realism. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the hand pose detection knowledge distillation method provided in the above embodiments, and will not be elaborated upon here.

[0102] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for knowledge distillation in hand posture detection, characterized in that, The hand posture detection knowledge distillation method includes: Acquire hand detection training images and synchronously input the hand detection training images into the teacher model and the student model to obtain the first coordinate information and first depth information of each key point output by the teacher model, and the second coordinate information and second depth information of each key point output by the student model; The first distillation loss value is determined based on the difference between the first coordinate information and the second coordinate information, and the second distillation loss value is determined based on the difference between the first depth information and the second depth information; The actual bone ratio is determined based on the second coordinate information and the second depth information. The actual bone ratio is compared with a preset constraint threshold range to obtain a comparison result. The third distillation loss value is determined based on the comparison result. The total loss value is determined based on the first distillation loss value, the second distillation loss value, and the third distillation loss value, and the student model is iteratively optimized based on the total loss value.

2. The hand posture detection knowledge distillation method as described in claim 1, characterized in that, The step of comparing the actual bone proportion with a preset constraint threshold range to obtain the comparison result includes: By comparing the proportion of any dimension in the actual skeleton proportion with the constraint threshold range of the corresponding dimension, comparison results under multiple dimensions are obtained. The actual skeletal proportions include the length ratio of each phalanx to the metacarpal bone, and the distance ratio between each first finger and second finger or palm root and between each third finger and fourth finger or palm root. The length ratio of any phalanx to the metacarpal bone, and the distance ratio between any first finger and second finger or palm root and between any third finger and fourth finger or palm root are all proportions in one dimension.

3. The hand posture detection knowledge distillation method as described in claim 2, characterized in that, The step of determining the third distillation loss value based on the comparison results includes: If the comparison result of any dimension is such that the proportion of the dimension exceeds the constraint threshold range of the dimension, then the deviation value of the proportion of the dimension from the threshold range of the dimension is determined. The comparison results of each dimension are iterated, and the third distillation loss value is determined according to each deviation value and the corresponding preset penalty coefficient.

4. The hand posture detection knowledge distillation method as described in claim 1, characterized in that, The step of determining the total loss value based on the first distillation loss value, the second distillation loss value, and the third distillation loss value includes: Based on the first coordinate information, the first depth information, the second coordinate information, the second depth information, the actual bone ratio, and the constraint threshold range, the degree of error influence of each error source factor in the hand detection training image is evaluated; The first weight of the first distillation loss value, the second weight of the second distillation loss value, and the third weight of the third distillation loss value are determined according to the degree of influence of the error. The total loss value is determined based on the first distillation loss value and the first weight, the second distillation loss value and the second weight, and the third distillation loss and the third weight.

5. The hand posture detection knowledge distillation method as described in claim 4, characterized in that, The error sources include coordinate drift, insufficient depth accuracy, and hand deformity. The steps for evaluating the impact of each error source in the hand detection training image, based on the first coordinate information, the first depth information, the second coordinate information, the second depth information, the actual bone proportion, and the constraint threshold range, include: By comparing the first coordinate information and the second coordinate information, the average coordinate error value is obtained; By comparing the first depth information and the second depth information, the average depth error value is obtained; By comparing the actual bone ratio with the constraint threshold range, the average deviation value of the ratio is obtained; By comparing the average coordinate error value, the average depth error value, and the average scale deviation value, the degree of error influence of each error source factor in the hand detection training image is obtained; The magnitude of the average coordinate error relative to the average depth error and the average proportional deviation is positively correlated with the degree of influence of the coordinate drift factor. The magnitude of the average depth error relative to the average coordinate error and the average proportional deviation is positively correlated with the degree of error influence of the insufficient depth accuracy factor. The magnitude of the proportional average deviation relative to the coordinate average error and the depth average error is positively correlated with the degree of error influence of the hand deformity factor.

6. The hand posture detection knowledge distillation method as described in claim 1, characterized in that, Before the step of comparing the actual bone proportion with a preset constraint threshold range to obtain the comparison result, the method further includes: Obtain a hand detection image dataset, and extract the key point coordinate information and reference depth information of each sample image in the hand detection image dataset; For any sample image, based on the key point coordinate information and reference depth information of the sample image, determine the relative proportions of the hand bones in each dimension of the sample image; After traversing each sample image, the constraint threshold range for each dimension is determined based on the maximum and minimum values ​​of the relative proportions in the same dimension.

7. The hand posture detection knowledge distillation method as described in claim 6, characterized in that, The step of determining the constraint threshold range for each dimension based on the maximum and minimum values ​​of the relative proportions within the same dimension includes: Based on the preset first margin and the maximum value among the relative proportions in the same dimension, the upper limit of the constraint threshold range in each dimension is determined; Based on the preset second margin and the minimum value among the relative proportions in the same dimension, the lower limit of the constraint threshold range in each dimension is determined.

8. The hand posture detection knowledge distillation method as described in claim 1, characterized in that, After the step of iteratively optimizing the student model based on the total loss value, the method further includes: A hand image to be detected is acquired and input into an iteratively optimized student model to obtain the inference coordinate information and inference depth information to be output through the iteratively optimized student model. The constraint threshold range is used to perform a proportional verification of the inference coordinate information and the inference depth information; If the ratio verification of the inference coordinate information and the inference depth information fails, the inference coordinate information and the inference depth information are linearly corrected until the linearly corrected inference coordinate information and inference depth information pass the ratio verification and are output.

9. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the hand posture detection knowledge distillation method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the hand posture detection knowledge distillation method as described in any one of claims 1 to 8.

Images