A method and system for hand visual pose estimation

Abstract

The application provides a hand visual pose estimation method and system, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a plurality of hand key point three-dimensional coordinates output by a visual perception model; constructing a hand local coordinate system based on a wrist key point and a palm reference key point; converting hand visual observation data to the coordinate system to obtain local hand key point data; determining a palm plane, skeleton direction observation information and joint bending observation information; performing a deterministic forward mapping based on a preset hand physical constraint model containing a rigid skeleton length constraint, a finger reference bending plane constraint and a joint motion range constraint to obtain a constrained pose parameter; and generating and outputting a low-dimensional hand pose vector. Through the method provided by the application, the stability, physical credibility and real-time application adaptability of the pose expression can be improved.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a method and system for estimating hand visual pose. Background Technology

[0002] With the development of smart terminals, robotic systems, and extended reality technologies, hand gestures, as a natural and intuitive human-computer interaction method, are increasingly being applied in scenarios such as gesture recognition, virtual object manipulation, remote robot control, digital human actuation, and immersive interaction. To acquire the user's hand gestures, these systems typically use cameras, depth sensors, or other sensing devices to collect hand images or spatial information. Based on image processing, machine learning, or deep learning models, they identify positional changes in key hand parts, thus providing a data foundation for subsequent interactive control, motion understanding, or posture actuation.

[0003] In related technologies, vision-based hand pose acquisition methods have been widely used due to their advantages such as convenient deployment, natural interaction, and low cost, as they do not require the wearing of dedicated sensors. However, in practical use, hand images are easily affected by factors such as changes in lighting, background interference, occlusion, motion blur, changes in viewpoint, and model detection errors, resulting in jitter, jumps, or local deviations in the positional results of key hand parts. Furthermore, the hand state data obtained by visual perception usually directly reflects the observation results in image space or 3D space, and there may still be inconsistencies between it and the actual hand movements, especially in scenarios of rapid movement, overlapping fingers, or complex gestures, which can easily lead to problems such as unstable pose expression, unnatural movement state, or abnormal downstream control response. For applications such as robot control, virtual hand actuation, and real-time interaction, directly using the above observation results for subsequent processing may increase the complexity of data processing and affect the real-time operation of the system on the terminal device.

[0004] Therefore, in vision-based hand pose acquisition and application, hand observation results are easily affected by complex environments and detection errors, pose expression stability and reliability are insufficient, and data processing adaptability in real-time interactive scenarios is poor, which have become urgent problems to be solved. Summary of the Invention

[0005] This application provides a hand visual pose estimation method and system, aiming to solve the technical problems in the existing vision-based hand pose estimation process, where hand observation data is easily affected by complex environments and detection errors, resulting in insufficient stability and reliability of pose expression, and difficulty in adapting well to real-time interaction and downstream control applications.

[0006] In a first aspect, this application provides a method for estimating hand visual pose, the method comprising: Acquire hand visual observation data, which includes the three-dimensional coordinates of multiple key hand points output by the visual perception model; Based on the wrist key point and palm reference key point among the multiple hand key points, a local coordinate system for the hand is constructed; The visual observation data of the hand is converted to the local coordinate system of the hand to obtain local key point data of the hand; Based on the aforementioned local hand key point data, determine the palm plane and the bone direction observation information and joint bending observation information corresponding to each finger; Based on a preset hand physical constraint model, a deterministic forward mapping is performed on the bone direction observation information and the joint bending observation information to obtain constrained posture parameters; wherein, the preset hand physical constraint model includes rigid bone length constraints, finger reference bending plane constraints and joint range of motion constraints, and the finger reference bending plane constraints are used to define the bending calculation plane of the corresponding finger according to the palm plane; Based on the constrained posture parameters, a low-dimensional hand posture vector is generated and output, which includes skeletal orientation parameters and joint bending angle parameters.

[0007] In one possible design, the construction of a local hand coordinate system based on the wrist key point and palm reference key point among the plurality of hand key points includes: From the plurality of hand key points, determine the wrist key point, the first palm reference key point, and the second palm reference key point to obtain a coordinate system reference point set; Based on the wrist key points and the first palm reference key points, determine the first direction basis vector; Based on the wrist key points, the first palm reference key points, and the second palm reference key points, the palm normal base vector is determined; Based on the first direction basis vector and the palm normal basis vector, an orthogonal coordinate basis is generated to obtain the local coordinate system of the hand.

[0008] In one possible design, the step of converting the hand visual observation data to the hand local coordinate system to obtain local hand key point data includes: Using the wrist key point as the local origin, the three-dimensional coordinates of the multiple hand key points are translated to obtain the translated key point data; Based on the orthogonal coordinate base, coordinate projection is performed on the translated key point data to obtain a set of local coordinate key points; The local coordinate keypoint set is sorted according to the keypoint identifiers of the multiple hand keypoints to obtain the local hand keypoint data.

[0009] In one possible design, determining the palm plane and the corresponding skeletal orientation and joint flexion observation information of each finger based on the local hand key point data includes: Based on the preset key point topology mapping relationship, each hand key point in the local hand key point data is configured with a corresponding hand geometric point identifier to obtain a hand key point topology model. Based on the aforementioned hand key point topology model, determine the set of bone segments formed by adjacent joint points; Based on the endpoint coordinates of each bone segment in the bone segment set, determine the bone orientation observation information corresponding to each bone segment; Based on the angular relationship between adjacent bone segments in the same finger, determine the joint bending observation information for each finger.

[0010] In one possible design, before performing a deterministic forward mapping on the skeletal orientation observation information and the joint flexion observation information based on a preset hand physical constraint model to obtain constrained posture parameters, the method further includes: The rigid bone length constraint is determined based on the preset bone length corresponding to each bone segment in the bone segment set; Based on the palm plane and the corresponding metacarpophalangeal joints of each finger, determine the reference bending plane corresponding to each finger; The finger reference bending plane constraint is determined based on the reference bending plane; The joint motion range constraint is determined based on the preset lower limit and preset upper limit of the angle corresponding to each joint; The rigid bone length constraint, the finger reference bending plane constraint, and the joint range of motion constraint are associated and configured to obtain the preset hand physical constraint model.

[0011] In one possible design, based on a preset hand physical constraint model, a deterministic forward mapping is performed on the skeletal orientation observation information and the joint bending observation information to obtain constrained posture parameters, including: Based on the bone orientation observation information, the candidate bone orientation corresponding to each bone segment is determined; Based on the rigid bone length constraint, the candidate bone direction is normalized, and the end point position of the bone segment is reconstructed according to the corresponding preset bone length to obtain the bone direction parameters constrained by length. Based on the constraints of the finger reference bending plane, the key joint points of the corresponding fingers are planarized to obtain planarized joint data. Based on the planarized joint data and the joint bending observation information, candidate parameters for the joint bending angle are determined. Based on the joint range of motion constraints, the candidate parameters of the joint bending angle are subjected to boundary processing to obtain the constrained posture parameters.

[0012] In one possible design, the planarization of the joint key points of the corresponding finger based on the finger reference bending plane constraint to obtain planarized joint data includes: Based on the palm plane, determine the palm plane normal vector to obtain the plane reference direction; Based on the metacarpophalangeal joints of the target finger and the plane reference direction, determine the target reference bending plane corresponding to the target finger; The interphalangeal joints and fingertip key points of the target finger are projected onto the target reference bending plane to obtain planarized key point data of the target finger; The planarized key point data of each finger are summarized to obtain the planarized joint data.

[0013] In one possible design, the boundary processing of the candidate parameters for the joint bending angle based on the joint range of motion constraints to obtain the constrained posture parameters includes: Obtain the preset lower limit and preset upper limit of the target joint to get the target joint limit range; The candidate parameters of the joint bending angle corresponding to the target joint are compared with the limit range of the target joint to obtain the angle over-limit judgment result. If the angle boundary determination result indicates that the candidate parameter of the joint bending angle is less than the preset angle lower limit, the preset angle lower limit is determined as the target joint bending angle parameter. If the angle boundary determination result indicates that the candidate parameter of the joint bending angle is greater than the preset angle upper limit, the preset angle upper limit is determined as the target joint bending angle parameter; If the angle boundary determination result indicates that the candidate parameter of the joint bending angle is within the target joint limit range, the candidate parameter of the joint bending angle is determined as the target joint bending angle parameter. Based on the target joint bending angle parameters corresponding to each target joint, the joint bending angle parameters in the constrained posture parameters are obtained.

[0014] In one possible design, generating and outputting a low-dimensional hand pose vector based on the constrained pose parameters includes: Multiple length-constrained bone orientation parameters are extracted from the constrained posture parameters to obtain a bone orientation parameter set; Each bone direction parameter in the bone direction parameter group is converted into azimuth and pitch parameters to obtain the angled bone direction parameter group. Multiple joint bending angle parameters are extracted from the constrained posture parameters to obtain a bending angle parameter set; According to the preset splicing order, the angled bone direction parameter group and the bending angle parameter group are spliced ​​together to obtain the low-dimensional hand posture vector; The low-dimensional hand pose vector is output to a target downstream application module, which includes at least one of a gesture recognition model, a virtual hand model, or a robot controller.

[0015] Secondly, this application provides a hand visual pose estimation system, the system comprising: The visual observation acquisition module is used to acquire hand visual observation data, which includes the three-dimensional coordinates of multiple key points of the hand output by the visual perception model. The local coordinate construction module is used to construct a local coordinate system for the hand based on the wrist key point and the palm reference key point among the multiple hand key points; The coordinate transformation module is used to transform the visual observation data of the hand to the local coordinate system of the hand to obtain local key point data of the hand; The observation information determination module is used to determine the observation information of the palm plane and the bone direction and joint bending of each finger based on the local key point data of the hand. The physical constraint mapping module is used to perform deterministic forward mapping on the bone direction observation information and the joint bending observation information based on a preset hand physical constraint model to obtain constrained posture parameters; wherein, the preset hand physical constraint model includes rigid bone length constraints, finger reference bending plane constraints and joint range of motion constraints, and the finger reference bending plane constraints are used to define the bending calculation plane of the corresponding finger according to the palm plane; The posture vector generation module is used to generate and output a low-dimensional hand posture vector based on the constrained posture parameters. The low-dimensional hand posture vector includes skeletal orientation parameters and joint bending angle parameters.

[0016] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method described in the first aspect or various possible designs of the first aspect.

[0017] Fourthly, this application provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to implement the method described in the first aspect or various possible designs of the first aspect.

[0018] This application provides a method and system for estimating hand visual pose. The method obtains basic observation data representing the current state of the hand by acquiring the three-dimensional coordinates of multiple hand key points output by a visual perception model. By constructing a local hand coordinate system based on wrist and palm reference key points and transforming the hand visual observation data to this local coordinate system, the influence of global position and viewpoint changes on pose expression is reduced, allowing subsequent processing to focus more on the hand's own pose changes. By determining palm plane, skeletal direction, and joint flexion observation information based on local hand key point data, the original visual observations are transformed into a pose analysis basis that better conforms to the hand's structural relationships. Furthermore, by including rigid bone length constraints and finger... By referencing a pre-defined hand physical constraint model with bending plane constraints and joint range of motion constraints, a deterministic forward mapping is performed on skeletal orientation observation information and joint bending observation information. This can limit abnormal results that do not conform to the hand structure and joint motion laws during the pose calculation process and reduce the computational burden caused by complex optimization solutions. Finally, a low-dimensional hand pose vector including skeletal orientation parameters and joint bending angle parameters is generated and output based on the constrained pose parameters. This results in a pose representation with good stability, physical reliability, and real-time application adaptability, thereby solving the problems of existing vision-based hand pose estimation, such as the susceptibility of observation results to complex environments and detection errors, insufficient pose representation reliability, and poor data processing adaptability in real-time interactive scenarios. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a hand visual pose estimation method and system provided in this application embodiment; Figure 2 A schematic diagram of key hand points provided in this application embodiment; Figure 3 A schematic diagram of the topological and physical constraints of key hand points provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a hand visual pose estimation system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims and drawings of this application are intended to cover non-exclusive inclusion.

[0022] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B can exist simultaneously, and B exists. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Furthermore, the terms "first," "second," etc., in the specification and claims of this application or in the aforementioned drawings are used to distinguish different objects rather than to describe a specific order, and may explicitly or implicitly include one or more of the features.

[0025] In the description of this application, unless otherwise stated, "multiple" and "at least two" mean two or more (including two), and similarly, "multiple groups" and "at least two groups" mean two or more (including two groups).

[0026] In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, "connected" or "linked" can refer not only to a physical connection, but also to an electrical connection or a signal connection. For instance, it can be a direct connection, i.e., a physical connection, or an indirect connection through at least one intermediate component, as long as the circuit is connected. It can also refer to the internal connection between two components. A signal connection can refer not only to a signal connection through a circuit, but also to a signal connection through a medium, such as radio waves. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, different technical features in this application can be combined with each other.

[0028] In related technologies, hand pose estimation methods based on monocular red-green-blue (RGB) cameras are widely used in scenarios such as gesture recognition, robot teleoperation, virtual hand actuation, and extended reality (XR) interaction. These methods typically acquire images of the user's hand using a monocular camera and utilize hand keypoint detection models to identify parts of the image such as the wrist, metacarpophalangeal joints, interphalangeal joints, and fingertips, thereby obtaining the spatial coordinates of multiple hand keypoints. For example, some hand pose estimation schemes can output the three-dimensional (3D) coordinates of twenty-one hand keypoints in real time. The 3D coordinates of each hand keypoint can include horizontal, vertical, and depth coordinates, thus forming high-dimensional observation data describing the current spatial state of the hand.

[0029] However, the hand keypoint coordinates directly output by the visual model mainly originate from image features and predictions from data-driven models, essentially representing a geometric estimation of the hand's appearance and spatial position. In practical applications, hand images are easily affected by factors such as lighting variations, background interference, motion blur, finger occlusion, viewpoint changes, and depth estimation errors, leading to jitter, jumps, or local deviations in the output keypoint coordinates. Furthermore, since these keypoint coordinates typically directly reflect the positional results in the observation space, they may not consistently conform to the true structure and movement patterns of the human hand. For example, the distance between adjacent joints may vary unreasonably, joint bending may exceed the normal range of motion, or unnatural local postures may occur in scenarios such as finger overlap or rapid movement. In addition, the three-dimensional coordinates of multiple hand keypoints form high-dimensional observation data, which may contain not only valid hand pose information but also global pose changes, detection noise, and redundant spatial information. Directly using this data for robot control, virtual hand model driving, or real-time interactive processing can easily increase the complexity of subsequent data processing and control mapping.

[0030] Therefore, in vision-based hand pose estimation, visual observation results are easily affected by complex environments and detection errors, the consistency between the output pose and the actual hand movement law is insufficient, and the adaptability of high-dimensional observation data in real-time interaction and downstream control is poor, which have become urgent problems to be solved.

[0031] To address the aforementioned technical issues, this application proposes a hand visual pose estimation method and system. This method no longer directly uses the 3D coordinates of multiple hand key points output by the visual perception model as the final pose result. Instead, it first constructs a local hand coordinate system based on wrist key points and palm reference key points to reduce the influence of changes in the overall hand position and viewing angle on pose expression. Then, it extracts palm plane, skeletal direction observation information, and joint bending observation information from the local hand key point data, transforming the original visual observation data into intermediate information that reflects the structural relationships of the hand. Furthermore, it introduces a pre-defined hand physical constraint model, including rigid bone length constraints, finger reference bending plane constraints, and joint range of motion constraints, and performs deterministic forward mapping on the aforementioned intermediate information to obtain constrained pose parameters that conform to the hand structure and joint motion laws. Finally, it generates a low-dimensional hand pose vector based on the constrained pose parameters, including skeletal direction parameters and joint bending angle parameters. This reduces observation data redundancy and computational complexity while improving the stability, reliability, and adaptability of pose expression for real-time interaction and downstream control applications.

[0032] Figure 1 This is a flowchart illustrating a hand visual pose estimation method provided in an embodiment of this application. Figure 1 As shown, the method provided in this application embodiment specifically includes S101 to S106, and S101 to S106 will be described in detail below.

[0033] It should be noted that the method provided in this application embodiment can be executed by a hand visual pose estimation system, which can be deployed on a server, industrial control computer, edge computing device, mobile terminal, extended reality interactive terminal, robot controller, or processing device integrated into a human-computer interaction system. This hand visual pose estimation system can communicate with a camera, visual perception model, robot controller, virtual hand model, gesture recognition model, or extended reality rendering engine to acquire hand visual observation data and output low-dimensional hand pose vectors for downstream applications.

[0034] S101. Obtain hand visual observation data, which includes the three-dimensional coordinates of multiple key hand points output by the visual perception model.

[0035] Among them, hand visual observation data can be obtained by processing hand images using a visual perception model, and hand images can be acquired by an RGB camera.

[0036] It should be noted that the visual perception model can be a model that can identify key parts of the hand and output the coordinates of key points, such as a hand key point detection model. Hand key points can include wrist key points, metacarpophalangeal joint key points, interphalangeal joint key points, and fingertip key points, which can characterize the hand structure.

[0037] In one example, the visual perception model can output the 3D coordinates of 21 hand keypoints, each of which includes lateral, longitudinal, and depth coordinates. If the i-th hand keypoint is denoted as p_i, then p_i can be represented as p_i = (x_i, y_i, z_i), where x_i represents the lateral coordinate of the i-th hand keypoint, y_i represents the longitudinal coordinate of the i-th hand keypoint, and z_i represents the depth coordinate of the i-th hand keypoint. The visual observation data of the hand, composed of the 3D coordinates of multiple hand keypoints, can be used to characterize the spatial observation position of key hand parts in the current frame.

[0038] In one example, the visual perception model can be the Mediapipe Hands hand keypoint detection model, whose output of twenty-one hand keypoints can be represented according to preset keypoint numbers.

[0039] It should be noted that Mediapipe Hands is an exemplary name for a hand keypoint detection model and does not constitute a limitation on the type of visual perception model. For this type of hand keypoint detection model, the output hand keypoints can include key parts such as the wrist, thumb joint, index finger joint, middle finger joint, ring finger joint, and little finger joint.

[0040] It should be noted that the number of hand keypoints output by the visual perception model can be determined according to the keypoint detection model used, and is not limited to 21; the coordinates of each hand keypoint can also be coordinates with depth estimation in the camera coordinate system, image coordinate system, or spatial coordinates after camera parameter transformation.

[0041] S102. Construct a local coordinate system for the hand based on the wrist key point and palm reference key point among multiple hand key points.

[0042] In one possible embodiment, the wrist key point can be used as the local origin of the hand local coordinate system, and the palm reference key point can be a key point located in the palm area that can reflect the palm unfolding direction or the palm plane direction.

[0043] For example, palm reference key points may include the metacarpophalangeal joint key points of the index finger and the metacarpophalangeal joint key points of the little finger, or other key points that can work together with wrist key points to determine the palm reference direction.

[0044] In this embodiment, by using wrist key points and palm reference key points, the position and orientation of the palm region relative to the original observation coordinate system can be determined, thereby constructing a local hand coordinate system to describe the changes in the hand's own posture.

[0045] S103. Convert the visual observation data of the hand to the local coordinate system of the hand to obtain the local key point data of the hand.

[0046] It should be noted that after constructing the local coordinate system of the hand, the three-dimensional coordinates of multiple key hand points output by the visual perception model can be transformed from the original observation coordinate system to the local coordinate system of the hand, so as to obtain the coordinates of each key hand point in the local coordinate system of the hand.

[0047] Local hand keypoint data can include the coordinates of multiple hand keypoints in the local hand coordinate system and their corresponding keypoint identifiers. This transformation process can reduce the influence of camera viewpoint, overall hand spatial position, and overall hand orientation changes on pose representation, allowing subsequent pose estimation to focus more on the relative changes in the hand's own structure.

[0048] S104. Based on the local key point data of the hand, determine the observation information of the palm plane and the corresponding bone direction and joint bending of each finger.

[0049] It should be noted that the palm plane can be determined based on the key points related to the palm region in the local hand key point data, and the skeletal orientation observation information can be determined based on the connection relationship between adjacent hand key points in each finger; the skeletal orientation observation information is used to characterize the spatial orientation relationship between adjacent joints.

[0050] Furthermore, joint bending observation information can be determined based on the angular relationship between adjacent bone segments in the same finger. This joint bending observation information is used to characterize the bending state of the corresponding joint. Thus, the original local hand keypoint data can be transformed into intermediate observation information that is easier to process with physical constraints.

[0051] S105. Based on the preset hand physical constraint model, perform deterministic forward mapping on the skeletal direction observation information and joint bending observation information to obtain constrained posture parameters.

[0052] The preset hand physical constraint model includes rigid bone length constraints, finger reference bending plane constraints, and joint range of motion constraints.

[0053] It should be noted that the preset hand physical constraint model is used to describe the hand's skeletal structure and joint movement patterns.

[0054] Rigid bone length constraint is used to limit the length of bone segments between adjacent joints, finger reference bending plane constraint is used to limit the bending calculation plane of the corresponding finger, and joint motion range constraint is used to limit the joint bending angle to be within the corresponding preset motion range.

[0055] Deterministic forward mapping refers to the process of obtaining constrained posture parameters through a single forward calculation based on skeletal orientation observation information and joint bending observation information according to preset calculation rules. This process does not require repeated iterative optimization.

[0056] In this embodiment, observation results that do not conform to the structural relationship of the hand and the law of joint movement can be constrained during the posture parameter generation process, so that the obtained constrained posture parameters have better physical reliability.

[0057] S106. Based on the constrained posture parameters, generate and output a low-dimensional hand posture vector, which includes bone orientation parameters and joint bending angle parameters.

[0058] It should be noted that the constrained posture parameters may include skeletal orientation parameters and joint bending angle parameters after being processed by a preset hand physical constraint model.

[0059] Skeletal orientation parameters are used to characterize the spatial orientation of the corresponding bone segment in the local coordinate system of the hand, while joint flexion angle parameters are used to characterize the degree of flexion of the corresponding joint.

[0060] In this embodiment, a low-dimensional hand pose vector is generated by combining the bone orientation parameters and joint bending angle parameters in a preset order. This low-dimensional hand pose vector is then output as the hand pose estimation result for use in gesture recognition, virtual hand driving, robot control, or real-time interactive processing.

[0061] This application's embodiments acquire the three-dimensional coordinates of multiple hand key points and construct a local hand coordinate system based on wrist key points and palm reference key points, which can reduce the impact of changes in the overall hand position and viewpoint on posture expression. By determining the palm plane, skeletal direction observation information, and joint bending observation information, and combining a preset hand physical constraint model including rigid bone length constraints, finger reference bending plane constraints, and joint range of motion constraints to perform deterministic forward mapping, abnormal postures that do not conform to the hand structure and joint movement laws can be restricted, and the computational burden caused by repeated iterative optimization can be reduced. By generating and outputting a low-dimensional hand posture vector including skeletal direction parameters and joint bending angle parameters, the stability, physical reliability, and real-time application adaptability of posture expression can be improved.

[0062] In one possible embodiment, the method steps shown in S102 can be implemented by S1021 to S1024, which are described in detail below.

[0063] S1021. Determine the wrist key point, the first palm reference key point, and the second palm reference key point from multiple hand key points to obtain the coordinate system reference point set.

[0064] It should be noted that the wrist key point can be used as a local origin reference point for constructing the local coordinate system of the hand, and the first palm reference key point and the second palm reference key point can be key points located in the palm area that can jointly reflect the palm unfolding direction and the palm plane direction.

[0065] In one example, the first palmar reference key point can be the metacarpophalangeal joint key point of the index finger, and the second palmar reference key point can be the metacarpophalangeal joint key point of the little finger.

[0066] In other examples, the first palm reference keypoint and the second palm reference keypoint can also be adaptively selected based on the keypoint definition output by the visual perception model, as long as they can jointly represent the spatial orientation of the palm area with the wrist keypoint.

[0067] In this embodiment, by determining the wrist key point, the first palm reference key point, and the second palm reference key point from multiple hand key points, a coordinate system reference point set for constructing the local coordinate system of the hand can be obtained.

[0068] S1022. Determine the first direction basis vector based on the wrist key point and the first palm reference key point.

[0069] The first direction basis vector is used to characterize a reference direction of the palm region in the local coordinate system of the hand.

[0070] In one embodiment, a first direction vector pointing from the wrist key point to the first palm reference key point can be determined based on the difference between the three-dimensional coordinates of the first palm reference key point and the three-dimensional coordinates of the wrist key point, and the first direction vector can be normalized to obtain the first direction basis vector.

[0071] For example, when the first palm reference key point is the metacarpophalangeal joint key point of the index finger, the first direction basis vector can be used to represent the direction from the wrist region to the metacarpophalangeal region of the index finger. Through this processing, the direction of one coordinate axis of the local hand coordinate system can be matched with the direction of the hand's own structure, which facilitates the subsequent expression of the position of each hand key point under the hand's own coordinate reference.

[0072] S1023. Based on the wrist key point, the first palm reference key point, and the second palm reference key point, determine the palm normal base vector.

[0073] Among them, the palm normal basis vector is used to characterize the normal direction of the palm plane.

[0074] In one embodiment, a first direction vector can be determined based on the wrist key point and the first palm reference key point, a second direction vector can be determined based on the wrist key point and the second palm reference key point, and then the candidate normal direction of the palm plane can be determined based on the cross product result between the first direction vector and the second direction vector.

[0075] Furthermore, the candidate normal direction can be normalized to obtain the palm normal basis vector. Through the above processing, the palm normal basis vector can reflect the orientation of the palm region relative to the original observation coordinate system, thereby providing the local coordinate system of the hand with coordinate axis directions perpendicular to the palm plane.

[0076] In practical applications, in order to ensure the consistency of the orientation of the local coordinate system of the hand, the orientation of the palm normal vector can be unified according to the recognition results of the left or right hand, the order of key points, or the preset coordinate system orientation rules, so that the local coordinate system of the hand constructed in adjacent frames or under different hand types has a consistent orientation expression.

[0077] S1024. Based on the first direction basis vector and the palm normal basis vector, generate an orthogonal coordinate basis to obtain the local coordinate system of the hand.

[0078] Among them, the orthogonal coordinate basis is used to express the local coordinates of hand visual observation data.

[0079] In one embodiment, a first direction basis vector can be used as an initial coordinate axis direction, and the palm normal basis vector can be used as another initial coordinate axis direction. The third direction basis vector is determined according to the cross product relationship between the two. Further, the above direction basis vectors are orthogonalized and normalized to obtain an orthogonal coordinate basis. The wrist key point is used as the local origin, and combined with the orthogonal coordinate basis, the local coordinate system of the hand can be obtained.

[0080] In a specific example, the three-dimensional coordinate p_0 corresponding to the wrist key point can be used as the local origin, and the three-dimensional coordinate p_5 corresponding to the index finger metacarpophalangeal joint key point and the three-dimensional coordinate p_17 corresponding to the little finger metacarpophalangeal joint key point can be used as palm reference points to construct a local coordinate system for the hand.

[0081] Specifically, the first direction basis vector can be determined based on the direction between p_5 and p_0, and the palm normal basis vector can be determined based on the spatial relationship between p_5, p_17, and p_0. For example, the coordinate basis of the local coordinate system of the hand can be determined according to the following relationships: e_z = normalize((p_5-p_0)×(p_17-p_0)); e_x = normalize(p_5-p_0); e_y = e_z×e_x.

[0082] Where e_z represents the palm normal basis vector, e_x represents the first direction basis vector pointing from the wrist key point to the index finger metacarpophalangeal joint key point, e_y represents the third direction basis vector determined by e_z and e_x, normalize represents normalization processing, and "×" represents vector cross product.

[0083] Using the above method, a local hand coordinate system can be obtained that changes with hand movement but reduces the influence of global rotation and translation.

[0084] The embodiments of this application can construct a local hand coordinate system that matches the hand structure based on wrist key points and palm reference key points, so that hand visual observation data can be transformed from the original observation coordinate reference to the hand's own coordinate reference, providing a stable coordinate basis for subsequent generation of local hand key point data, determination of the palm plane and physical constraint mapping, thereby reducing the impact of changes in the overall hand position and viewing angle on the posture estimation results.

[0085] In one possible embodiment, the method steps shown in S103 can be implemented by S1031 to S1033, which are described in detail below.

[0086] S1031. Using the wrist key point as the local origin, translate the three-dimensional coordinates of multiple hand key points to obtain the translated key point data.

[0087] It should be noted that after constructing the local coordinate system of the hand, the wrist key points can be used as the local origin of the local coordinate system of the hand.

[0088] Specifically, the relative coordinate difference between the three-dimensional coordinates of each hand key point and the three-dimensional coordinates of the wrist key point can be calculated to obtain the positional data of each hand key point relative to the wrist key point. Through this translation process, the influence of the overall translation position of the hand in the original observation coordinate system can be eliminated, so that the translated key point data mainly represents the spatial distribution relationship of each hand key point relative to the wrist key point.

[0089] In one example, if the 3D coordinates of a hand keypoint are p_i and the 3D coordinates of a wrist keypoint are p_0, the translated coordinates of the hand keypoint can be determined based on the coordinate difference between p_i and p_0. After performing the above processing on multiple hand keypoints, the translated keypoint data can be obtained.

[0090] S1032. Based on the orthogonal coordinate base, perform coordinate projection on the translated key point data to obtain a set of local coordinate key points.

[0091] It should be noted that after obtaining the translated keypoint data, the translated coordinates of each hand keypoint can be projected onto the coordinate axes defined by the orthogonal coordinate base to obtain the local coordinates of each hand keypoint in the local coordinate system of the hand. Thus, the hand keypoint data in the original observation coordinate system can be converted into keypoint data under the hand's own coordinate reference.

[0092] Coordinate projection can be understood as performing directional decomposition on the translated keypoint data and the basis vectors in each direction of the orthogonal coordinate system, thereby obtaining the coordinate components of each hand keypoint in the local coordinate system of the hand. Through this coordinate projection process, the influence of overall hand rotation, camera viewpoint changes, and differences in the observation coordinate system on the expression of hand posture can be reduced, resulting in a set of local coordinate keypoints.

[0093] In one example, after transforming the 3D coordinates of multiple hand keypoints to the hand local coordinate system, a normalized set of local keypoint coordinates {p'_i|i=0,1,…,20} can be obtained. Here, p'_i represents the coordinates of the i-th hand keypoint in the hand local coordinate system. Compared to the keypoint coordinates in the original observation coordinate system, p'_i can more stably represent the relative positional relationships within the hand's own structure and can serve as input data for subsequent execution of bone length constraints, finger reference bending plane constraints, and joint range of motion constraints.

[0094] S1033. Sort the set of local coordinate key points according to the key point identifiers of multiple hand key points to obtain local hand key point data.

[0095] It should be noted that the multiple hand keypoints output by the visual perception model typically have corresponding keypoint identifiers. These identifiers indicate the location type of the keypoint within the hand structure, such as wrist keypoints, metacarpophalangeal joint keypoints, interphalangeal joint keypoints, or fingertip keypoints. After obtaining the set of local coordinate keypoints, these keypoints can be sorted and organized according to a preset keypoint identifier order, ensuring that the sorted keypoint data matches the hand's topology.

[0096] In one example, the set of local coordinate key points can be arranged in the order from wrist to fingers and from metacarpophalangeal joints to fingertips; alternatively, it can be arranged according to the key point numbering order preset by the visual perception model. Through the above sorting process, structurally clear local hand key point data can be obtained, which facilitates subsequent determination of palm plane, skeletal direction observation information, and joint bending observation information based on key point identifiers.

[0097] The embodiments of this application can convert hand visual observation data in the original observation coordinate system into local hand key point data in the local hand coordinate system, so that the position expression of each hand key point is based on the wrist key point and the local hand coordinate system as a unified reference, reducing the impact of overall hand translation, overall rotation and observation angle changes on subsequent posture estimation, and providing a standardized data foundation for subsequent determination of palm plane and bone direction observation information and joint bending observation information corresponding to each finger.

[0098] In one possible embodiment, the method steps shown in S104 can be implemented by S1041 to S1044, which are described in detail below.

[0099] S1041. Based on the preset key point topology mapping relationship, configure the corresponding hand geometric point identifier for each hand key point in the local hand key point data to obtain the hand key point topology model.

[0100] It should be noted that the preset key point topology mapping relationship is used to characterize the correspondence between each hand key point output by the visual perception model and the hand geometry.

[0101] Specifically, the position type of each hand key point in the hand structure can be determined based on the key point identifier, and corresponding hand geometric point identifiers can be configured for it.

[0102] For example, some hand keypoints can be configured as wrist geometric points, thumb joint geometric points, index finger joint geometric points, middle finger joint geometric points, ring finger joint geometric points, and little finger joint geometric points, etc. Thus, the local hand keypoint data can be organized into a hand keypoint topology model with hand structural relationships.

[0103] Figure 2 This is a schematic diagram illustrating the numbering of key points on a hand, as provided in an embodiment of this application. Figure 2 As shown, the visual perception model can output twenty-one hand key points. These twenty-one key points are identified according to preset key point numbers 0 to 20, and form a hand key point topology structure through skeletal connection lines. Among them, key point 0 is the wrist key point, serving as the reference point for the hand key point topology structure; key points 1 to 4 are the key points corresponding to the thumb; key points 5 to 8 are the key points corresponding to the index finger; key points 9 to 12 are the key points corresponding to the middle finger; key points 13 to 16 are the key points corresponding to the ring finger; and key points 17 to 20 are the key points corresponding to the little finger.

[0104] Specifically, keypoint 0 corresponds to the wrist keypoint WRIST; keypoint 1 corresponds to the thumb carpometacarpophalangeal joint THUMB_CMC; keypoint 2 corresponds to the thumb metacarpophalangeal joint THUMB_MCP; keypoint 3 corresponds to the thumb interphalangeal joint THUMB_IP; keypoint 4 corresponds to the thumb fingertip THUMB_TIP; keypoint 5 corresponds to the index finger metacarpophalangeal joint INDEX_FINGER_MCP; keypoint 6 corresponds to the index finger proximal interphalangeal joint INDEX_FINGER_PIP; keypoint 7 corresponds to the index finger distal interphalangeal joint INDEX_FINGER_DIP; keypoint 8 corresponds to the index fingertip INDEX_FINGER_TIP; keypoint 9 corresponds to the middle finger metacarpophalangeal joint MIDDLE_FINGER_MCP; and keypoint 10 corresponds to the middle finger proximal interphalangeal joint MIDDLE_FINGER. _PIP, key point 11 corresponds to the middle finger's distal interphalangeal joint MIDDLE_FINGER_DIP, key point 12 corresponds to the middle fingertip MIDDLE_FINGER_TIP; key point 13 corresponds to the ring finger's metacarpophalangeal joint RING_FINGER_MCP, key point 14 corresponds to the ring finger's proximal interphalangeal joint RING_FINGER_PIP, key point 15 corresponds to the ring finger's distal interphalangeal joint RING_FINGER_DIP, key point 16 corresponds to the ring fingertip RING_FINGER_TIP; key point 17 corresponds to the little finger's metacarpophalangeal joint PINKY_MCP, key point 18 corresponds to the little finger's proximal interphalangeal joint PINKY_PIP, key point 19 corresponds to the little finger's distal interphalangeal joint PINKY_DIP, key point 20 corresponds to the little fingertip PINKY_TIP.

[0105] The keypoint topology mapping relationship is shown in Table 1. The hand geometric point identifiers in Table 1 describe the abstract geometric points in the hand's topological structure, the keypoint identifiers represent the keypoint numbers output by the visual perception model, and the standard keypoint names represent the names of the corresponding keypoints in the visual perception model. Through this preset keypoint topology mapping relationship, multiple hand keypoints output by the visual perception model can be converted into a hand keypoint topology model with a clear meaning of hand structure.

[0106] Table 1 Preset Key Point Topology Mapping Relationship In this table, WRIST represents the wrist, THUMB represents the thumb, INDEX_FINGER represents the index finger, MIDDLE_FINGER represents the middle finger, RING_FINGER represents the ring finger, and PINKY represents the little finger; CMC represents the wrist-metacarpophalangeal joint, MCP represents the metacarpophalangeal joint, IP represents the interphalangeal joint, PIP represents the proximal interphalangeal joint, DIP represents the distal interphalangeal joint, and TIP represents the fingertip. The above standard keypoint names are merely an exemplary naming method for keypoints in a visual perception model and do not constitute a limitation on the type of visual perception model or the naming method for keypoints.

[0107] The preset keypoint topology mapping relationship shown in Table 1 establishes a correspondence between keypoint identifiers, hand geometric point identifiers, and the meaning of hand structure. When determining the set of bone segments, the connection relationship between adjacent joints can be determined based on the hand geometric point identifiers; similarly, the hand keypoint topology model can be used to determine the palm plane, bone direction observation information, and joint bending observation information.

[0108] In this embodiment, the hand keypoint topology model may include palm keypoints for determining the palm plane and finger keypoints for determining the flexion state of each finger bone segment and joint. This hand keypoint topology model clarifies the connections and dependencies between keypoints, providing a structural basis for subsequently determining bone segment sets, bone orientation observation information, and joint flexion observation information.

[0109] S1042. Based on the topological model of key hand points, determine the set of bone segments formed by adjacent joint points.

[0110] The set of bone segments is used to represent the connection relationship between adjacent joints in the hand structure. In one possible embodiment, adjacent joints in the same finger can be identified as a bone segment according to the predefined joint connection order in the hand keypoint topology model.

[0111] For example, the connection between the metacarpophalangeal joint key point and the proximal interphalangeal joint key point can be defined as one skeletal segment, the connection between the proximal interphalangeal joint key point and the distal interphalangeal joint key point can be defined as another skeletal segment, and the connection between the distal interphalangeal joint key point and the fingertip key point can be defined as yet another skeletal segment.

[0112] It should be noted that the set of bone segments can include multiple bone segments corresponding to the thumb, or multiple bone segments corresponding to the index, middle, ring, and little fingers. By determining the set of bone segments, discrete local hand keypoint data can be transformed into structured data with skeletal connections.

[0113] S1043. Based on the endpoint coordinates of each bone segment in the bone segment set, determine the bone direction observation information corresponding to each bone segment.

[0114] Among them, the bone orientation observation information is used to characterize the spatial orientation of each bone segment in the local coordinate system of the hand.

[0115] In one embodiment, for each bone segment in the bone segment set, the coordinates of its starting endpoint and ending endpoint can be obtained, and the bone orientation observation information of the bone segment can be determined based on the direction of the ending endpoint coordinates relative to the starting endpoint coordinates. This bone orientation observation information can be represented as a direction vector, a unit direction vector, or other parameters that can represent spatial orientation.

[0116] In this embodiment, by determining the bone orientation observation information, the observation orientation of each finger bone in the local coordinate system of the hand can be obtained, providing a data basis for the subsequent generation of bone orientation parameters by combining rigid bone length constraints.

[0117] S1044. Based on the angular relationship between adjacent bone segments in the same finger, determine the joint bending observation information for each finger.

[0118] Among them, joint bending observation information is used to characterize the bending state of the corresponding finger joints.

[0119] In one embodiment, the bending observation information corresponding to the joint between two adjacent bone segments in the same finger can be determined based on the directional relationship between the two bone segments.

[0120] For example, for a particular finger, the bending observation information corresponding to the proximal interphalangeal joint can be determined based on the direction of the skeletal segment from the metacarpophalangeal joint to the proximal interphalangeal joint, and the direction of the skeletal segment from the proximal interphalangeal joint to the distal interphalangeal joint; the bending observation information corresponding to the distal interphalangeal joint can also be determined based on the direction of the skeletal segment from the proximal interphalangeal joint to the distal interphalangeal joint, and the direction of the skeletal segment from the distal interphalangeal joint to the fingertip.

[0121] It should be noted that joint flexion observation information can be candidate values ​​for joint angles, directional relationship data required for joint angle calculation, or other intermediate information that can characterize the degree of joint flexion. Determining the joint flexion observation information provides a foundation for subsequently combining finger reference flexion plane constraints and joint range of motion constraints to obtain constrained posture parameters.

[0122] The embodiments of this application can convert local hand key point data into a set of bone segments, bone direction observation information and joint bending observation information according to the hand topology. This makes the original key point coordinates no longer just discrete point data, but form intermediate observation information that can reflect the connection relationship of hand bones and the bending relationship of joints, thereby providing structured input for the deterministic forward mapping of the subsequent preset hand physical constraint model.

[0123] In one possible embodiment, prior to the method step shown in S105, the method further includes S201 to S205, which are described in detail below.

[0124] S201. Determine the rigid bone length constraint based on the preset bone length corresponding to each bone segment in the bone segment set.

[0125] It should be noted that each bone segment in the bone segment set is used to represent the connection relationship between adjacent joints, and the preset bone length corresponding to each bone segment can be determined based on the human hand bone structure parameters, user calibration data, or preset hand model parameters.

[0126] For the same user or the same hand model, the bone length between adjacent joints usually remains relatively stable during hand movement. Therefore, the preset bone length corresponding to each bone segment can be used as a rigid bone length constraint to limit abnormal changes in bone length caused by detection errors, occlusion, or depth estimation deviations during visual observation.

[0127] In one example, a corresponding preset bone length can be configured for bone segments such as the metacarpophalangeal joint to the proximal interphalangeal joint, the proximal interphalangeal joint to the distal interphalangeal joint, and the distal interphalangeal joint to the fingertip. The bone segment identifier is associated with the corresponding preset bone length and stored to obtain a rigid bone length constraint.

[0128] S202. Based on the palm plane and the corresponding metacarpophalangeal joints of each finger, determine the reference bending plane corresponding to each finger.

[0129] The reference bending plane is used to provide a planar reference for calculating the joint bending of the corresponding finger.

[0130] It should be noted that the palm plane can be determined by key points related to the palm region in the local hand key point data, and the metacarpophalangeal joints can be used to characterize the position where each finger connects to the palm. Based on the palm plane and the metacarpophalangeal joints corresponding to the target finger, a reference bending plane corresponding to the target finger can be determined, so that the subsequent bending state of the target finger can be described within this reference bending plane.

[0131] In one example, the reference bending plane can be determined based on the normal direction of the palm plane and the position of the metacarpophalangeal joint of the target finger. For the index, middle, ring, and little fingers, their respective reference bending planes can be determined to accommodate the positional differences of different fingers within the palm region.

[0132] S203. Determine the finger reference bending plane constraint based on the reference bending plane.

[0133] It should be noted that the finger reference bending plane constraint is used to limit the bending calculation plane of the corresponding finger, so that the joint bending observation information of the corresponding finger can be analyzed within the reference bending plane. Through this constraint, the influence of lateral finger jitter, key point offset, or visual errors on joint bending calculation can be reduced, making the subsequently obtained joint bending angle parameters more consistent with the structural relationship of finger bending motion.

[0134] In one example, the finger reference bending plane constraint may include the target finger identifier, the reference bending plane corresponding to the target finger, and the interphalangeal joint information to be constrained. Therefore, in the subsequent deterministic forward mapping process, the joint key points of the corresponding finger can be planarized according to this finger reference bending plane constraint.

[0135] S204. Determine the joint range of motion constraints based on the preset lower limit and preset upper limit of the angle corresponding to each joint.

[0136] The preset lower limit and preset upper limit are used to represent the range of bending angles allowed for the corresponding joints.

[0137] It should be noted that the range of motion can vary between different fingers and joints. Therefore, each joint can be configured with a corresponding preset lower angle limit and a preset upper angle limit. Based on the preset lower angle limit and preset upper angle limit for each joint, the joint range of motion constraints can be determined.

[0138] In one example, the joint range of motion constraint may include a joint identifier, a preset lower angle limit, and a preset upper angle limit. This constraint can be used to limit candidate parameters for joint flexion angles, thereby preventing the generation of joint flexion angle parameters that exceed the preset range of motion.

[0139] S205. Configure the rigid bone length constraint, finger reference bending plane constraint, and joint range of motion constraint together to obtain a preset hand physical constraint model.

[0140] It should be noted that the rigid bone length constraint, finger reference bending plane constraint, and joint range of motion constraint define the hand posture from three aspects: bone length, finger bending space, and joint range of motion, respectively. These three types of constraints can be configured by associating them with finger identifiers, bone segment identifiers, and joint identifiers in the hand keypoint topology model, so that each bone segment can correspond to a corresponding preset bone length, each finger can correspond to a corresponding reference bending plane, and each joint can correspond to a corresponding range of motion.

[0141] Through the above-described configuration, a preset hand physical constraint model can be obtained. This preset hand physical constraint model can serve as the constraint basis for subsequent deterministic forward mapping, enabling skeletal orientation observation information and joint bending observation information to be processed under unified hand structure constraints, thereby providing a model basis for generating constrained posture parameters.

[0142] The embodiments of this application can uniformly configure the relationship between the length of the hand bones, the relationship between the plane of finger bending, and the relationship between the range of motion of the joints into a preset physical constraint model of the hand. This allows the subsequent posture parameter generation process to no longer rely solely on the results of visual observation points, but to combine the hand structure and the laws of joint movement to constrain the observation information, thereby improving the stability and physical reliability of the constrained posture parameters.

[0143] In one possible embodiment, various constraints in the preset hand physical constraint model can be represented by mathematical relationships so as to constrain the bone orientation observation information and joint bending observation information during the deterministic forward mapping process.

[0144] For rigid bone length constraints, if a bone segment consists of a starting hand geometry point X and an ending hand geometry point Y, and the preset bone length corresponding to the bone segment is L_XY, and the coordinates of the starting hand geometry point X and the ending hand geometry point Y are p_X and p_Y respectively, then in an ideal case, the bone segment length satisfies L_XY=||p_X-p_Y||.

[0145] Wherein, L_XY is a preset normal value, which can be determined based on the average human hand model, user hand calibration data, or target robot hand structure parameters.

[0146] Furthermore, during deterministic forward mapping, the direction of the bone segment can be determined based on the coordinates corresponding to the starting hand geometry point X and the ending hand geometry point Y, and the direction of the bone segment can be constrained based on the preset bone length.

[0147] For example, the direction vector corresponding to the bone segment XY can be expressed as: v_XY=(p_Y-p_X) / L_XY; where v_XY represents the direction vector corresponding to the bone segment XY.

[0148] It should be noted that, in the case of errors in visual observation, candidate directions can be determined first based on p_Y-p_X, then the candidate directions can be normalized, and the relative positions of the endpoints of the bone segments can be reconstructed in combination with the preset bone length L_XY, so that the reconstructed bone segments satisfy the rigid bone length constraint.

[0149] For multi-point coplanar constraints, if the coordinates of a set of hand geometric points q_1, q_2, q_3, and q_4 are q_1, q_2, q_3, and q_4 respectively, then the mixed product relationship can be used to limit these hand geometric points to lie in the same reference plane. The multi-point coplanar constraint can be expressed as: ((q_2-q_1)×(q_3-q_1))·(q_4-q_1)=0; where "×" represents the cross product and "·" represents the dot product. This multi-point coplanar constraint can limit the deviation of multiple joints in the same finger from the corresponding bending calculation plane, thereby reducing non-coplanar offsets caused by visual observation errors.

[0150] For a two-plane perpendicular constraint, if the normal vector of the palm plane is n_palm and the normal vector of the bending calculation plane corresponding to the target finger is n_finger, then the spatial relationship between the bending calculation plane of the target finger and the palm plane can be limited by the dot product relationship between the two.

[0151] In one example, when the target finger is the index finger, the normal vector of the bending calculation plane corresponding to the index finger can be denoted as n_INDex, and the vertical constraint can be expressed as: n_palm·n_INDex=0. Similarly, when the target finger is the middle finger, ring finger, or little finger, the normal vectors of the corresponding bending calculation planes can be determined respectively, and constraints can be applied based on the same dot product relationship. Through these two-plane vertical constraints, the joint bending of the target finger can be mainly carried out in a plane that matches the palm structure, thereby reducing the influence of the lateral degree of freedom of the finger in the palm plane on the pose estimation.

[0152] For proportional position constraints, if the target hand geometric point C is located on the line connecting the first reference hand geometric point A and the second reference hand geometric point B, and satisfies AC = λ·AB, then the coordinate c corresponding to the target hand geometric point C can be expressed as: c = a + λ·(ba); where a represents the coordinate corresponding to the first reference hand geometric point A, b represents the coordinate corresponding to the second reference hand geometric point B, and λ represents the preset proportional coefficient. Through proportional position constraints, the position of the target hand geometric point can be regularized based on the hand's topological structure, maintaining a preset relative positional relationship between the target hand geometric point and its reference hand geometric points.

[0153] For joint range of motion constraints, if the candidate parameter of the joint bending angle corresponding to the target joint is α_k, the lower limit of the preset angle corresponding to the target joint is α_k^min, and the upper limit of the preset angle corresponding to the target joint is α_k^max, then the joint bending angle of the target joint should satisfy the following relationship: α_k^min≤α_k≤α_k^max; when the candidate parameter of the joint bending angle α_k exceeds the above range, boundary processing can be performed to obtain the target joint bending angle parameter α_k^clamped. The boundary processing can be expressed as: α_k^clamped=max(α_k^min,min(α_k,α_k^max)); where α_k^clamped represents the target joint bending angle parameter after joint range of motion constraint processing.

[0154] Through the above boundary processing, candidate parameters of joint bending angles that are below the preset lower limit can be adjusted to the preset lower limit, candidate parameters of joint bending angles that are above the preset upper limit can be adjusted to the preset upper limit, and candidate parameters of joint bending angles that are within the preset angle range can be retained, thereby avoiding the generation of posture results that exceed the joint movement range.

[0155] By using the aforementioned rigid bone length constraints, multi-point coplanar constraints, two-plane perpendicular constraints, proportional position constraints, and joint range of motion constraints, the pre-defined hand physical constraint model can constrain hand visual observation data from multiple aspects such as bone length, hand topology, finger bending plane, and joint range of motion, making the constrained posture parameters obtained by deterministic forward mapping more consistent with hand structural relationships and joint movement laws.

[0156] In one possible embodiment, deterministic forward mapping can be understood as a forward computation process based on local hand keypoint data. Specifically, various constraints in a pre-defined hand physical constraint model can be applied in parallel based on the normalized set of local keypoint coordinates to obtain constrained posture parameters. For a predefined bone segment, candidate bone directions can be determined based on the hand geometric point coordinates at both ends of the bone segment, and the corresponding length-constrained bone direction parameters can be determined by combining the pre-defined bone length. For a set of points constrained by the finger reference bending plane or geometric topological relationships, one or more candidate joint bending angle parameters can be determined based on the corresponding coplanar, perpendicular, or proportional positional relationships. The calculated candidate joint bending angle parameters can be further processed based on the joint range of motion constraints. Thus, local hand keypoint data can be forward mapped to constrained posture parameters that conform to the hand structure without repeated iterative optimization.

[0157] In one example, the finger reference bending plane constraint restricts the calculation of the joint bending angles of the index, middle, ring, and little fingers to the corresponding bending calculation plane. Since the bending calculation plane is determined based on the palm plane and the metacarpophalangeal joints of the target fingers, this constraint reduces the influence of the lateral degrees of freedom of the fingers in the palm plane on pose estimation, allowing the bending state of the corresponding fingers to be primarily described by the joint bending angles. This provides a basis for converting high-dimensional hand visual observation data into low-dimensional hand pose vectors.

[0158] In one possible embodiment, the method steps shown in S105 can be implemented by S1051 to S1055, which are described in detail below.

[0159] S1051. Based on the bone orientation observation information, determine the candidate bone orientation corresponding to each bone segment.

[0160] It should be noted that skeletal orientation observation information can characterize the observation orientation of each bone segment in the local coordinate system of the hand. For each bone segment in the set of bone segments, the direction from the starting joint point to the ending joint point can be determined based on the coordinates of the starting and ending joint points corresponding to that bone segment, serving as the candidate bone orientation for that bone segment. The candidate bone orientation reflects the initial orientation information derived from the keypoint coordinates output by the visual perception model, and can be subsequently corrected and normalized under the constraints of a preset hand physical constraint model.

[0161] S1052. Based on rigid bone length constraints, the candidate bone directions are normalized, and the relative positions of the bone segment endpoints are reconstructed according to the corresponding preset bone length to obtain the bone direction parameters constrained by length.

[0162] Among them, rigid bone length constraint is used to constrain the bone segment length between adjacent joints.

[0163] Specifically, the candidate bone orientation can be normalized to obtain the unit orientation information of the corresponding bone segment; then, combined with the preset bone length corresponding to the bone segment, the endpoint position or orientation expression of the bone segment can be reconstructed to obtain length-constrained bone orientation parameters. Through this processing, abnormal changes in bone segment length caused by visual observation errors can be reduced, so that the bone orientation parameters retain both the observation orientation information and conform to the preset bone length relationship.

[0164] It should be noted that the bone orientation parameters constrained by length can be orientation vectors after normalization, or endpoint relative position parameters after reconstruction based on preset bone length. The specific form can be determined according to the subsequent generation method of low-dimensional hand pose vectors.

[0165] In one example, for any bone segment in the set of bone segments, the bone segment can be denoted as a bone segment consisting of the starting hand geometric point X and the ending hand geometric point Y, and the preset bone length corresponding to the bone segment can be denoted as L_XY.

[0166] Here, L_XY is a preset normal value, which can be determined based on the average human hand model, user hand calibration data, or target robot hand structural parameters. During deterministic forward mapping, the candidate bone orientation can be determined based on the coordinates corresponding to the starting hand geometric point X and the ending hand geometric point Y, and the relative positions of the bone segment endpoints can be reconstructed by combining the preset bone length L_XY, thereby eliminating or reducing the degrees of freedom related to abnormal changes in bone length in visual observation.

[0167] S1053. Based on the finger reference bending plane constraint, the key joint points of the corresponding finger are planarized to obtain planarized joint data.

[0168] Among them, the finger reference bending plane constraint is used to limit the bending calculation plane of the corresponding finger.

[0169] In one embodiment, the interphalangeal joints, fingertip key points, or relevant key points used to calculate the joint bending angle in the target finger can be planarized according to the reference bending plane corresponding to the target finger, so that the relevant key points are normalized into the corresponding bending calculation plane. The resulting planarized joint data can reduce the impact of lateral offset, key point jitter, or depth estimation errors on joint bending calculation.

[0170] It should be noted that different fingers can have their own corresponding reference bending planes. The planarization process can be performed on the index finger, middle finger, ring finger, and little finger separately, or it can be performed on some fingers as needed.

[0171] S1054. Based on planar joint data and joint bending observation information, determine candidate parameters for joint bending angle.

[0172] After obtaining the planarized joint data, the candidate parameters for the joint bending angle of the corresponding joint can be determined by combining the joint bending observation information.

[0173] Specifically, the bending angle of the target joint can be calculated based on the directional relationship between adjacent bone segments after planarization, thus obtaining candidate parameters for the joint bending angle. These candidate parameters characterize the initial degree of bending of the target joint in the corresponding bending calculation plane.

[0174] It should be noted that the candidate parameters for joint flexion angle can correspond to the proximal interphalangeal joint, distal interphalangeal joint, or other joints that require flexion representation. These candidate parameters are intermediate results obtained based on visual observation information and planarization processing; further boundary processing is required in conjunction with joint range of motion constraints.

[0175] S1055. Based on the joint range of motion constraints, the candidate parameters of the joint bending angle are processed to obtain the constrained posture parameters.

[0176] The joint range of motion constraint is used to limit the allowed range of motion of the corresponding joint. Candidate parameters for joint flexion angles can be compared with the preset lower and upper limits of the corresponding joint angle, and boundary processing can be performed on the candidate parameters based on the comparison results. If a candidate parameter for joint flexion angle exceeds the preset range of motion of the corresponding joint, it can be adjusted to the corresponding boundary value; if a candidate parameter for joint flexion angle is within the preset range of motion, it can be retained as the corresponding joint flexion angle parameter.

[0177] In one possible embodiment, the constrained pose parameters may include length-constrained skeletal orientation parameters and joint flexion angle parameters after boundary processing. By using the skeletal orientation parameters and joint flexion angle parameters as constrained pose parameters, a physically constrained pose parameter basis can be provided for the subsequent generation of low-dimensional hand pose vectors.

[0178] The embodiments of this application can process the skeletal direction observation information and joint bending observation information obtained by visual observation through skeletal length constraint, finger bending plane constraint and joint range of motion constraint in sequence to obtain constrained posture parameters. Since this process performs deterministic forward mapping according to preset calculation rules, it does not require repeated iterative optimization. Therefore, it can improve the physical reliability of posture parameters while reducing the computational burden of posture calculation, and adapt to real-time hand posture estimation applications.

[0179] In this embodiment, the deterministic forward mapping does not merely compress the hand visual observation data numerically. Instead, it embeds rigid bone length constraints, finger reference bending plane constraints, geometric topological relationship constraints, and joint range of motion constraints into the posture parameter generation process before generating the low-dimensional hand pose vector. This reduces the dimensionality of the hand visual observation data while simultaneously correcting for local jitter, instantaneous errors, abnormal bone lengths, and abnormal joint angles in the visual perception model output, resulting in lower-dimensional hand pose vectors with better physical reliability and stability.

[0180] In one possible embodiment, the method steps shown in S1053 can be implemented by Sa1 to Sa4, which are described in detail below.

[0181] Sa1. Determine the normal vector of the palm plane based on the palm plane to obtain the plane reference direction.

[0182] It should be noted that the palm plane can be determined based on key points related to the palm region in local hand key point data, such as wrist key points, first palm reference key points, and second palm reference key points.

[0183] The palm plane normal vector is used to represent the direction perpendicular to the palm plane, and it can serve as a directional reference for determining the plane for calculating the bending of each finger. After determining the palm plane normal vector, this palm plane normal vector can be used as a plane reference direction so that the target reference bending plane corresponding to the target finger can be determined subsequently based on this plane reference direction.

[0184] Sa2. Based on the metacarpophalangeal joints and plane reference direction of the target finger, determine the target reference bending plane corresponding to the target finger.

[0185] The target finger can be any finger for which the bending angle needs to be calculated, and the metacarpophalangeal joint of the target finger can be used to characterize the position where the target finger connects to the palm.

[0186] Specifically, the metacarpophalangeal joint of the target finger can be used as a spatial positioning reference for the target reference bending plane, and the orientation of the target reference bending plane can be determined in combination with the plane reference direction, so that the target reference bending plane can match the bending movement direction of the target finger.

[0187] In one embodiment, the target reference bending plane can be a plane passing through the metacarpophalangeal joint of the target finger and having a predetermined spatial relationship with the palm plane. For example, the target reference bending plane can be perpendicular or approximately perpendicular to the palm plane, so that the bending calculation of the target finger is mainly performed in a plane that matches the palm structure. Different fingers can each determine their own corresponding target reference bending plane, thereby accommodating the positional differences of each finger in the palm region.

[0188] Sa3. Project the interphalangeal joints and fingertip key points of the target finger onto the target reference bending plane to obtain the planarized key point data of the target finger.

[0189] It should be noted that the interphalangeal joints and fingertip keypoints of the target finger can be used to calculate the joint bending state of the target finger. Since the keypoint coordinates output by the visual perception model may be affected by occlusion, jitter, or depth estimation bias, some keypoints of the target finger may deviate from a reasonable bending calculation plane. Therefore, the interphalangeal joints and fingertip keypoints of the target finger can be projected onto the target reference bending plane, so that these keypoints are normalized to the same bending calculation plane, resulting in planarized keypoint data of the target finger.

[0190] In one example, the projection positions of each interphalangeal joint and fingertip keypoint in the target finger onto the target plane can be determined based on the planar position and normal direction of the target reference bending plane. The coordinates of the projected keypoints are then used as the planarized keypoint data of the target finger. This process reduces the impact of lateral finger offset and local keypoint errors on the calculation of joint bending angles.

[0191] Sa4. Summarize the planarized key point data of the target finger corresponding to each finger to obtain the planarized joint data.

[0192] In one embodiment, the above-described planarization process can be performed on at least one of the index, middle, ring, and little fingers to obtain the planarized key point data of each target finger. Subsequently, the planarized key point data of each target finger can be summarized according to finger identifiers, joint identifiers, or a preset processing order to obtain planarized joint data.

[0193] It should be noted that the planarized joint data can include the planarized interphalangeal joints and fingertip key points of each finger used to calculate the joint flexion angle. This planarized joint data can serve as the data basis for subsequently determining candidate parameters for the joint flexion angle, ensuring that the calculation of the joint flexion angle is based on a plane that matches the relationship with the hand structure.

[0194] The embodiments of this application can determine the corresponding target reference bending plane for each finger based on the palm plane, and regularize the key points used for joint bending calculation into the corresponding target reference bending plane, thereby reducing the impact of lateral offset, key point jitter and depth error on bending angle calculation in visual observation, and improving the stability and physical reliability of subsequent joint bending angle candidate parameters.

[0195] In one possible embodiment, the method steps shown in S1055 can be implemented by Sc1 to Sc6, which are described in detail below.

[0196] Sc1: Obtain the preset lower limit and preset upper limit of the target joint to obtain the target joint limit range.

[0197] It should be noted that the target joint can be any joint that needs to be constrained by bending angle, such as the proximal interphalangeal joint or distal interphalangeal joint of the index, middle, ring, or little finger, or other hand joints that need to be constrained by angle.

[0198] Different target joints can correspond to different preset lower and upper angle limits. The preset lower angle limit represents the minimum allowable bending angle of the target joint, and the preset upper angle limit represents the maximum allowable bending angle of the target joint. Based on the preset lower and upper angle limits corresponding to the target joint, the target joint's limiting range can be obtained.

[0199] In one example, the lower and upper limits of a preset angle can be retrieved from preset joint range of motion data based on the joint identifier of the target joint. The preset joint range of motion data can pre-store the correspondence between joint identifiers and angle ranges.

[0200] Sc2. Compare the candidate parameters of the joint bending angle corresponding to the target joint with the limit range of the target joint to obtain the angle over-limit judgment result.

[0201] The candidate parameter for the joint bending angle can be an initial angle parameter obtained based on planarized joint data and joint bending observation information. This candidate parameter can be compared with the preset lower and upper angle limits within the target joint's limiting range to determine whether it falls within that range.

[0202] The angle out-of-bounds determination result can include three situations: the candidate parameter of the joint bending angle is less than the preset lower limit, the candidate parameter of the joint bending angle is greater than the preset upper limit, and the candidate parameter of the joint bending angle is within the target joint limit range. Based on this determination result, the subsequent processing method for the candidate parameter of the joint bending angle can be determined.

[0203] Sc3. If the angle overshoot determination result indicates that the candidate parameter of the joint bending angle is less than the preset angle lower limit, the preset angle lower limit shall be determined as the target joint bending angle parameter.

[0204] It should be noted that when the candidate parameter of the joint flexion angle is less than the preset lower limit of the target joint, it indicates that the candidate angle obtained by visual observation is already below the allowable range of motion of the target joint. In this case, the candidate parameter of the joint flexion angle can be trimmed to the preset lower limit, and the preset lower limit can be determined as the target joint flexion angle parameter. This process can avoid outputting angle results that are below the reasonable range of motion of the target joint.

[0205] Sc4. If the angle overshoot determination result indicates that the candidate parameter of the joint bending angle is greater than the preset angle upper limit, the preset angle upper limit shall be determined as the target joint bending angle parameter.

[0206] It should be noted that when the candidate parameter of the joint bending angle is greater than the preset upper limit of the target joint, it indicates that the candidate angle obtained by visual observation has exceeded the maximum allowable range of motion of the target joint. In this case, the candidate parameter of the joint bending angle can be trimmed to the preset upper limit of the angle, and the preset upper limit of the angle can be determined as the target joint bending angle parameter. This process can avoid outputting excessive bending results that exceed the reasonable range of motion of the target joint.

[0207] Sc5. If the angle overshoot determination result indicates that the candidate parameter of the joint bending angle is within the target joint limit range, the candidate parameter of the joint bending angle shall be determined as the target joint bending angle parameter.

[0208] It should be noted that when a candidate parameter for the joint flexion angle is within the target joint's limiting range, it indicates that the candidate angle conforms to the preset range of motion corresponding to the target joint. In this case, the candidate parameter for the joint flexion angle can be directly determined as the target joint flexion angle parameter without boundary clipping. Thus, while preserving the effective posture information observed visually, it can be ensured that the output joint flexion angle parameter meets the joint range of motion constraints.

[0209] Sc6. Based on the target joint bending angle parameters corresponding to each target joint, obtain the joint bending angle parameters in the constrained posture parameters.

[0210] In one possible embodiment, the aforementioned limiting process can be performed on multiple target joints respectively to obtain the target joint bending angle parameters corresponding to each target joint. Subsequently, the bending angle parameters of each target joint can be organized according to the joint identifier of the target joint or a preset joint order to obtain the joint bending angle parameters in the constrained posture parameters. The joint bending angle parameters can be used together with the length-constrained bone direction parameters for subsequent generation of low-dimensional hand posture vectors.

[0211] The embodiments of this application can perform boundary processing on candidate parameters of joint bending angle based on joint motion range constraints, so that candidate parameters below the preset lower limit are clipped to the lower limit, and candidate parameters above the preset upper limit are clipped to the upper limit, while retaining candidate parameters within the target joint limit range, thereby ensuring that the obtained joint bending angle parameters are within the preset motion range, reducing abnormal results such as reverse joint, excessive bending or unnatural posture, and improving the physical reliability of constrained posture parameters.

[0212] In one possible embodiment, the method steps shown in S106 can be implemented by S1061 to S1065, which are described in detail below.

[0213] S1061. Extract multiple length-constrained bone orientation parameters from the constrained posture parameters to obtain a bone orientation parameter set.

[0214] It should be noted that the constrained posture parameters may include bone orientation parameters after rigid bone length constraints. Bone orientation parameters characterize the spatial orientation of the corresponding bone segment in the local hand coordinate system. Multiple bone orientation parameters can be extracted from the constrained posture parameters based on preset bone segment identifiers and organized according to the corresponding bone segment order to obtain a set of bone orientation parameters.

[0215] In one example, the skeletal orientation parameter set may include orientation parameters of multiple bone segments corresponding to the thumb, as well as orientation parameters of bone segments in the index, middle, ring, and little fingers used to characterize finger posture. By extracting the skeletal orientation parameter set, spatial orientation parameters can be provided for subsequent generation of low-dimensional hand posture vectors.

[0216] S1062. Convert each bone direction parameter in the bone direction parameter group into azimuth and pitch parameters to obtain the angled bone direction parameter group.

[0217] Among them, the azimuth and pitch parameters are used to represent the skeletal orientation parameters in angular form. It should be noted that a three-dimensional skeletal orientation parameter can be expressed using azimuth and pitch parameters, thereby reducing the dimensionality of parameter expression while preserving the spatial orientation information of the skeletal skeleton.

[0218] Specifically, for each bone orientation parameter in the bone orientation parameter group, its azimuth and pitch angles in the local coordinate system of the hand can be calculated, and the calculated azimuth and pitch angle parameters can be used as the angular representation of the corresponding bone orientation parameter.

[0219] Through the above processing, an angled bone orientation parameter set can be obtained, so that the bone orientation information can be represented in a parameter form that is more suitable for splicing and downstream processing.

[0220] S1063. Extract multiple joint bending angle parameters from the constrained posture parameters to obtain a bending angle parameter set.

[0221] It should be noted that the constrained posture parameters may also include joint flexion angle parameters after joint range of motion constraints. Joint flexion angle parameters characterize the degree of flexion of the corresponding joint. Multiple joint flexion angle parameters can be extracted from the constrained posture parameters based on preset joint identifiers and organized according to the corresponding joint sequence to obtain a set of flexion angle parameters.

[0222] In one example, the bending angle parameter set may include bending angle parameters corresponding to one or more interphalangeal joints of the index, middle, ring, and little fingers. Since these joint bending angle parameters have been constrained by joint range of motion, they can, to some extent, avoid abnormal angle results exceeding the preset range of motion.

[0223] S1064. According to the preset splicing order, splice the angled bone direction parameter group and the bending angle parameter group to obtain a low-dimensional hand posture vector.

[0224] The preset splicing order is used to determine the arrangement of each bone orientation parameter and each joint bending angle parameter in the low-dimensional hand posture vector.

[0225] In one embodiment, the angled bone direction parameter set can be arranged first according to a preset bone segment order, and then the bending angle parameter set can be arranged according to a preset joint order. The two can then be concatenated sequentially to obtain a low-dimensional hand pose vector. Alternatively, other fixed concatenation orders can be set according to the input format requirements of downstream application modules.

[0226] In one example, a low-dimensional hand pose vector may include multiple azimuth parameters, multiple pitch parameters, and multiple joint flexion parameters. Because this low-dimensional hand pose vector is generated from constrained pose parameters, it can express the orientation of the hand bones and the joint flexion state in a lower dimension.

[0227] In one specific example, the length-constrained bone orientation parameters can include multiple unit orientation vectors. Eight unit orientation vectors can be calculated from the constrained attitude parameters and represented as v_1 to v_8. Each unit orientation vector can be represented in spherical coordinates as an azimuth parameter θ and a pitch parameter φ, thus the eight unit orientation vectors can correspond to sixteen angularized bone orientation parameters.

[0228] In one example, using the hand geometry point identifiers shown in Table 1, the eight unit direction vectors can correspond to the four bone directions of the thumb and the proximal phalanges of the index, middle, ring, and little fingers, respectively. Specifically, the four bone directions of the thumb can include the bone segment directions formed by points A to B, B to C, C to D, and D to E; the proximal phalanges of the index, middle, ring, and little fingers can respectively include the bone segment directions formed by points F to G, J to K, N to O, and R to S.

[0229] It should be noted that the palm plane normal vector and the palm unfolding angle formed by points F, A, and R can be normalized during the construction and transformation of the local hand coordinate system, or treated as fixed reference information in the pre-defined hand structure. Therefore, they do not need to be treated as independent degrees of freedom in the low-dimensional hand pose vector. As a result, the low-dimensional hand pose vector can more effectively express the direction of finger bones and the bending state of joints, without redundantly representing the overall hand pose or fixed palm structure information.

[0230] Furthermore, eight joint bending angle parameters can be collected from the constrained posture parameters and denoted as α_1 to α_8 respectively. Each joint bending angle parameter can contribute one dimension, so the eight joint bending angle parameters can form an eight-dimensional bending angle parameter set.

[0231] In one example, the eight joint flexion angle parameters can correspond to the flexion angles of the proximal interphalangeal joint of the index finger, the distal interphalangeal joint of the index finger, the proximal interphalangeal joint of the middle finger, the distal interphalangeal joint of the middle finger, the proximal interphalangeal joint of the ring finger, the distal interphalangeal joint of the ring finger, the proximal interphalangeal joint of the little finger, and the distal interphalangeal joint of the little finger, respectively. Based on the hand geometry point markings shown in Table 1, the above eight joint flexion angle parameters can correspond to ∠G, ∠H, ∠K, ∠L, ∠O, ∠P, ∠S, and ∠T, respectively.

[0232] In one example, sixteen angled bone direction parameters and eight joint bending angle parameters can be spliced ​​together in a preset splicing order to form a 24-dimensional low-dimensional hand pose vector Θ. The low-dimensional hand pose vector Θ can be represented as: Θ=[θ_1, φ_1, ..., θ_8, φ_8, α_1, ..., α_8] T ∈ .

[0233] Where θ_1 to θ_8 represent the azimuth parameters corresponding to the eight unit direction vectors, φ_1 to φ_8 represent the pitch angle parameters corresponding to the eight unit direction vectors, and α_1 to α_8 represent the flexion angle parameters of the eight joints. It represents a 24-dimensional real vector space.

[0234] By using the above stitching method, high-dimensional hand visual observation data formed by the three-dimensional coordinates of multiple hand key points can be converted into low-dimensional hand pose vectors with lower dimensions and clear physical meaning.

[0235] In one possible implementation, if the visual perception model outputs the 3D coordinates of N hand key points, the original dimension of the hand visual observation data is 3N. The low-dimensional hand pose vector can consist of K unit direction vector parameters and M planar angle parameters. Each unit direction vector can be represented by two parameters: azimuth and pitch angle. Each planar angle parameter is represented by an angle value. Therefore, the dimension of the low-dimensional hand pose vector can be 2K+M. By making 2K+M less than 3N, the dimensionality of the original hand visual observation data can be reduced while preserving effective pose information such as hand bone orientation and joint flexion, thus improving the data processing efficiency of subsequent downstream applications. For the example where the visual perception model outputs twenty-one hand key points, N can be 21, and the original observation data is sixty-three dimensions. When K is 8 and M is 8, the low-dimensional hand pose vector is twenty-four dimensions, thereby realizing the conversion from high-dimensional key point coordinates to low-dimensional pose parameters.

[0236] S1065. Output the low-dimensional hand pose vector to the target downstream application module, which includes at least one of a gesture recognition model, a virtual hand model, or a robot controller.

[0237] It should be noted that the target downstream application module can be a functional module that requires the hand pose estimation results. The low-dimensional hand pose vector can be used as input data for the target downstream application module for subsequent gesture classification, virtual hand pose driving, robot dexterity hand control, or real-time interaction processing.

[0238] In one example, when the target downstream application module is a gesture recognition model, the low-dimensional hand pose vector can be used to represent the current gesture state; when the target downstream application module is a virtual hand model, the low-dimensional hand pose vector can be used to drive the joint pose changes of the virtual hand model; when the target downstream application module is a robot controller, the low-dimensional hand pose vector can be used to generate the control input for the robot's dexterous hand or manipulator.

[0239] This application's embodiments can extract skeletal orientation parameters and joint bending angle parameters from constrained posture parameters, and then convert the skeletal orientation parameters into angular representations and sequentially concatenate them with the joint bending angle parameters to obtain a low-dimensional hand posture vector. Therefore, while preserving effective hand posture information, the dimensionality of the data representation can be reduced, making the output results more suitable for downstream applications such as gesture recognition, virtual hand actuation, and robot control, thus improving the real-time application adaptability of the posture estimation results.

[0240] Through the above embodiments, the hand visual pose estimation method and system provided in this application have at least the following effects: First, by introducing rigid bone length constraints, finger reference bending plane constraints, geometric topological relationship constraints, and joint motion range constraints during the pose estimation process, the output low-dimensional hand pose vector can conform to the preset hand physical model, thereby reducing abnormal results that do not conform to the hand structure and movement laws, such as anti-joint, bone stretching, finger penetration, or joint angle exceeding limits; Second, by using deterministic forward mapping to complete the pose parameter generation, the pose solution is mainly based on vector operations, coordinate transformation, and angle calculation, without relying on repeated iterative optimization and large-scale matrix inversion, therefore... First, it can reduce computational complexity and improve the real-time processing capability of edge devices. Second, by converting high-dimensional visual observation data of multiple hand key points into low-dimensional hand pose vectors including skeletal orientation parameters and joint bending angle parameters, it can retain effective hand pose information while reducing data redundancy, making the output results more suitable for downstream applications such as gesture recognition, virtual hand actuation, robot teleoperation, and real-time interactive control. Third, by embedding physical constraints synchronously during the dimensionality reduction process, it can normalize local jitter, instantaneous errors, and abnormal points in the output of the visual perception model, making the pose estimation process have a similar effect to physical filtering, thereby improving the stability and robustness of the low-dimensional hand pose vector.

[0241] Figure 3 This is a schematic diagram illustrating the topological and physical constraints of key hand points as provided in an embodiment of this application. Figure 3 As shown, the left side displays the 3D input of 21 hand keypoints output by the visual perception model. Each hand keypoint can include three coordinate components: horizontal coordinate, vertical coordinate, and depth coordinate. Therefore, the 21 hand keypoints can form 60 three-dimensional hand visual observation data. In other words, the hand visual observation data can be understood as a high-dimensional observation vector composed of the x, y, and z coordinate values ​​of the 21 keypoints, which reflects the spatial observation position of each key part of the hand in the current frame.

[0242] Furthermore, such as Figure 3As shown in the middle section, based on the preset key point topological mapping relationship, the twenty-one hand key points output by the visual perception model can be mapped to the corresponding hand geometric point identifiers from point A to point U. Specifically, point A corresponds to the wrist key point; points B, C, D, and E correspond to the thumb's carpometacarpophalangeal joint, thumb metacarpophalangeal joint, thumb interphalangeal joint, and thumb tip, respectively; points F, G, H, and I correspond to the index finger's metacarpophalangeal joint, index finger's proximal interphalangeal joint, index finger's distal interphalangeal joint, and index finger tip, respectively; points J, K, L, and M correspond to the middle finger's metacarpophalangeal joint, middle finger's proximal interphalangeal joint, middle finger's distal interphalangeal joint, and middle finger tip, respectively; points N, O, P, and Q correspond to the ring finger's metacarpophalangeal joint, ring finger's proximal interphalangeal joint, ring finger's distal interphalangeal joint, and ring finger tip, respectively; and points R, S, T, and U correspond to the little finger's metacarpophalangeal joint, little finger's proximal interphalangeal joint, little finger's distal interphalangeal joint, and little finger tip, respectively. Through the above mapping relationship, the discrete keypoint coordinates output by the visual perception model can be converted into a topological model of hand keypoints with clear hand structure meaning.

[0243] like Figure 3 As shown, adjacent hand geometric points can form corresponding skeletal segments. Specifically, points A to B, B to C, C to D, and D to E can form the thumb skeletal direction chain, corresponding to skeletal segments AB, BC, CD, and DE, respectively; points F to G, J to K, N to O, and R to S can form the proximal phalangeal bone directions of the index, middle, ring, and little fingers, respectively, corresponding to skeletal segments FG, JK, NO, and RS. In one embodiment, the length of the above-mentioned skeletal segments can be configured as a preset skeletal length, so the skeletal length itself can be treated as a fixed structural parameter rather than as an independent degree of freedom in the low-dimensional hand pose vector.

[0244] exist Figure 3 In the illustrated embodiment, the palm plane can be determined based on points A, F, J, N, and R. The palm plane serves as a fundamental reference for constructing the local coordinate system of the hand and determining the calculation plane for finger flexion. Taking the index finger as an example, points F, G, H, and I determine the index finger flexion plane, which can be perpendicular or approximately perpendicular to the palm plane. By constraining the relevant key points of the index finger within this flexion plane, the flexion angles of the index finger joints can be calculated within this plane, such as the joint flexion angle θ_G corresponding to point G and the joint flexion angle θ_H corresponding to point H. Thus, the flexion state of the index finger can be primarily described by the planar joint angles without introducing additional coordinate degrees of freedom corresponding to the lateral displacement of the finger. Similarly, the middle, ring, and little fingers can also have their respective planar joint angles determined based on their corresponding flexion calculation planes.

[0245] In one embodiment, the hand posture representation based on fixed bone length and variable joint posture may include length parameters, unit direction vector parameters, and planar angle parameters. The length parameter corresponds to the preset bone length of each bone segment. Since this type of bone length can be predetermined based on an average human hand model, user calibration data, or the structural parameters of the target robot hand, it does not contribute to the degree of posture freedom. The unit direction vector parameter is used to represent the variable bone orientation. Each three-dimensional unit direction vector has a fixed length of 1 and can be represented by two parameters: azimuth and pitch. Therefore, each unit direction vector contributes two dimensions. The planar angle parameter is used to represent the joint bending angle obtained in the corresponding bending calculation plane. Each planar angle parameter contributes one dimension.

[0246] In a specific example, Figure 3 The variable pose parameters can include eight unit direction vectors and eight planar joint angles. The eight unit direction vectors can correspond to the thumb bone direction chains AB, BC, CD, DE, and the proximal phalangeal directions FG, JK, NO, RS of the index, middle, ring, and little fingers, respectively. Each unit direction vector can be converted into an azimuth parameter θ and a pitch angle parameter φ, thus forming sixteen angularized bone direction parameters. The eight planar joint angles can correspond to the bending angles of the proximal and distal interphalangeal joints of the index, middle, ring, and little fingers, respectively, based on... Figure 3 The geometric points of the hand can be represented as the joint bending angles at points G, H, K, L, O, P, S, and T.

[0247] It should be noted that the palm plane normal vector can be eliminated as a coordinate reference during the construction of the local hand coordinate system and coordinate normalization; the palm unfolding angle formed by points F, A, and R can be processed as reference information for fixing the palm structure. Therefore, the direction parameters corresponding to the palm plane normal vector and the plane angle parameters corresponding to the palm unfolding angle can be excluded as independent degrees of freedom in the low-dimensional hand posture vector. Based on this, the embodiments of this application ultimately retain sixteen angled skeletal direction parameters corresponding to eight unit direction vectors, and eight joint bending angle parameters corresponding to eight plane joint angles, thereby forming a twenty-four-dimensional low-dimensional hand posture vector.

[0248] Furthermore, in a more general embodiment, if the visual perception model outputs the 3D coordinates of N hand key points, then the dimension of the original hand visual observation data is 3N; if the low-dimensional hand pose vector consists of K unit direction vector parameters and M planar angle parameters, since each unit direction vector can be represented by two angle parameters and each planar angle parameter can be represented by one angle value, the dimension of the low-dimensional hand pose vector can be represented as 2K+M. By making 2K+M less than 3N, the data representation dimension can be reduced while retaining effective hand pose information. Figure 3 The example shown has 21 hand key points, with N being 21. The original hand visual observation data is 63-dimensional. When K is 8 and M is 8, the low-dimensional hand pose vector is 24-dimensional, thus realizing the conversion from high-dimensional key point coordinates to low-dimensional physically reliable pose parameters.

[0249] pass Figure 3 The topological and physical constraint relationship of the hand key points shown in this application embodiment can convert the high-dimensional key point coordinates output by the visual perception model into a low-dimensional hand pose vector composed of bone direction parameters and joint bending angle parameters. In the conversion process, physical constraints such as fixed bone length, palm plane, finger bending plane and joint range of motion are introduced, thereby improving the stability, physical reliability and real-time application adaptability of pose expression.

[0250] In one specific application embodiment, the implementation process of the hand visual pose estimation method provided in this application embodiment can be described by taking the implementation of real-time hand pose estimation on an embedded robot controller as an example.

[0251] First, during the system deployment phase, the algorithm program corresponding to the hand visual pose estimation method can be implemented in C++ and compiled and optimized for the target hardware platform. The target hardware platform can include an ARM Cortex-A53 processor or other embedded processors capable of deploying vision processing and pose calculation programs. ARM stands for Advanced RISC Machines; C++ is a general-purpose programming language. The compiled and optimized algorithm program can be integrated into the robot control software stack, enabling the robot controller to perform hand visual observation data processing, pose parameter calculation, and low-dimensional hand pose vector output on the end-device.

[0252] Secondly, during the data input phase, the robot can be equipped with an RGB camera to capture images of the user's hand. In one example, the RGB camera can capture images at a frame rate of 30 FPS. The captured hand images can be input into a lightweight hand keypoint detection model, which can output the 3D coordinates of twenty-one hand keypoints. The 3D coordinates can include the horizontal coordinates, vertical coordinates, and estimated depth coordinates in the image plane.

[0253] In one possible embodiment, the 3D coordinates output by the visual perception model can be back-projected onto the camera coordinate system based on pre-calibrated camera intrinsic parameters, resulting in a set of 3D coordinates in the camera coordinate system. This set of 3D coordinates is then used as the visual observation data for the hand. Thus, the 3D coordinates of 21 key hand points can form 60 3D input vectors, providing input data for subsequent construction of the local hand coordinate system and the execution of deterministic forward mapping.

[0254] In the pose calculation stage, coordinate normalization can be performed first. Specifically, the 3D coordinates p_0 corresponding to the wrist key point can be obtained and used as the local origin of the hand local coordinate system; according to the preset key point topological mapping relationship, the 3D coordinates p_5 corresponding to the index finger metacarpophalangeal joint key point and the 3D coordinates p_17 corresponding to the little finger metacarpophalangeal joint key point can be obtained; then, the coordinate basis of the hand local coordinate system can be determined based on p_0, p_5, and p_17. For example, the coordinate basis of the hand local coordinate system can be determined according to e_z=normalize((p_5-p_0)×(p_17-p_0)), e_x=normalize(p_5-p_0), and e_y=e_z×e_x, where e_z represents the palm normal basis vector, e_x represents the first direction basis vector pointing from the wrist key point to the index finger metacarpophalangeal joint key point, e_y represents the third direction basis vector, normalize represents normalization processing, and "×" represents vector cross product. Subsequently, the three-dimensional coordinates of all hand key points can be transformed to the local coordinate system of the hand to obtain the normalized set of local key point coordinates {p'_i}.

[0255] After coordinate normalization is completed, physical constraint embedding calculations can be performed in parallel based on the local key point coordinate set {p'_i}.

[0256] In a specific example, physical constraint embedding computation may include unit direction vector calculation, planar bending angle calculation, and joint angle limiting processing. The unit direction vector calculation can be illustrated using the thumb wrist-palm segment as an example. Based on the preset keypoint topological mapping relationship shown in Table 1, the thumb wrist-palm segment can correspond to the bone segment formed by points A to B, where point A corresponds to the wrist keypoint p_0, and point B corresponds to the thumb wrist-palm joint keypoint p_1. The candidate bone direction of bone segment AB can be determined based on the coordinates of points A and B in the local hand coordinate system, and then normalized or reconstructed using the preset bone length L_AB corresponding to bone segment AB, resulting in the unit direction vector v_AB corresponding to bone segment AB. Further, the unit direction vector v_AB can be converted to spherical coordinates to obtain the azimuth parameter θ_AB and pitch parameter φ_AB corresponding to bone segment AB. Thus, the spatial direction of the thumb wrist-palm segment can be converted into angularized bone direction parameters used to compose a low-dimensional hand pose vector.

[0257] The calculation of planar bending angles can be illustrated using the proximal interphalangeal joint of the index finger as an example. Based on the preset keypoint topological mapping relationship shown in Table 1, point F corresponds to the keypoint of the metacarpophalangeal joint of the index finger, point G corresponds to the keypoint of the proximal interphalangeal joint of the index finger, point H corresponds to the keypoint of the distal interphalangeal joint of the index finger, and point I corresponds to the keypoint of the fingertip of the index finger. Since the plane for calculating index finger bending can be constrained to be perpendicular or approximately perpendicular to the palm plane, the bending motion of points G, H, and I can be constrained within a target reference bending plane that passes through point F and satisfies a preset spatial relationship with the palm plane. Within this target reference bending plane, the candidate parameters for the joint bending angle corresponding to point G, i.e., the candidate parameters for the bending angle corresponding to the proximal interphalangeal joint of the index finger, can be determined based on the angle between vectors FG and GH. Through this process, the bending state of the proximal interphalangeal joint of the index finger can be mainly described by a planar angle, without the need to introduce additional coordinate degrees of freedom to describe lateral offset.

[0258] For joint angle limitation processing, we can continue to use the proximal interphalangeal joint of the index finger as an example. We can query the preset joint motion range data to obtain the preset lower and upper limits of the angle corresponding to point G. For example, if the preset angle range of the joint corresponding to point G is 0° to 120°, and the candidate parameter of the joint bending angle calculated based on the target reference bending plane is 130°, then this candidate parameter can be trimmed to 120°, and 120° can be determined as the target joint bending angle parameter corresponding to point G. This avoids abnormal posture results that exceed the joint motion range due to visual detection errors, occlusion, or depth estimation deviations.

[0259] It should be noted that the bone segments AB, points F, G, H, and I mentioned above are only used to illustrate specific examples of physical constraint embedding calculations. For other bone segments and other joints, the unit direction vector calculation, planar bending angle calculation, and joint angle constraint processing can be performed in the same way to obtain multiple bone direction parameters and multiple joint bending angle parameters, which can then be used to generate low-dimensional hand pose vectors.

[0260] During the output assembly stage, multiple calculated skeletal orientation parameters and multiple joint bending angle parameters can be collected and generated into a low-dimensional hand pose vector according to a preset splicing order. In one example, azimuth and pitch angle parameters corresponding to eight unit orientation vectors can be collected to form sixteen angled skeletal orientation parameters; eight joint bending angle parameters can also be collected to form an eight-dimensional bending angle parameter set; then, the sixteen angled skeletal orientation parameters and the eight joint bending angle parameters are spliced ​​together to obtain a twenty-four-dimensional low-dimensional hand pose vector. This low-dimensional hand pose vector can be used to represent the user's current physically reliable hand pose.

[0261] In the output application phase, the hand visual pose estimation system can send the generated low-dimensional hand pose vector to the dexterous hand inverse kinematics (IK) solver via the robot's internal communication bus. The dexterous hand inverse kinematics solver can calculate the target angles of each joint or motor of the dexterous hand based on the low-dimensional hand pose vector, and send the target angles to the corresponding actuators to achieve real-time reproduction of the user's hand pose.

[0262] In one example, the hand visual pose estimation method can process a single frame of hand visual observation data in milliseconds, for example, less than 5ms on the target processor. Because this method uses deterministic forward mapping to generate pose parameters, and primarily relies on coordinate transformation, vector operations, and angle calculations to achieve pose resolution, without depending on iterative optimization and large-scale matrix inversion, it can reduce the computational load of single-frame pose resolution. Therefore, it is suitable for deployment in edge devices such as robot controllers, extended reality interactive terminals, and mobile terminals.

[0263] Figure 4 This is a schematic diagram of the structure of a hand visual pose estimation system provided in an embodiment of this application. Figure 4 As shown, the hand visual pose estimation system 400 includes a visual observation acquisition module 401, a local coordinate construction module 402, a coordinate transformation module 403, an observation information determination module 404, a physical constraint mapping module 405, and a pose vector generation module 406.

[0264] It should be noted that, Figure 4Each module shown can be implemented through software programs, hardware circuits, or a combination of software and hardware. Each module can be integrated into the same processing device or deployed on different processing devices and work together through communication connections.

[0265] The visual observation acquisition module 401 is used to acquire hand visual observation data, which includes the three-dimensional coordinates of multiple key points of the hand output by the visual perception model.

[0266] The visual observation and acquisition module 401 can communicate with a camera, a visual perception model, or a data interface for outputting hand keypoint coordinates to receive hand keypoint coordinate data corresponding to the current frame or consecutive image frames. Multiple hand keypoints can include wrist keypoints, metacarpophalangeal joint keypoints, interphalangeal joint keypoints, and fingertip keypoints, etc., which can characterize the hand structure.

[0267] The local coordinate construction module 402 is used to construct a local coordinate system for the hand based on the wrist key point and the palm reference key point among multiple hand key points.

[0268] Among them, the wrist key point can serve as the local origin reference of the hand's local coordinate system, and the palm reference key point can be a key point that reflects the palm's unfolding direction or the palm's planar direction. The local coordinate construction module 402 can determine the spatial orientation of the palm region based on the wrist key point and the palm reference key point, and generate a hand local coordinate system to describe the changes in the hand's own posture.

[0269] The coordinate transformation module 403 is used to transform the visual observation data of the hand to the local coordinate system of the hand to obtain local key point data of the hand.

[0270] Specifically, the coordinate transformation module 403 can perform translation, projection, or coordinate transformation on the three-dimensional coordinates of multiple hand key points based on the local hand coordinate system determined by the local coordinate construction module 402, so that the coordinate representation of each hand key point is converted from the coordinate representation under the original observation coordinate system to the coordinate representation under the local hand coordinate system. The resulting local hand key point data can include the coordinates of each hand key point in the local hand coordinate system and the corresponding key point identifier.

[0271] The observation information determination module 404 is used to determine the observation information of the palm plane and the corresponding bone direction and joint bending of each finger based on the local key point data of the hand.

[0272] Specifically, the observation information determination module 404 can organize local hand key point data into a hand key point topology model based on a preset key point topology mapping relationship, and determine the connection relationships between relevant key points in the palm region, each bone segment, and each joint based on this hand key point topology model. Furthermore, the observation information determination module 404 can determine bone orientation observation information based on the spatial direction between adjacent joint points, and determine joint bending observation information based on the angular relationship between adjacent bone segments in the same finger.

[0273] The physical constraint mapping module 405 is used to perform deterministic forward mapping on the skeletal orientation observation information and joint bending observation information based on the preset hand physical constraint model to obtain constrained posture parameters. The preset hand physical constraint model includes rigid bone length constraints, finger reference bending plane constraints and joint motion range constraints. The finger reference bending plane constraints are used to define the bending calculation plane of the corresponding finger according to the palm plane.

[0274] Specifically, the physical constraint mapping module 405 can regularize the bone direction observation information according to the rigid bone length constraint, perform planarization processing on the key points required for joint bending calculation according to the finger reference bending plane constraint, and perform boundary processing on the candidate parameters of joint bending angle according to the joint motion range constraint, thereby obtaining constrained posture parameters that conform to the hand bone structure and joint motion law.

[0275] The posture vector generation module 406 is used to generate and output a low-dimensional hand posture vector based on constrained posture parameters. The low-dimensional hand posture vector includes skeletal orientation parameters and joint bending angle parameters.

[0276] The posture vector generation module 406 can extract skeletal orientation parameters and joint bending angle parameters from the constrained posture parameters, and combine them according to a preset splicing order to obtain a low-dimensional hand posture vector representing the current hand posture. This low-dimensional hand posture vector can be output to downstream application modules such as gesture recognition models, virtual hand models, robot controllers, extended reality rendering engines, or dexterous hand inverse kinematics solvers for gesture recognition, virtual hand actuation, robot teleoperation, or real-time interactive control.

[0277] In one possible embodiment, the visual observation acquisition module 401 can correspond to the visual perception interface module for receiving the hand keypoint coordinate stream output by the visual perception model; the local coordinate construction module 402 and the coordinate transformation module 403 can jointly implement coordinate transformation and normalization processing; the physical constraint mapping module 405 can incorporate prior constraint information such as hand bone length, finger reference bending plane, and joint range of motion, and is used to perform the calculation of constrained posture parameters; the posture vector generation module 406 can be used to assemble intermediate calculation results into a unified low-dimensional hand posture vector. The above system embodiment and the aforementioned method embodiment belong to the same technical concept, and the specific implementation of each module can refer to the corresponding steps in the aforementioned method embodiment, which will not be repeated here.

[0278] Through the aforementioned hand visual pose estimation system, this embodiment of the application can obtain the three-dimensional coordinates of key hand points using the visual observation acquisition module 401, reduce the influence of overall hand position and viewpoint changes on pose expression using the local coordinate construction module 402 and coordinate transformation module 403, convert local hand key point data into intermediate observation information related to bone direction and joint bending using the observation information determination module 404, embed hand physical constraints during pose parameter generation using the physical constraint mapping module 405, and finally output a low-dimensional hand pose vector by the pose vector generation module 406. This improves the stability, physical reliability, and real-time application adaptability of the output pose results.

[0279] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes a memory 501 and a processor 502.

[0280] The memory 501 can be a separate physical unit, connected to the processor 502 via a bus 503. Alternatively, the memory 501 and processor 502 can be integrated and implemented in hardware. The memory 501 stores program instructions, which the processor 502 calls to execute the operations performed by the hand visual pose estimation system in any of the above method embodiments.

[0281] Optionally, when some or all of the methods in the above embodiments are implemented by software, the electronic device 500 may also include only the processor 502. A memory 501 for storing programs is located outside the electronic device 500, and the processor 502 is connected to the memory via circuits / wires to read and execute the programs stored in the memory. The processor 502 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor 502 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0282] The memory 501 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0283] For example, this application provides a chip including: an interface circuit and a logic circuit. The interface circuit is used to receive signals from other chips outside the chip and transmit them to the logic circuit, or to send signals from the logic circuit to other chips outside the chip. The logic circuit is used to perform the operations performed by the hand visual pose estimation system in the above method embodiments.

[0284] For example, this application provides a computer-readable storage medium storing computer program instructions thereon, which are executed by the processor of an electronic device to cause the electronic device to perform the operations performed by the hand visual pose estimation system in the above method embodiments.

[0285] For example, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the operations performed by the hand visual pose estimation system in the above method embodiments.

[0286] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for estimating hand visual pose, characterized in that, The method includes: Acquire hand visual observation data, which includes the three-dimensional coordinates of multiple key hand points output by the visual perception model; Based on the wrist key point and palm reference key point among the multiple hand key points, a local coordinate system for the hand is constructed; The visual observation data of the hand is converted to the local coordinate system of the hand to obtain local key point data of the hand; Based on the aforementioned local hand key point data, determine the palm plane and the bone direction observation information and joint bending observation information corresponding to each finger; Based on a preset hand physical constraint model, a deterministic forward mapping is performed on the bone direction observation information and the joint bending observation information to obtain constrained posture parameters; wherein, the preset hand physical constraint model includes rigid bone length constraints, finger reference bending plane constraints and joint range of motion constraints, and the finger reference bending plane constraints are used to define the bending calculation plane of the corresponding finger according to the palm plane; Based on the constrained posture parameters, a low-dimensional hand posture vector is generated and output, which includes skeletal orientation parameters and joint bending angle parameters.

2. The method according to claim 1, characterized in that, The construction of a local hand coordinate system based on the wrist key point and palm reference key point among the plurality of hand key points includes: From the plurality of hand key points, determine the wrist key point, the first palm reference key point, and the second palm reference key point to obtain a coordinate system reference point set; Based on the wrist key points and the first palm reference key points, determine the first direction basis vector; Based on the wrist key points, the first palm reference key points, and the second palm reference key points, the palm normal base vector is determined; Based on the first direction basis vector and the palm normal basis vector, an orthogonal coordinate basis is generated to obtain the local coordinate system of the hand.

3. The method according to claim 2, characterized in that, The step of converting the visual observation data of the hand to the local coordinate system of the hand to obtain local key point data of the hand includes: Using the wrist key point as the local origin, the three-dimensional coordinates of the multiple hand key points are translated to obtain the translated key point data; Based on the orthogonal coordinate base, coordinate projection is performed on the translated key point data to obtain a set of local coordinate key points; The local coordinate keypoint set is sorted according to the keypoint identifiers of the multiple hand keypoints to obtain the local hand keypoint data.

4. The method according to claim 1, characterized in that, The step of determining the palm plane and the corresponding skeletal direction observation information and joint bending observation information of each finger based on the local hand key point data includes: Based on the preset key point topology mapping relationship, each hand key point in the local hand key point data is configured with a corresponding hand geometric point identifier to obtain a hand key point topology model. Based on the aforementioned hand key point topology model, determine the set of bone segments formed by adjacent joint points; Based on the endpoint coordinates of each bone segment in the bone segment set, determine the bone orientation observation information corresponding to each bone segment; Based on the angular relationship between adjacent bone segments in the same finger, determine the joint bending observation information for each finger.

5. The method according to claim 4, characterized in that, Before performing a deterministic forward mapping on the skeletal orientation observation information and the joint flexion observation information based on a preset hand physical constraint model to obtain constrained posture parameters, the method further includes: The rigid bone length constraint is determined based on the preset bone length corresponding to each bone segment in the bone segment set; Based on the palm plane and the corresponding metacarpophalangeal joints of each finger, determine the reference bending plane corresponding to each finger; The finger reference bending plane constraint is determined based on the reference bending plane; The joint motion range constraint is determined based on the preset lower limit and preset upper limit of the angle corresponding to each joint; The rigid bone length constraint, the finger reference bending plane constraint, and the joint range of motion constraint are associated and configured to obtain the preset hand physical constraint model.

6. The method according to claim 1, characterized in that, The method, based on a preset hand physical constraint model, performs a deterministic forward mapping on the skeletal orientation observation information and the joint bending observation information to obtain constrained posture parameters, including: Based on the bone orientation observation information, the candidate bone orientation corresponding to each bone segment is determined; Based on the rigid bone length constraint, the candidate bone direction is normalized, and the end point position of the bone segment is reconstructed according to the corresponding preset bone length to obtain the bone direction parameters constrained by length. Based on the constraints of the finger reference bending plane, the key joint points of the corresponding fingers are planarized to obtain planarized joint data. Based on the planarized joint data and the joint bending observation information, candidate parameters for the joint bending angle are determined. Based on the joint range of motion constraints, the candidate parameters of the joint bending angle are subjected to boundary processing to obtain the constrained posture parameters.

7. The method according to claim 6, characterized in that, The step of performing planarization processing on the key joint points of the corresponding finger based on the finger reference bending plane constraint to obtain planarized joint data includes: Based on the palm plane, determine the palm plane normal vector to obtain the plane reference direction; Based on the metacarpophalangeal joints of the target finger and the plane reference direction, determine the target reference bending plane corresponding to the target finger; The interphalangeal joints and fingertip key points of the target finger are projected onto the target reference bending plane to obtain planarized key point data of the target finger; The planarized key point data of each finger are summarized to obtain the planarized joint data.

8. The method according to claim 6, characterized in that, The process of performing boundary processing on the candidate parameters of the joint bending angle based on the joint range of motion constraints to obtain the constrained posture parameters includes: Obtain the preset lower limit and preset upper limit of the target joint to get the target joint limit range; The candidate parameters of the joint bending angle corresponding to the target joint are compared with the limit range of the target joint to obtain the angle over-limit judgment result. If the angle boundary determination result indicates that the candidate parameter of the joint bending angle is less than the preset angle lower limit, the preset angle lower limit is determined as the target joint bending angle parameter. If the angle boundary determination result indicates that the candidate parameter of the joint bending angle is greater than the preset angle upper limit, the preset angle upper limit is determined as the target joint bending angle parameter; If the angle boundary determination result indicates that the candidate parameter of the joint bending angle is within the target joint limit range, the candidate parameter of the joint bending angle is determined as the target joint bending angle parameter. Based on the target joint bending angle parameters corresponding to each target joint, the joint bending angle parameters in the constrained posture parameters are obtained.

9. The method according to claim 6, characterized in that, The step of generating and outputting a low-dimensional hand pose vector based on the constrained pose parameters includes: Multiple length-constrained bone orientation parameters are extracted from the constrained posture parameters to obtain a bone orientation parameter set; Each bone direction parameter in the bone direction parameter group is converted into azimuth and pitch parameters to obtain the angled bone direction parameter group. Multiple joint bending angle parameters are extracted from the constrained posture parameters to obtain a bending angle parameter set; According to the preset splicing order, the angled bone direction parameter group and the bending angle parameter group are spliced ​​together to obtain the low-dimensional hand posture vector; The low-dimensional hand pose vector is output to a target downstream application module, which includes at least one of a gesture recognition model, a virtual hand model, or a robot controller.

10. A hand visual pose estimation system, characterized in that, The system includes: The visual observation acquisition module is used to acquire hand visual observation data, which includes the three-dimensional coordinates of multiple key points of the hand output by the visual perception model. The local coordinate construction module is used to construct a local coordinate system for the hand based on the wrist key point and the palm reference key point among the multiple hand key points; The coordinate transformation module is used to transform the visual observation data of the hand to the local coordinate system of the hand to obtain local key point data of the hand; The observation information determination module is used to determine the observation information of the palm plane and the bone direction and joint bending of each finger based on the local key point data of the hand. The physical constraint mapping module is used to perform deterministic forward mapping on the bone direction observation information and the joint bending observation information based on a preset hand physical constraint model to obtain constrained posture parameters; wherein, the preset hand physical constraint model includes rigid bone length constraints, finger reference bending plane constraints and joint range of motion constraints, and the finger reference bending plane constraints are used to define the bending calculation plane of the corresponding finger according to the palm plane; The posture vector generation module is used to generate and output a low-dimensional hand posture vector based on the constrained posture parameters. The low-dimensional hand posture vector includes skeletal orientation parameters and joint bending angle parameters.

Images