A hand state labeling method and system based on multi-source geometric representation and decoupled temporal decoding

Abstract

The application relates to a hand state labeling method and system based on multi-source geometric representation and decoupled time sequence decoding. Multi-source observation data of a hand is encoded to obtain multi-source representation, including camera perception image representation, three-dimensional geometric representation and hand shape prior representation; the multi-source representation is spliced with time position coding and then mapped to a unified feature space; source type embedding and view position embedding are introduced to perform cross-source representation fusion through a cross-source attention module to obtain multi-source context representation; the multi-source context representation is input into parallel posture decoders and wrist displacement decoders to respectively predict hand joint rotation parameters and wrist global displacement of each frame; the hand joint rotation parameters, the wrist global displacement and the subject hand shape prior are input into a parameterized hand model to generate a hand state corresponding to the frame. Compared with the prior art, the application can output high-precision hand states based on multi-source vision and geometric observation, and reduce posture misjudgment and grid error caused by individual hand shape differences.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for annotating hand state based on multi-source geometric representation and decoupled temporal decoding. Background Technology

[0002] In embodied intelligence and robotic dexterity tasks, model training requires not only image or video-level observations but also precise hand joint poses, wrist poses, hand dimensions, and hand-object interaction geometry. First-person videos can provide large-scale human manipulation priors, but their core data format is primarily video and action semantics, not directly providing high-precision, uniform-scale 3D hand states. While first-person human videos are valuable for robot imitation learning and vision-language-action models, without stable 3D hand supervision, downstream strategies still struggle to learn fine contact and high-degree-of-freedom finger movements. The MANO parametric hand model provides a compact representation of hand pose, shape, and mesh vertices. However, existing networks often fail to adequately distinguish the roles of image appearance, 3D point clouds, subject hand shape priors, and close-range hand observations in different prediction targets, and lack neural network decoding mechanisms oriented towards pose and displacement specialization.

[0003] Existing technologies mainly include three types of models. The first type is the Transformer or convolutional network model that transforms a single-view image into a hand mesh, using a visual encoder to extract image features and then regressing MANO parameters or mesh vertices. The second type is the video temporal model, which adds recurrent networks, temporal convolutions, or Transformers to frame-by-frame visual features to improve the continuity of hand trajectory. The third type is the multi-view or point cloud-assisted model, which enhances 3D localization capabilities through image features or point cloud features from multiple perspectives. While the above solutions can improve the accuracy of 3D reconstruction in some scenarios, they often simply stitch together or average features from different sources without establishing differentiated decoding paths for hand pose and wrist displacement.

[0004] The existing technology has the following drawbacks: Insufficient feature fusion: Existing models often directly stitch together multi-view images, point clouds and hand shape priors, lacking camera geometric perception and cross-source alignment mechanisms, resulting in spatial semantic mismatch between features from different sources.

[0005] Coupling of pose and displacement prediction: The physical meanings of finger joint pose and global wrist translation are different. If the same decoder is used for unified regression, it is easy to have problems such as accurate local joints but global position drift, or stable global position but incorrect finger details.

[0006] Insufficient utilization of the subject's hand shape scale: Differences in hand shape among different people will affect the position of mesh vertices and joints. If the subject's hand shape prior is not explicitly injected into the network, it is easy to misinterpret the differences in hand shape as changes in posture.

[0007] Insufficient temporal consistency in occluded scenes: Single-frame models are prone to predicting jumps when objects touch or self-occlude, while ordinary temporal smoothing may lose fine-grained motion changes, making it difficult to maintain both stability and agile motion details at the same time.

[0008] The training supervision objective is singular: using only keypoint or mesh vertex regression loss makes it difficult to constrain the consistency between the predicted hand mesh and the observable 3D geometry, and also makes it difficult to simultaneously optimize local pose, global translation and temporal trajectory. Summary of the Invention

[0009] The purpose of this invention is to address the problems of insufficient feature fusion, coupling of pose and displacement prediction, and insufficient temporal consistency in occlusion scenarios in the prior art, and to provide a hand state annotation method and system based on multi-source geometric representation and decoupled temporal decoding.

[0010] The objective of this invention can be achieved through the following technical solutions: As a first aspect of the present invention, a hand state annotation method based on multi-source geometric representation and decoupled temporal decoding is provided, the steps of which include: Acquire multi-source observation data of the hand and encode the multi-source observation data to obtain multi-source representations, including camera-sensed image representations, three-dimensional geometric representations, and hand shape prior representations; Multi-source representations are concatenated with temporal location encodings and mapped to a unified feature space; source type embedding and viewpoint location embedding are introduced, and a cross-source attention module is used to fuse cross-source representations to obtain multi-source context representations; The multi-source context representation is input into the parallel pose decoder and wrist displacement decoder to predict the hand joint rotation parameters and global wrist displacement for each frame, respectively. The predicted hand joint rotation parameters, global wrist displacement, and prior input of the main hand shape are used to parameterize the hand model and generate the hand state of the corresponding frame.

[0011] As a preferred technical solution, the multi-source observation data includes: multi-view hand images, close-up hand images, camera geometric parameters, depth maps or point cloud geometry, and prior knowledge of the subject's hand shape.

[0012] As a preferred technical solution, the attitude decoder and displacement decoder introduce bidirectional temporal attention, taking the multi-source context representations of the current frame and the previous and next frames as input, and outputting a joint representation after bidirectional temporal attention calculation.

[0013] As a preferred technical solution, the attitude decoder is composed of multiple attitude attention decoding blocks stacked together; The pose decoder is based on hand appearance tokens, close-range observation tokens, local point cloud tokens, and hand shape prior tokens. The input includes the average shape and pose obtained based on hand appearance and hand shape priors, as well as hand pose query terms from the current frame for three consecutive frames. The pose attention decoding block first performs self-attention calculation on its input parameters, then uses hand displacement query terms to perform cross-attention calculation on the 3D geometry token and the camera-sensed image token in turn; and finally outputs through a feedforward network. The output of the attitude decoder is passed through an attitude prediction head to predict the hand joint rotation parameters for three consecutive frames starting from the current frame.

[0014] As a preferred technical solution, the wrist displacement decoder is composed of multiple displacement attention decoding blocks stacked together; The displacement decoder is based on 3D point cloud tokens, camera geometry tokens, and joint representations. The input includes learnable hand displacement query terms from three consecutive frames starting from the current frame. The displacement attention decoding block first performs self-attention calculation on its input parameters, then uses hand displacement query terms to perform cross-attention calculation on the 3D geometric tokens; and finally outputs the data through a feedforward network. The output of the displacement decoder is fed through a displacement prediction head to predict the global wrist displacement for three consecutive frames starting from the current frame.

[0015] As a preferred technical solution, the attitude decoder and displacement decoder introduce bidirectional temporal attention, and for continuous frame input, the occluded joints are recovered using the context of the preceding and following frames.

[0016] As a preferred technical solution, the method uses a supervisory signal to construct a combined supervisory loss during the training phase, including one or more of the following: joint rotation loss, wrist displacement loss, hand shape prior loss, 3D keypoint loss, mesh vertex loss, image reprojection loss, point cloud-mesh consistency loss, and temporal smoothing loss.

[0017] As a second aspect of the present invention, a hand state annotation system based on multi-source geometric representation and decoupled temporal decoding is provided, the system executing the hand state annotation method described above, including: The data acquisition module is used to acquire multi-source observation data of the hand. The camera-aware image representation construction module extracts image tokens from multi-view hand images and close-up hand images through an image encoder; encodes camera geometric parameters into camera geometric tokens; and fuses the image tokens and camera geometric tokens to obtain the camera-aware image tokens. The 3D geometric representation construction module, for depth map or point cloud geometric input, backprojects or maps the visible hand area into a set of 3D points, and extracts 3D geometric tokens through the point cloud encoder; The main hand shape prior representation construction module maps the estimated or given hand shape parameters to hand shape prior tokens for different individuals or different hand shapes, or encodes the parameterized hand mesh vertices under the average pose as hand shape prior tokens; The fusion module employs cross-source attention, combining source type embedding, temporal position embedding, and viewpoint position embedding to perform cross-source representation fusion on multi-source representations, thereby obtaining multi-source contextual representations. The decoding module includes a dual-branch temporal decoder that decouples attitude and displacement, predicting the hand joint rotation parameters and global wrist displacement for each frame based on multi-source context representations, respectively. The hand state annotation module parameterizes the hand model by taking the hand joint rotation parameters, the global displacement of the wrist, and the prior input of the main hand shape, generates the hand state of the corresponding frame, and combines the hand states of consecutive frames into a hand state sequence.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention encodes multi-view images, close-range hand observations, 3D geometric information, and prior knowledge of the subject's hand shape into a fusionable multi-source representation, enabling neural networks to no longer rely solely on a single image appearance for hand state regression. Camera geometric encoding and cross-source attention fusion mechanisms enhance spatial alignment between different viewpoints and modalities, maintaining good 3D scale consistency and geometric stability even in complex scenarios such as hand-object contact, self-occlusion, and rapid local movements.

[0019] 2) This invention further designs hand joint pose prediction and wrist global displacement prediction as a decoupled dual-branch temporal decoding structure, enabling local finger pose recovery and overall spatial localization to utilize more suitable information sources. The pose branch focuses on using image appearance, close-range observation, and prior knowledge of the subject's hand shape to recover fine-grained joint changes, while the displacement branch focuses on using point cloud geometry and camera geometry to recover the wrist's spatial position. This reduces the problem of mutual interference between local pose and global displacement, and reduces pose misjudgment and mesh error caused by individual hand shape differences.

[0020] 3) This invention also utilizes bidirectional temporal attention and combined training objectives to jointly constrain joint rotation, wrist displacement, 3D keypoints, hand mesh, image reprojection, point cloud geometric consistency, and temporal smoothness within a continuous frame range, ensuring that the output parameterized hand state remains consistent across parameter space, image space, and 3D geometric space. The resulting hand state sequence is more suitable for downstream tasks such as high-quality data annotation, hand-object interaction understanding, and robot dexterity manipulation strategy learning. Attached Figure Description

[0021] Figure 1 This is a flowchart of the hand state annotation method based on multi-source geometric representation and decoupled temporal decoding of the present invention.

[0022] Figure 2 This is a flowchart of the multi-source input preprocessing and characterization encoding process in this invention.

[0023] Figure 3 This is a flowchart of the cross-source representation fusion and attitude / wrist displacement decoupling decoding process in this invention.

[0024] Figure 4 This is a schematic diagram of the hand state annotation model based on multi-source geometric representation and decoupled temporal decoding of the present invention.

[0025] Figure 5 This is a flowchart of the parameterized hand state generation, combined training, and inference output process in this invention. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0027] Example 1 This invention proposes a hand state annotation method based on multi-source geometric representation and decoupled temporal decoding. This method decouples first-person observation acquisition from high-precision external geometric annotation, allowing the same human operation segment to serve as both policy observation data and a supervised learning method for obtaining MANO hand states. Figure 1 As shown, the flow of this method is as follows.

[0028] Step 1: Construction and preprocessing of multi-source inputs.

[0029] Step 1.1: Receive multi-source observation data related to hand state estimation, including one or more of the following: multi-view hand images, close-up hand images, depth maps or point cloud geometry, camera geometric parameters, and subject hand shape priors.

[0030] Step 1.2: Crop the hand region, normalize the scale, and align the time of the image input to ensure that image observations from different sources can correspond to the hand state within the same time window.

[0031] Step 1.3: Perform coordinate transformation, invalid point filtering, and downsampling on the depth map or point cloud input to obtain the hand point cloud or depth geometric representation for 3D geometric reasoning.

[0032] Step 1.4: Normalize and encode the main hand shape parameters or parameterized hand mesh to form the hand shape prior input used to constrain the differences in hand size and shape.

[0033] Step 2: Multi-source feature encoding, such as Figure 2 As shown, the multi-source representation encoding process is as follows.

[0034] Step 2.1: Input the multi-view hand images and close-up hand images into the image encoder to extract image tokens, obtain hand appearance tokens, which are used to represent finger contours, local textures, contact areas and visible joint information, as well as close-up observation tokens.

[0035] Step 2.2: Encode the camera intrinsic parameters, extrinsic parameters, viewpoint relationships, or ray geometry into camera geometry tokens, and inject them into image tokens to obtain camera-aware image tokens, enabling image features to have viewpoint perception capabilities.

[0036] Step 2.3: Input the depth map or point cloud geometry into the 3D geometry encoder to extract the 3D geometry token, which is used to characterize the surface structure, spatial position, palm orientation and scale information of the hand.

[0037] Step 2.4: Input the subject's hand shape parameters or parameterized hand mesh into the hand shape prior encoder to generate a hand shape prior token, which is used to distinguish individual hand shape differences from actual posture changes.

[0038] Step 3: Cross-source representation fusion, such as Figure 3 As shown, the cross-source fusion and decoupled timing decoding process is as follows.

[0039] Step 3.1: Concatenate or map the camera-sensing image token, camera geometry token, 3D geometry token, hand prior token, and temporal location code to a unified feature space.

[0040] Step 3.2: Adaptively fuse tokens from different sources through the cross-source attention module, enabling the network to select effective information based on the visibility, occlusion status, and geometric reliability of the current frame.

[0041] Step 3.3: Introduce source type embedding and viewpoint position embedding during the fusion process to enable the network to distinguish different feature sources such as images, point clouds, hand priors and close-range observations, thereby reducing cross-source semantic mismatches.

[0042] Step 3.4: Output a unified multi-source context representation as a shared input for subsequent pose decoding and wrist displacement decoding.

[0043] Step 4: Decoupling and decoding of posture and wrist displacement.

[0044] Step 4.1: For continuous frame input, both decoders introduce bidirectional temporal attention, taking the multi-source context representations of the current frame and the frames before and after as input. After bidirectional temporal attention calculation, a joint representation is output. The output shape is the same as the input, but the representation of each frame incorporates features from the frames before and after, enabling the recovery of occluded joints and maintenance of motion continuity by utilizing the context representations output by the frame cross-source attention module.

[0045] Step 4.2: The pose decoder receives features related to local finger movements from the joint representation, focusing on hand appearance, close-range observation, local point cloud, and hand shape priors to predict the hand joint rotation parameters for each frame. The local point cloud is obtained by cropping the complete 3D point cloud using a mask obtained from the segmentation model.

[0046] Step 4.3: The displacement decoder receives features related to global spatial localization from the joint representation, focusing on 3D point cloud, camera geometry, and global context to predict the global wrist displacement in each frame. The global context refers to the features corresponding to several frames preceding and following a given frame. The features of each frame are obtained through the multi-source feature encoder described above and fused using a bidirectional temporal attention module.

[0047] Step 4.4: The attitude decoder and displacement decoder retain independent network parameters and can exchange information through lightweight cross-attention or residual connections, thereby avoiding mutual interference between local attitude prediction and global displacement prediction.

[0048] like Figure 4 As shown, the decoder adopts a Transformer Decoder structure, which is composed of multiple attention decoding blocks stacked together. Cross attention is used within the attention decoding blocks, and information is exchanged between attention decoding blocks through residual connections.

[0049] The input to the pose decoder includes the average shape and pose obtained based on prior knowledge of hand appearance and hand shape. And from the current frame t Learnable hand pose query terms for three consecutive frames (Query token). The attention decoding block in the pose decoder performs self-attention calculation on its input parameters, and then uses the hand displacement query tokens learned during model training to perform cross-attention calculation on the 3D geometry token and the camera-sensed image token in turn; then it outputs through a feedforward network. Finally, the output of the pose decoder predicts the hand joint rotation parameters for three consecutive frames starting from the current frame through a pose prediction head.

[0050] The input to the shift decoder includes data from the current frame. tLearnable hand displacement query terms for three consecutive frames (Query token). The attention decoding block in the displacement decoder performs self-attention calculation on its input parameters, and then uses the hand displacement query terms learned during model training to perform cross-attention calculation on the 3D geometric token; then it outputs through a feedforward network. Finally, the output of the displacement decoder predicts the global wrist displacement for three consecutive frames starting from the current frame through a displacement prediction head.

[0051] Compared to the displacement decoder, the pose decoder performs additional cross-attention operations on image features to extract hand pose features from image information.

[0052] By decoupling the attitude decoder and the displacement decoder, the finger joint attitude recovery and wrist global displacement estimation are optimized respectively, improving the consistency between local details and global scale, and solving the problem of attitude and displacement coupling interference.

[0053] Step 5: Parameterize hand state generation, such as... Figure 5 As shown, the steps are as follows.

[0054] Step 5.1: Input the predicted hand joint rotation parameters, global wrist displacement, and prior knowledge of the main hand shape into the parameterized hand model.

[0055] Step 5.2: Generate the hand state of the corresponding frame by parametrically modeling the hand, including 3D hand key points and hand mesh vertices.

[0056] Step 5.3: Combine the hand states of consecutive frames into a hand state sequence to represent the hand posture changes, spatial motion trajectory and hand geometry during the complete operation.

[0057] Step 6: Combined training and inference output.

[0058] Step 6.1: During the training phase, construct a combined loss function based on the available supervision signals, including one or more of the following: joint rotation loss, wrist displacement loss, hand shape prior loss, 3D keypoint loss, mesh vertex loss, image reprojection loss, point cloud-mesh consistency loss, and temporal smoothing loss.

[0059] Step 6.2: By combining training objectives, the parameter space, image space, and three-dimensional geometric space are simultaneously constrained to ensure that the hand state output by the network has scale consistency, geometric consistency, and temporal continuity.

[0060] Step 6.3: Inference phase. After receiving observations from multiple sources, the network directly outputs parameterized hand states of consecutive frames, including hand joint rotation, global wrist displacement, 3D key points, and hand mesh.

[0061] Step 6.4: The output results can be used for downstream tasks such as high-quality hand state annotation, hand-object interaction understanding, and robot dexterity operation strategy learning.

[0062] Example 2 As another embodiment of the present invention, this embodiment provides a hand state annotation system based on multi-source geometric representation and decoupled temporal decoding. It takes multi-source hand observations as input, including but not limited to multi-view images, first-person or close-up hand images, depth or point cloud geometry, camera parameters, and prior knowledge of the subject's hand shape; the output is parameterized hand state, including hand joint rotation, global wrist displacement, subject's hand shape parameters, and the resulting hand mesh and keypoints.

[0063] A camera-aware image representation construction module is implemented. For hand images from different viewpoints, hand appearance tokens are extracted using a shared or partially shared visual encoder. Camera intrinsic and extrinsic parameters, or ray directions, are encoded into camera geometry tokens, which are then additively fused, stitched together, or cross-attention fused with the image tokens. This design enables the model to distinguish the spatial meaning of the same hand region from different viewpoints, avoiding misinterpreting changes in viewpoint as changes in hand pose.

[0064] A 3D geometric representation construction module. For depth map or point cloud geometric input, the visible hand region is back-projected or mapped into a set of 3D points, and 3D geometric tokens are extracted using a point cloud encoder. These 3D geometric tokens retain 3D coordinates, local neighborhood structure, and optional color information, providing direct constraints for wrist displacement, palm orientation, and absolute scale. This geometric branch can be computed in parallel with the image branch and selectively invoked in subsequent decoding stages via cross-attention.

[0065] The module constructs the primary hand shape prior representation. For different individuals or different hand shapes, it maps the estimated or given hand shape parameters into hand shape prior tokens, or encodes the parameterized hand mesh vertices under the average pose into hand shape prior tokens. These hand shape prior tokens participate in pose decoding, enabling the network to distinguish between hand shape scale differences and actual pose changes when predicting joint rotations, thereby reducing mesh errors caused by differences in palm width and finger length.

[0066] The fusion module employs a multi-source Transformer or equivalent attention structure, inputting camera-perceived image tokens, close-range hand observation tokens, point cloud geometry tokens, and hand shape prior tokens into a unified cross-source attention module. This module allows for setting source type embedding, temporal location embedding, and viewpoint location embedding, enabling the network to explicitly know whether each token originates from an image, point cloud, hand shape prior, or close-range observation, and adaptively select the information source at different prediction stages.

[0067] The decoding module employs a dual-branch temporal decoder that decouples attitude and displacement. The attitude decoder takes hand shape prior tokens, image tokens, close-range observation tokens, and local point cloud tokens as its main inputs, and outputs the hand joint rotation parameters for each frame. The displacement decoder takes point cloud geometry tokens, camera geometry tokens, and global context tokens as its main inputs, and outputs the 3D wrist displacement for each frame. The attitude and displacement decoders can exchange information through lightweight cross-attention or residual connections, but retain independent parameters, thereby avoiding mutual interference between local attitude and global displacement.

[0068] To improve stability during continuous operations, bidirectional temporal attention is introduced into the decoding module. For multi-frame observations within a time window, both the pose decoder and the displacement decoder can access the context of preceding and following frames, enabling occluded frames to recover the finger state with the help of neighboring visible frames, while simultaneously maintaining the subtle changes in rapid finger movements through learnable attention. This temporal modeling is not a simple post-processing smoothing, but rather an end-to-end trainable module embedded within the neural network.

[0069] During the training phase, a combined supervised loss mechanism is employed, including: joint rotation loss, 3D keypoint loss, mesh vertex loss, point cloud-to-mesh geometric consistency loss, hand shape parameter regularization loss, temporal velocity / acceleration smoothing loss, and multi-view reprojection loss. Through these losses, the network simultaneously constrains the parameter space, image space, and 3D geometric space, improving the physical consistency of the output hand state and the quality of data usable for robot learning.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A hand state annotation method based on multi-source geometric representation and decoupled temporal decoding, characterized in that the steps are as follows: include: Acquire multi-source observation data of the hand and encode the multi-source observation data to obtain multi-source representations, including camera-sensed image representations, three-dimensional geometric representations, and hand shape prior representations; Multi-source representations are concatenated with temporal location encodings and mapped to a unified feature space. Source type embedding and viewpoint location embedding are introduced, and cross-source representation fusion is performed through a cross-source attention module to obtain multi-source context representations. The multi-source context representation is input into the parallel pose decoder and wrist displacement decoder to predict the hand joint rotation parameters and global wrist displacement for each frame, respectively. The predicted hand joint rotation parameters, global wrist displacement, and prior input of the main hand shape are used to parameterize the hand model and generate the hand state of the corresponding frame.

2. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 1, characterized in that, The multi-source observation data includes: multi-view hand images, close-up hand images, camera geometric parameters, depth maps or point cloud geometry, and prior knowledge of the subject's hand shape.

3. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 2, characterized in that, The camera-sensing image representation is specifically encoded as follows: Multi-view hand images and close-up hand images are input into an image encoder to extract image tokens, resulting in hand appearance tokens and close-up observation tokens, respectively. Camera geometric parameters are encoded into camera geometric tokens and fused with image tokens to obtain camera-perceived image tokens.

4. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 2, characterized in that, The specific encoding of the three-dimensional geometric representation is as follows: For depth map or point cloud geometry input, the visible hand area is back-projected or mapped into a set of 3D points, and a 3D geometric token is extracted by a point cloud encoder; the 3D geometric token retains 3D coordinates, local neighborhood structure and optional color information.

5. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 1, characterized in that, The specific encoding of the hand shape prior representation is as follows: For different individuals or different hand shapes, the estimated or given hand shape parameters are mapped to hand shape prior tokens, or the parameterized hand mesh vertices under the average pose are encoded as hand shape prior tokens.

6. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 1, characterized in that, The attitude decoder and displacement decoder introduce bidirectional temporal attention, taking the multi-source context representations of the current frame and the previous and next frames as input, and outputting a joint representation after bidirectional temporal attention calculation.

7. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 6, characterized in that, The attitude decoder is composed of multiple attitude attention decoding blocks stacked together; The pose decoder is based on hand appearance tokens, close-range observation tokens, local point cloud tokens, and hand shape prior tokens. The input includes the average shape and pose obtained based on hand appearance and hand shape priors, as well as hand pose query terms from the current frame for three consecutive frames. The pose attention decoding block first performs self-attention calculation on its input parameters, then uses hand displacement query terms to perform cross-attention calculation on the 3D geometry token and the camera-sensed image token in turn; and finally outputs through a feedforward network. The output of the attitude decoder is passed through an attitude prediction head to predict the hand joint rotation parameters for three consecutive frames starting from the current frame.

8. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 6, characterized in that, The wrist displacement decoder is composed of multiple displacement attention decoding blocks stacked together; The displacement decoder is based on 3D point cloud tokens, camera geometry tokens, and joint representations. The input includes learnable hand displacement query terms from three consecutive frames starting from the current frame. The displacement attention decoding block first performs self-attention calculation on its input parameters, then uses hand displacement query terms to perform cross-attention calculation on the 3D geometric tokens; and finally outputs the data through a feedforward network. The output of the displacement decoder is fed through a displacement prediction head to predict the global wrist displacement for three consecutive frames starting from the current frame.

9. The hand state annotation method based on multi-source geometric representation and decoupled temporal decoding according to claim 1, characterized in that, The method employs supervised signals to construct a combined supervised loss during the training phase, including one or more of the following: joint rotation loss, wrist displacement loss, hand shape prior loss, 3D keypoint loss, mesh vertex loss, image reprojection loss, point cloud-mesh consistency loss, and temporal smoothing loss.

10. A hand state annotation system based on multi-source geometric representation and decoupled temporal decoding, characterized in that, The system executes the hand state labeling method as described in any one of claims 1-9, including: The data acquisition module is used to acquire multi-source observation data of the hand. The camera-aware image representation construction module extracts image tokens from multi-view hand images and close-up hand images through an image encoder; encodes camera geometric parameters into camera geometric tokens; and fuses the image tokens and camera geometric tokens to obtain the camera-aware image tokens. The 3D geometric representation construction module, for depth map or point cloud geometric input, backprojects or maps the visible hand area into a set of 3D points, and extracts 3D geometric tokens through the point cloud encoder; The main hand shape prior representation construction module maps the estimated or given hand shape parameters to hand shape prior tokens for different individuals or different hand shapes, or encodes the parameterized hand mesh vertices under the average pose as hand shape prior tokens; The fusion module employs cross-source attention, combining source type embedding, temporal position embedding, and viewpoint position embedding to perform cross-source representation fusion on multi-source representations, thereby obtaining multi-source contextual representations. The decoding module includes a dual-branch temporal decoder that decouples attitude and displacement, predicting the hand joint rotation parameters and global wrist displacement for each frame based on multi-source context representations, respectively. The hand state annotation module parameterizes the hand model by taking the hand joint rotation parameters, the global displacement of the wrist, and the prior input of the main hand shape, generates the hand state of the corresponding frame, and combines the hand states of consecutive frames into a hand state sequence.

Images