A method for three-dimensional hand-object interaction pose estimation based on implicit space constraints

Abstract

A three-dimensional hand-object interaction pose estimation method based on implicit space constraint, through a multi-scale feature extraction network, obtains two-dimensional appearance features of the hand and the object, simultaneously constructs sampling point sets of the hand and the object in an implicit three-dimensional space, and introduces a space constraint relationship between the hand and the object in the feature construction stage through bidirectional coordinate transformation; further, the sampling points in the implicit space are mapped to a two-dimensional feature plane, cross-modal feature sampling is performed on the multi-scale feature map, and point features that fuse two-dimensional appearance information and three-dimensional geometric information are formed; based on the spatial adjacency relationship between the point features, geometric structure enhancement processing is performed on the hand and object point features respectively, so that the modeling capability for joint structure, contact area and object surface morphology is improved; finally, under the unified space constraint condition, hand pose parameters and object pose parameters are jointly predicted. The present application can effectively depict the space relationship between the hand and the object without explicit three-dimensional grid reconstruction, and significantly improves the stability and accuracy of three-dimensional hand-object pose estimation in a complex interaction scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D geometric modeling, and human-computer interaction, specifically to a 3D hand-object interaction pose estimation method based on implicit spatial constraints. Background Technology

[0002] 3D hand-object interaction pose estimation is a key foundational technology in human-computer interaction, virtual reality, augmented reality, and robotics. Its goal is to simultaneously recover the 3D joint pose of the human hand and the 6D spatial pose of the interacting object under 2D image input conditions. In virtual reality and augmented reality applications, accurately reconstructing the user's hand grasping motion and the object's movement state can significantly enhance the realism and immersion of virtual interactions. In robotics operation and teaching scenarios, perceiving the contact relationship and relative pose between the human hand and the object enables high-precision modeling and reproduction of complex grasping behaviors. This technology also has significant engineering application value in fields such as intelligent manufacturing and medical assistance.

[0003] From a technological development perspective, early hand-object pose estimation methods largely relied on multi-view camera systems or depth sensors to obtain geometric models of the hand and object through explicit 3D reconstruction. However, these methods have high hardware requirements and complex system deployment, making them unsuitable for lightweight and real-time interactive scenarios. With the development of deep learning technology, 3D hand-object interaction pose estimation methods based on monocular RGB images have gradually become a research hotspot. Existing methods typically use convolutional neural networks to extract 2D image features and combine them with parameterized hand models or pose regression networks to predict hand pose and object pose, respectively.

[0004] To improve geometric consistency, some existing techniques attempt to introduce explicit geometric constraints during the modeling process. For example, this involves reconstructing a 3D mesh model of the hand or object and modeling the contact relationship between the hand and object at the vertex space or surface level. While these methods can theoretically provide relatively fine geometric representations, they still have significant shortcomings in practical applications: Firstly, explicit mesh modeling is highly sensitive to image segmentation accuracy and geometric reconstruction quality, easily introducing accumulated errors in interactive scenarios with severe occlusion, rapid movement, or complex lighting. Secondly, mesh-level geometric inference typically involves high computational complexity and memory overhead, which is detrimental to real-time inference and embedded deployment. Furthermore, explicit geometric constraints often require precise correspondence modeling between the hand and object surfaces, which is difficult to implement in engineering and lacks robustness.

[0005] In real-world hand-object interaction scenarios, severe mutual occlusion and high-density contact are common between the hand and the object, making it easy for texture information in key areas of 2D images to be missing or aliased. Under these conditions, relying directly on explicit geometric reconstruction or explicit contact modeling during pose estimation often fails to stably depict the true spatial relationship between the hand and the object, easily leading to problems such as unreasonable structure and insufficient stability in pose prediction results.

[0006] Therefore, effectively characterizing the spatial constraints between the hand and the object during the feature construction and pose inference stages without relying on explicit 3D mesh reconstruction has become a key technical problem that urgently needs to be solved in this field. Compared with explicit geometric modeling, implicit spatial constraint modeling based on the feature level can guide the network to learn the relative positional relationship and geometric consistency between the hand and the object in a high-dimensional feature space while maintaining computational efficiency and system robustness. This provides a more engineering-feasible solution for 3D hand-object interaction pose estimation under complex occlusion conditions. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention aims to provide a three-dimensional hand-object interaction pose estimation method based on implicit spatial constraints. Under monocular RGB image input conditions, this method constructs a bidirectional coordinate representation of the hand and object in implicit three-dimensional space. This allows the hand features to perceive the spatial distribution of the object during the generation stage, while the object features perceive the grasping area of ​​the hand, thereby establishing a spatial constraint relationship between the two at the feature level. Based on this, by combining two-dimensional appearance features and three-dimensional geometric information, the hand pose parameters and object pose parameters are jointly predicted, effectively improving the stability and accuracy of pose estimation in complex interaction scenarios.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for estimating the pose of a 3D hand-object interaction based on implicit spatial constraints, characterized by the following steps:

[0010] S1. Acquire a monocular RGB image and process the RGB image through a multi-scale feature extraction network to obtain two-dimensional features for characterizing the appearance information of the hand and the object; generate a set of sampling points for the hand and a set of sampling points for the object in the implicit three-dimensional space (SDF), and perform coordinate transformation processing on the sampling point set to construct a two-way coordinate representation containing the spatial relationship between the hand and the object.

[0011] S2. Map the hand sampling point set and the object sampling point set to the image plane where the two-dimensional features are located, and perform feature sampling on the two-dimensional features at multiple scales to form point features that fuse two-dimensional appearance information and three-dimensional spatial information across modalities.

[0012] S3. Based on the spatial adjacency relationship between the point features, perform geometric enhancement processing on the hand point features and the object point features respectively;

[0013] S4. Based on the hand-object point set coordinates, jointly predict the hand pose parameters and object pose parameters, and output the three-dimensional hand pose and object pose results.

[0014] Furthermore, the construction process of the bidirectional coordinate transformation includes:

[0015] The hand sampling point set is translated with reference to the geometric center of the object or a preset reference point, so that the hand sampling point set is represented in the object space. This allows the spatial position information of the object to be incorporated into the hand features during the generation stage, and this process allows the hand to perceive the spatial distribution of the object. The object sampling point set is translated with reference to the root joint of the hand or a preset reference point, so that the object sampling point set is represented in the hand space. This allows the object features to reflect the contact position distribution of the hand during the generation stage, and this process allows the object to perceive the grasping intention of the hand. Through the above processing methods, the spatial constraint relationship between the hand and the object is established in the feature construction stage, so as to reduce the pose estimation error caused by occlusion or changes in viewpoint.

[0016] Furthermore, the construction process of the cross-modal fusion includes:

[0017] The hand-object point set with bidirectional coordinate representation is transformed into the camera coordinate system through the hand root joint position and scaling factor. Then, the 3D points are projected onto the image plane according to the camera intrinsic matrix, and the projected coordinates are normalized. Corresponding features are sampled on each layer of the feature pyramid. By sampling the semantic features of the projected points on the multi-scale feature map of the feature pyramid, cross-modal fusion point features are formed. These features contain both local SDF geometric information and multi-scale image semantics.

[0018] Furthermore, the construction process for the enhanced geometry includes:

[0019] In the hand sampling point set and the object sampling point set, local neighborhood relationships are constructed based on the three-dimensional spatial distance between points, and the point features within the neighborhood are aggregated to extract geometric features reflecting the local spatial structure. Specifically, a multi-scale neighborhood processing method is used for the hand sampling point set to simultaneously enhance the overall structural information of the hand as well as the geometric representation of local areas such as fingertips and knuckles. A neighborhood processing method oriented towards rigid structures is used for the object sampling point set to enhance the continuity of the object surface and the overall spatial stability.

[0020] Furthermore, the prediction process for the hand pose parameters and object pose parameters includes:

[0021] In the hand pose prediction process, the spatial distribution information corresponding to the object sampling point set is introduced to impose spatial constraints on the hand pose calculation process to avoid unreasonable hand configurations. In the object pose prediction process, the spatial distribution information corresponding to the hand sampling point set is introduced to impose auxiliary constraints on the object pose calculation process to improve the stability of object pose prediction. Through the above methods, the co-estimation of hand pose and object pose under the same spatial constraints is achieved.

[0022] The advantages and beneficial effects of this invention are:

[0023] This invention constructs a bidirectional coordinate representation of the hand and the object in implicit 3D space, enabling the hand and the object to establish spatial constraint relationships during the feature generation stage, effectively alleviating the posture instability problem caused by occlusion. By fusing 2D appearance information and 3D geometric information and introducing geometric structure enhancement processing, it improves the modeling ability of fingers, contact areas, and object surface structures. It adopts a hand-object collaborative joint prediction mechanism to improve the stability and consistency of object pose estimation while ensuring the rationality of hand posture. Practical application verification shows that this method has good generalization ability and is applicable to a variety of complex hand-object interaction scenarios. Attached Figure Description

[0024] Figure 1 A flowchart illustrating an example of the present invention;

[0025] Figure 2 This is a schematic diagram of the method model of the present invention;

[0026] Figure 3 This is a schematic diagram of the bidirectional coordinate transformation module structure proposed in this invention;

[0027] Figure 4 This is a schematic diagram of the cross-modal fusion module structure proposed in this invention;

[0028] Figure 5 This is a schematic diagram of the geometric enhancement module proposed in this invention;

[0029] Figure 6 The hand-object pose estimation results of the present invention on the HO3D dataset and the DEX-YCB dataset are visualized. Detailed Implementation

[0030] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0031] A method for estimating the pose of a 3D hand-object interaction based on implicit spatial constraints includes the following steps:

[0032] Step 1: Two-dimensional feature extraction and implicit space point set construction. Obtain a monocular RGB image. After preprocessing the input image such as cropping, scaling, and normalization, it is fed into a multi-scale feature extraction network to extract multi-scale two-dimensional features used to represent the appearance information of the hand and object. and Meanwhile, hand sampling point sets are generated in the implicit three-dimensional space based on the Signed Distance Function (SDF). With object sampling point set The sampling point set is then subjected to coordinate transformation to construct a bidirectional coordinate representation; the process is shown in formulas (1)-(3);

[0033] (1)

[0034] (2)

[0035] (3)

[0036] We used ResNet-50 as the backbone feature extraction network to extract multi-scale features of the hand and object at different stages of its output. and Low-level features ( It has a high resolution and contains rich edge and texture information; high-level features ( It has a larger receptive field and encodes stronger semantic information. Represents the geometric center or preset reference point of an object. This represents the root joint of the hand or a preset reference point. Through this transformation, the hand point set is represented in the object coordinate system, and the object point set is represented in the hand coordinate system, thereby establishing a spatial relationship between the hand and the object in implicit space.

[0037] Step 2: Cross-modal 2D-3D feature mapping fusion. The hand sampling point set and object sampling point set obtained in Step 1 are mapped to the image plane where the 2D features are located, and feature sampling is performed on the multi-scale 2D feature map to form point features that fuse 2D appearance information and 3D spatial information.

[0038] Specifically, based on the camera intrinsic parameter matrix The 3D points are projected onto the image plane, and the projected coordinates are normalized to the range ([-1, 1]) to adapt to PyTorch's grid_sample operation. For the projected coordinates... and We use grid_sample in each layer of the feature pyramid ( , The corresponding features are upsampled, and the semantic features of the projection points are sampled on the multi-scale feature map of the feature pyramid to form cross-scale image-geometry fusion point features. These features contain both local SDF geometric information and multi-scale image semantics. The process is shown in formulas (4)-(7).

[0039] (4)

[0040] (5)

[0041] (6)

[0042] (7)

[0043] in and These are the bidirectional coordinates of the hand and the object obtained in step 1, respectively, and K represents the camera's intrinsic parameter matrix. This represents a two-dimensional operation projected onto a plane, where x, y, and z represent the three-dimensional coordinates of the hand and the object, respectively. This is a cross-scale feature fusion operator, which means that continuous spatial feature sampling is performed on points on a multi-scale feature map, and the original two-dimensional mapping features are fused with three-dimensional points in terms of spatial scale.

[0044] Step 3: Geometric Enhancement Processing. The hand and object point features obtained in Step 2 are fed into this module. Based on the 3D spatial distance relationship between the point features, geometric enhancement processing is performed on the hand and object point features respectively. For any point p, its local neighborhood is constructed. It also aggregates the point features within the neighborhood and extracts local geometric structure information;

[0045] To address the highly deformable, highly free, and complex chain-like dependencies of the hand, we employ a topology-aware coding strategy using multi-scale graph convolution. Large receptive field graph convolution captures the overall geometric changes and global hand pose of the hand; small receptive field graph convolution focuses on local areas such as finger joints and fingertips, explicitly modeling the spatial dependencies between 21 joints. Specifically, we construct local neighborhood relationships of the hand point set based on the Euclidean distance between points. In each layer of geometric coding, we perform convolutional aggregation on the features of points within the neighborhood. First, we perform channel mapping on the features of points within the neighborhood, then normalize and perform nonlinear activation. Subsequently, we aggregate the features within the neighborhood through max pooling to extract structures robust to local geometric changes. The process is shown in equations (8)-(10).

[0046] (8)

[0047] (9)

[0048] (10)

[0049] in The preset radius of the hand's neighborhood is used to control the local geometric range. This represents a two-dimensional convolution operation. Indicates batch normalization, This represents the LeakyReLU activation function.

[0050] To address the rigid structure and surface geometric continuity of objects, we employ a topologically-aware coding strategy using multi-scale graph convolution. Large receptive field graph convolution is used to capture the overall contour of the object; small receptive field graph convolution focuses on learning the local surface characteristics of the object, used for geometric modeling of the spatial distribution and local surface structure of the object's surface. Specifically, we construct the local neighborhood relationships of the object's point set based on the Euclidean distance between points. In each layer of geometric coding, we use convolution and pooling modules with the same branch structure as the hand but independent parameters to perform convolutional aggregation processing on the point features within the neighborhood. First, channel mapping is performed on the point features within the neighborhood, followed by normalization and nonlinear activation. Subsequently, max pooling is used to aggregate the features within the neighborhood to extract structures robust to local geometric changes. The process is shown in equations (11)-(13).

[0051] (11)

[0052] (12)

[0053] (13)

[0054] in This is a preset radius of the object's neighborhood, used to control the local geometric range. This represents a two-dimensional convolution operation. Indicates batch normalization, This represents the LeakyReLU activation function.

[0055] For hand features, the aforementioned convolutional-pooling structure can be stacked in multiple layers to progressively expand the receptive field and enhance the joint modeling ability of the overall hand structure and local details (such as fingertips and knuckles). After multi-layer encoding, an enhanced set of geometric features of the hand is obtained. For object features, it can effectively suppress the interference of local noise points and enhance the spatial continuity of the object surface. The enhanced set of object geometric features is obtained after multi-layer encoding. As shown in formula (14).

[0056] (14)

[0057] in and These represent the number of hand sampling points and object sampling points in implicit space, respectively.

[0058] Step 4: Joint pose prediction. The enhanced geometric features of the hand and object from Step 3 are fed into the MANO regressor. Based on the hand-object point set coordinates, the hand pose parameters and object pose parameters are jointly predicted. In the hand pose prediction process, the spatial distribution information of the object point set is introduced to impose spatial constraints on the hand pose calculation process; in the object pose prediction process, the spatial distribution information of the hand point set is introduced to provide auxiliary constraints on the object pose calculation process. The joint prediction process is shown in Equation (15).

[0059] (15)

[0060] in, , These represent the hand posture parameters and shape parameters, respectively. Represents the object's pose parameters. This is a joint regression function. The final output is the 3D hand pose and object pose results.

[0061] Implementation:

[0062] Figure 1 This is a flowchart illustrating the 3D hand-object interaction pose estimation method based on implicit spatial constraints, according to an embodiment of the present invention. After preparing the dataset, the 3D hand-object interaction pose estimation network is trained and inferred.

[0063] S1. Prepare the dataset and perform preprocessing.

[0064] A monocular RGB image dataset containing hand-object interaction scenes is selected. Preprocessing operations such as alignment, cropping, scaling, and normalization are performed on the original images to reduce background interference and unify the input scale. In this embodiment, the input image size is set to 256×256. Corresponding hand-object point data is sampled in the symbolic distance field (SDF) to construct shape constraints in implicit space. The signed distance between the hand and the object is calculated using Grasping Field and DeepSDF, followed by SDF surface sampling to generate SDF 3D sampling point data.

[0065] S2. Construct a 3D hand-object interaction pose estimation network based on implicit spatial constraints.

[0066] like Figure 2As shown, the network includes a multi-scale feature extraction module, a bidirectional coordinate transformation module, a cross-modal fusion module, a geometric structure enhancement module, and a joint pose regression module.

[0067] In the multi-scale feature extraction module, ResNet-50 is used as the backbone network, combined with a Feature Pyramid Network (FPN) to extract two-dimensional features at different levels. Low-level features have high spatial resolution and contain rich edge and texture information; high-level features have a larger receptive field and encode stronger semantic information, providing a foundation for subsequent multi-scale sampling mapping.

[0068] In the bidirectional coordinate transformation module, a set of hand sampling points and a set of object sampling points are generated in the implicit three-dimensional space based on the signed distance function (SDF), and the sampling points are subjected to bidirectional coordinate transformation.

[0069] like Figure 3 As shown, the hand sampling point set is translated with reference to the geometric center of the object or a preset reference point, so that the hand point set is represented in the object space; the object sampling point set is translated with reference to the root joint of the hand or a preset reference point, so that the object point set is represented in the hand space. Then, the translated 3D points are aligned with the original root node and the new 3D point coordinates are calculated to ensure that they are correctly input into the network.

[0070] In the cross-modal fusion module, the sampling points of the hand and object in the implicit space are projected onto the image plane containing the two-dimensional features, and corresponding continuous spatial features are sampled on the multi-scale feature map of the feature pyramid, such as... Figure 4 As shown. Specifically, 3D points are projected onto a 2D plane based on the camera intrinsic parameter matrix to form projected coordinates, which are then normalized to the range [−1,1] to adapt to PyTorch's grid_sample operation. By sampling the projected points on multi-scale feature maps, point features that fuse 2D appearance information and 3D geometric information are constructed using feature sizes of different amplitudes (stride a ... stride x), providing input for subsequent geometric structure enhancement.

[0071] In the geometry enhancement module, the fused hand and object point features obtained from the cross-modal fusion module are fed into the geometry enhancement module. This module constructs local neighborhoods based on the 3D spatial distance relationships between points and performs convolution and pooling operations on the point features within the neighborhoods to extract local geometric structure information, such as... Figure 5As shown, we designed different neighborhood sizes for feature extraction based on the different structures of the hand and the object: Considering the hand's high deformability, high joint freedom, and complex topological dependencies, we sampled larger neighborhoods (large neighborhood k1=40, small neighborhood k2=20) to capture the hand's joint structure; considering the object's rigid structure and surface continuity, we used smaller neighborhoods (large neighborhood k1=20, small neighborhood k2=10) to capture the object's surface morphology. First, we performed simple feature extraction on the input hand and object features. Then, we designed two modules, A and B. Module A consists of convolutional layers, normalization layers, and activation layers, while module B consists of pooling layers. In the large neighborhood, the hand-object features are fed into the corresponding A and B modules to capture the geometric changes of the hand, the global hand pose, and the overall outline of the object. In the small neighborhood, the hand-object features are only captured by the A module to capture local areas such as the finger joints of the hand and the local surface features of the object. The features captured by the large and small neighborhoods are then concatenated and fed into the A and B modules again for feature refinement. After multi-layer convolution-pooling encoding, enhanced hand geometric features and object geometric features are obtained respectively.

[0072] S3. Input the data into the hand-object interaction pose estimation network to train it in order to estimate the hand-object pose.

[0073] We input the preprocessed image data and npy format point data from S1 into the network, extract hand joint information and object 6D pose information through the hand-object interaction pose estimation algorithm, and then use the MANO parametric hand model regressor to reconstruct the 3D hand pose. Figure 6 The results of this method on the HO3D and DexYCB datasets are shown: the left side is the image projected onto the original image with the estimated hand-object pose, the middle side is the front view of the generated 3D hand-object interaction pose, and the right side is a multi-angle display view of the same pose.

[0074] S4, Joint Attitude Regression and Prediction.

[0075] The enhanced geometric features of the hand and object are fed into the joint pose regression module for collaborative prediction of hand pose parameters and object pose parameters under the same spatial constraints. During hand pose prediction, spatial distribution information of the object point set is incorporated to constrain the hand configuration and avoid unreasonable interleaving or distortion; during object pose prediction, spatial distribution information of the hand point set is incorporated to improve the stability of pose estimation. In this embodiment, a MANO parametric hand model is used to regress the hand pose, and the network parameters are optimized using a loss function. The overall loss function is defined as follows:

[0076] (16)

[0077] in The hand pose loss is represented by the joint coordinate regression loss and the pose parameter regularization term, and the relevant loss is calculated using L2. The object pose loss is represented by two components: rotation loss and translation loss. The relevant loss is calculated using L2. The SDF reconstruction loss is used to ensure that the network can accurately learn the implicit geometric representations of the hand and the object. The loss is calculated using a point-level L1 distance function. To represent the contact field loss, we learn the contact probability distribution between the hand and the object, using a binary cross-entropy function; , , , These are the weighting coefficients for hand pose, object pose, SDF reconstruction, and contact field loss, respectively.

Claims

1. A three-dimensional hand-object interaction pose estimation method based on implicit spatial constraints, characterized in that, Includes the following steps: Step 1: Two-dimensional feature extraction and implicit spatial point set construction: A monocular RGB image is acquired, and after preprocessing operations such as cropping, scaling, and normalization, the image is fed into a multi-scale feature extraction network. ResNet-50 is used as the backbone feature extraction network to extract multi-scale two-dimensional features from its outputs at different stages, representing information about the hand and object appearance. and Hand sampling point sets are generated in implicit 3D space based on the Signed Distance Function (SDF). With object sampling point set And perform coordinate transformation on the sampling point set to construct a two-way coordinate representation, which includes the spatial relationship between the hand and the object. The process is shown in formulas (1)-(3). (1)； (2)； (3)； Among them, low-level features ( It has a high resolution and contains rich edge and texture information; high-level features ( It has a larger receptive field and encodes stronger semantic information. Represents the geometric center or preset reference point of an object. This transformation represents the root joint of the hand or a preset reference point. It allows the hand point set to be represented in the object coordinate system, and at the same time, the object point set to be represented in the hand coordinate system, thereby establishing a spatial relationship between the hand and the object in implicit space. Step 2, Cross-modal 2D-3D feature mapping fusion: The hand sampling point set and object sampling point set obtained in step 1 are mapped to the image plane where the two-dimensional features are located, and feature sampling is performed on the two-dimensional features at multiple scales to form point features that fuse two-dimensional appearance information and three-dimensional spatial information across modalities. Based on the camera intrinsic parameter matrix The 3D points are projected onto the image plane, and the projected coordinates are normalized to the range ([-1,1]) to adapt to PyTorch's grid_sample operation; for the projected coordinates and Use grid_sample at each level of the feature pyramid ( , The corresponding features are upsampled, and the semantic features of the projection points are sampled on the multi-scale feature map of the feature pyramid to form cross-scale image-geometry fusion point features. These features contain both local SDF geometric information and multi-scale image semantics. The process is shown in formulas (4)-(7). (4)； (5)； (6)； (7)； in, and These are the bidirectional coordinates of the hand and the object obtained in step 1, respectively, and K represents the camera's intrinsic parameter matrix. This represents a two-dimensional operation projected onto a plane, where x, y, and z represent the three-dimensional coordinates of the hand and the object, respectively. This is a cross-scale feature fusion operator, which means that continuous spatial feature sampling is performed on points on a multi-scale feature map, and the original two-dimensional mapping features are fused with three-dimensional points in terms of spatial scale. Step 3: Geometric Enhancement Processing The hand and object point features obtained in step 2 are fed into this module. Based on the three-dimensional spatial distance relationship between the point features, geometric enhancement processing is performed on the hand point features and the object point features respectively: for any point p, its local neighborhood is constructed. It also aggregates the point features within the neighborhood and extracts local geometric structure information; Step 31: Considering the characteristics of the hand—high deformability, extremely high degree of freedom, and complex chain-like joint dependencies: First, channel mapping is performed on the features of points in the neighborhood, followed by normalization and nonlinear activation. Then, the features in the neighborhood are aggregated through max pooling to extract structures that are robust to local geometric changes. The process is shown in formulas (8)-(10). (8) (9) (10) in, The preset radius of the hand's neighborhood is used to control the local geometric range. This represents a two-dimensional convolution operation. Indicates batch normalization, This represents the LeakyReLU activation function; Step 32: Considering the characteristics of the rigid structure and surface geometric continuity of the object: First, channel mapping is performed on the features of points in the neighborhood, followed by normalization and nonlinear activation. Then, the features in the neighborhood are aggregated through max pooling to extract structures that are robust to local geometric changes. The process is shown in equations (11)-(13). (11) (12) (13) in This is a preset radius of the object's neighborhood, used to control the local geometric range. This represents a two-dimensional convolution operation. Indicates batch normalization, This represents the LeakyReLU activation function; Step 33: After multi-layer coding, an enhanced set of object geometric features is obtained. As shown in formula (14); (14) in and These represent the number of hand sampling points and object sampling points in implicit space, respectively; Step 4, Joint Attitude Prediction: The enhanced geometric features of the hand and object in step 3 are fed into the MANO regressor. Based on the hand-object point set coordinates, the hand pose parameters and object pose parameters are jointly predicted. In the hand pose prediction process, the spatial distribution information of the object point set is introduced to apply spatial constraints to the hand pose calculation process. In the object pose prediction process, the spatial distribution information of the hand point set is introduced to apply auxiliary constraints to the object pose calculation process. The joint prediction process is shown in formula (15). (15) in, , These represent the hand posture parameters and shape parameters, respectively. Represents the object's pose parameters. The joint regression function ultimately outputs the three-dimensional hand pose and object pose results.

2. The method for estimating the pose of a 3D hand-object interaction based on implicit spatial constraints according to claim 1, characterized in that, The construction process of the bidirectional coordinate transformation includes: The hand sampling point set is translated with reference to the geometric center of the object or a preset reference point, so that the hand sampling point set is represented in the object space. This allows the spatial position information of the object to be incorporated into the hand features during the generation stage, and this process allows the hand to perceive the spatial distribution of the object. The object sampling point set is translated with reference to the root joint of the hand or a preset reference point, so that the object sampling point set is represented in the hand space. This allows the object features to reflect the contact position distribution of the hand during the generation stage, and this process allows the object to perceive the grasping intention of the hand. Through the above processing methods, the spatial constraint relationship between the hand and the object is established in the feature construction stage, so as to reduce the pose estimation error caused by occlusion or changes in viewpoint.

3. The method for estimating the pose of a 3D hand-object interaction based on implicit spatial constraints according to claim 1, characterized in that, The construction process of the cross-modal fusion includes: The hand-object point set with bidirectional coordinate representation is transformed into the camera coordinate system through the hand root joint position and scaling factor. Then, the 3D points are projected onto the image plane according to the camera intrinsic matrix, and the projected coordinates are normalized. Corresponding features are sampled on each layer of the feature pyramid. By sampling the semantic features of the projected points on the multi-scale feature map of the feature pyramid, cross-modal fusion point features are formed. These features contain both local SDF geometric information and multi-scale image semantics.

4. The method for estimating the pose of a three-dimensional hand-object interaction based on implicit spatial constraints according to claim 1, characterized in that, The process of constructing the geometric enhancement includes: In the hand sampling point set and the object sampling point set, local neighborhood relationships are constructed based on the three-dimensional spatial distance between points, and the point features within the neighborhood are aggregated to extract geometric features reflecting the local spatial structure. Specifically, a multi-scale neighborhood processing method is used for the hand sampling point set to simultaneously enhance the overall structural information of the hand as well as the geometric representation of local areas such as fingertips and knuckles. A neighborhood processing method oriented towards rigid structures is used for the object sampling point set to enhance the continuity of the object surface and the overall spatial stability.

5. The method for estimating the pose of a three-dimensional hand-object interaction based on implicit spatial constraints according to claim 1, characterized in that, The prediction process for the hand pose parameters and the object pose parameters includes: In the hand pose prediction process, the spatial distribution information corresponding to the object sampling point set is introduced to impose spatial constraints on the hand pose calculation process to avoid unreasonable hand configurations. In the object pose prediction process, the spatial distribution information corresponding to the hand sampling point set is introduced to impose auxiliary constraints on the object pose calculation process to improve the stability of object pose prediction. Through the above methods, the co-estimation of hand pose and object pose under the same spatial constraints is achieved.

Images