A hand-object interaction pose estimation method based on text prompts

Abstract

This invention discloses a hand-object interaction pose estimation method based on text prompts, relating to the fields of computer vision and natural language processing. The method includes: constructing a hand-object interaction pose prediction model, inputting image data and text data; the model consists of a feature extraction layer, a feature fusion layer, a feature supervision layer, and an output layer; the feature extraction layer inputs a 2D image into a ResNet-50 network to extract image features; in the feature fusion layer, global text prompts are encoded by a global text encoder and input into a multilayer perceptron, mapped to global dynamic parameters, and fused with image features to obtain global fused features; local text prompts are encoded by a local text encoder and input into a multilayer perceptron, mapped to local dynamic parameters, and fused with global fused features to obtain harmonic features; the feature supervision layer sequentially constructs global supervision using global text and image features, constructs joint supervision using joint names and corresponding image features, and constructs pixel supervision using joint names and their corresponding heatmaps, optimizing the overall network framework; the output layer outputs the attributes and position of each pixel in the image according to the matching degree.

Description

Technical Field

[0001] This application relates to the fields of computer vision and natural language processing, and in particular to a method for estimating hand-object interaction pose based on text prompts. Background Technology

[0002] Vision-based hand interaction can intuitively express human intentions in augmented reality operations, sign language dialogue, and human-computer interaction, and has become an important means in human activity scenarios. However, vision-based hand interaction inevitably results in joint occlusion, including occlusion by other parts of the hand, by another hand, and by objects. This loss of visual information due to occlusion leads to a decrease in the accuracy of vision-based hand pose estimation and hand model reconstruction. Precise textual guidance can overcome the unreliability of purely visual cues under occlusion conditions, explicitly encoding semantic priors about occluded areas and spatial interaction patterns, and providing flexible and complementary constraints to recover geometrically reasonable poses.

[0003] In existing technologies, feature-level fusion is often used, such as simple concatenation, addition, or multiplication between image features and text. However, this can easily introduce redundant information and cannot effectively integrate the advantages of the two modalities. Neither of these technologies takes into account the complexity of hand-object interaction or the problem of hand joints being frequently occluded. Moreover, textual descriptions of images often remain superficial, such as simply describing the shape of the hand or the hand's interaction with an object. This further leads to a significant decrease in the accuracy of hand-object estimation under occlusion conditions.

[0004] Therefore, there is an urgent need for an occlusion representation text prompt method for hand-object interaction pose estimation to improve the hand-object estimation accuracy under occlusion conditions. Summary of the Invention

[0005] Therefore, it is necessary to provide a hand-object interaction pose estimation method based on text prompts to address the above-mentioned technical problems.

[0006] The present invention adopts the following technical solution: This invention provides a method for estimating hand-object interaction pose based on text prompts, comprising: The system acquires 2D images of hand-object interaction, global hand-object interaction relationship text prompts, and local finger joint occlusion text prompts, and inputs them into a hand-object interaction posture prediction model. The hand-object interaction posture prediction model includes a feature extraction layer, a feature fusion layer, a feature supervision layer, and a feature output layer. In the feature extraction layer, the 2D image of hand-object interaction is input into the trained ResNet-50 network to obtain image features; In the feature fusion layer, the global hand-object interaction text prompt is input into the global text encoder for encoding to obtain global text features, which are then input into the multilayer perceptron and mapped to global text guidance dynamic parameters. Based on the global text guidance dynamic parameters, the global text features are fused with image features to obtain global fused features. The text prompt for local finger joint occlusion is input into the local text encoder for encoding to obtain local text features. The local text features are input into the multilayer perceptron and mapped to local text guidance dynamic parameters. Through the local text guidance dynamic parameters, the local text features are fused with the global fused features to obtain multi-scale fused features. In the feature supervision layer, the global text features are fused with the image features and then globally aligned with the multi-scale fused features; the local text features are fused with the image features and then locally aligned and pixel aligned with the globally aligned multi-scale fused features to obtain the aligned multi-scale fused features. In the feature output layer, the aligned multi-scale fused features are processed by the 3D field learning module to learn the signed distance field information of the hand and the object. After passing through the field-guided pose regression module, the final 3D pose parameters of the hand and the object are obtained.

[0007] Preferably, the 2D image of the hand-object interaction is input into a trained ResNet-50 network to obtain image features, specifically including: Input the 2D image of the hand-object interaction into the ResNet-50 network; ResNet-50 network is used to obtain hand-object interaction 2D image features from hand-object interaction 2D images; The 2D image features of hand-object interaction are input into spatial attention mechanism and channel attention mechanism, and the image features are obtained by combining the two.

[0008] Preferably, global alignment includes: Global text features and image features are fused to obtain a global fused feature map; Align the multi-scale fusion features with the global fusion feature map to obtain the globally aligned multi-scale fusion features.

[0009] Preferably, local alignment specifically includes: The image features are processed using multi-head attention to obtain the processed image features; The local text features are fused with the processed image features to obtain a local fused feature map; Align the local fused feature map with the globally aligned multi-scale fused feature map to obtain the locally aligned multi-scale fused feature map.

[0010] Preferably, pixel alignment specifically includes: Extracting joint heatmaps from image features; The joint heatmap is fused with local text features to obtain the fused joint text features; The fused joint text features are then fused with the image features to obtain a joint fusion feature map; Align the locally aligned multi-scale fusion features with each other to obtain the pixel-aligned multi-scale fusion features, i.e., the aligned multi-scale fusion features.

[0011] Preferably, the weights within the local text editor remain unchanged during the training of the hand-object interaction pose prediction model.

[0012] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: This invention provides a text-based hand-object interaction pose estimation method. A hand-object interaction pose prediction model is constructed, which generates detailed text descriptions through a dual-scale structured text prompt generation module. A feature fusion module hierarchically combines text semantic information with image features to enhance the features of the hand-object contact area while suppressing background interference. This allows the model to initially focus on areas in the image consistent with the text description. Next, more specific occlusion text information is introduced. By locally constraining and adjusting the fused features, the global semantic intent is addressed first during the fusion process. Local constraints adjust details, enabling the model to more accurately capture the occlusion of hand joints and the details of hand-object interaction. A feature supervision module utilizes multi-level supervision to improve the model's feature learning ability at different granularities, thereby enhancing the accuracy of hand-object pose estimation. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 A flowchart illustrating a hand-object interaction pose estimation method based on text prompts provided by this invention; Figure 2 A flowchart of text generation for a hand-object interaction pose estimation method based on text prompts provided by this invention; Figure 3 A schematic diagram of a pose prediction model for a hand-object interaction pose estimation method based on text prompts provided by the present invention; Figure 4 A schematic diagram illustrating image feature extraction for a hand-object interaction pose estimation method based on text prompts provided by this invention; Figure 5 A schematic diagram illustrating feature fusion for a hand-object interaction pose estimation method based on text prompts provided by this invention; Figure 6 This invention provides a feature supervision module diagram for a hand-object interaction pose estimation method based on text prompts. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the specification without creative effort are within the scope of protection of this application.

[0016] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating a hand-object interaction pose estimation method based on text prompts according to the present invention, specifically including the following steps: S101: Acquire 2D images of hand-object interaction, global hand-object interaction relationship text prompts, and text prompts for local finger joint occlusion, and input them into the hand-object interaction posture prediction model; the hand-object interaction posture prediction model includes a feature extraction layer, a feature fusion layer, a feature supervision layer, and a feature output layer. See the model diagram below. Figure 3 .

[0018] Specifically, this invention first provides a text prompt template that generates global hand-object interaction text prompts and local finger joint occlusion text prompts. The prompts include the contact state between the hand and the object (whether the object is grasped or not) and finger joint occlusion information. See also... Figure 2 This is a flowchart for text generation. In the diagram, green represents self-occlusion joints, red represents object-occlusion joints, and light blue represents visible joints.

[0019] S102: In the feature extraction layer, see Figure 4 The diagram illustrates the feature extraction process. The 2D image of the hand-object interaction is input into the ResNet50 network to obtain image features. This includes: inputting the 2D image of the hand-object interaction into the ResNet50 network; the ResNet50 network obtaining the 2D image features of the hand-object interaction; and inputting the image features into the spatial attention mechanism and the channel attention mechanism to obtain the enhanced image features.

[0020] Specifically, the feature extraction layer uses a network composed of ResNet50 to encode and decode the input 2D image.

[0021] S103: In the feature fusion layer, see Figure 5 The global hand-object interaction text prompts are input into a global text encoder for encoding, obtaining global text features, which are then input into a multilayer perceptron and mapped to global text guidance dynamic parameters. Based on the global text guidance dynamic parameters, the global text features are fused with image features to obtain global fused features. The text prompts for local finger joint occlusion are input into a local text encoder for encoding, obtaining local text features. The local text features are then input into a multilayer perceptron and mapped to local text guidance dynamic parameters. Through the local text guidance dynamic parameters, the local text features are fused with the global fused features to obtain multi-scale fused features.

[0022] Specifically, the weights within the text editor remain unchanged during training.

[0023] Specifically, the global text-guided dynamic parameters and the local text-guided dynamic parameters are used to adjust the scale and offset of the image features and the global fusion features, respectively.

[0024] Specifically, this phased fusion approach avoids the conflict between high-level guidance and low-level structural adjustments, making the fusion process more consistent with human visual reasoning. Specifically, to fully utilize textual information to guide model learning, this invention employs two pre-trained visual language models, CLIP and Long-CLIP, as text encoders to encode hand-object interaction text and occluded text, respectively, to obtain corresponding text embedding features. To maintain linguistic consistency, we freeze the weights of these two text encoders, ensuring the stability and consistency of text features and avoiding semantic drift caused by weight updates during training. To enable the extracted text features to better adapt to subsequent fusion tasks, we introduce a lightweight multilayer perceptron to map text embedding features to dynamic parameters. α n and β n These two parameters will be used to modulate image features to achieve the fusion of text and image features. Specifically, α n and β n These parameters are used to adjust the scale and offset of image features, thereby dynamically incorporating textual semantic information into the image features. This is achieved by using dynamic parameters. α n and β n The image features are modulated to further optimize the fusion results, enabling the model to more accurately capture the occlusion of hand joints and the details of hand-object interaction. The specific process is as follows: Figure 5The diagram illustrates image-text feature fusion. By enhancing the features of the hand-object contact area while suppressing background interference, the model can initially focus on regions in the image consistent with the text description. The purpose of this step is to utilize global semantic information to provide a foundation for subsequent feature refinement.

[0025] S104: In the feature supervision layer, see Figure 6 The diagram illustrates feature supervision. The global text features are fused with the image features and then globally aligned with the multi-scale fused features. The local text features are fused with the image features and then locally aligned and pixel-aligned with the globally aligned multi-scale fused features to obtain the aligned multi-scale fused features.

[0026] Optionally, global alignment specifically includes: fusing global text features with image features to obtain a global fused feature map; and aligning the multi-scale fused features with the global fused feature map to obtain globally aligned multi-scale fused features.

[0027] Specifically, multi-scale fusion features are supervised using global text features. First, multi-scale visual features are extracted from the multi-scale fusion features and fused with the text features of hand-object interaction. Then, the normalized fusion features are combined with new image features to calculate... F sc By adjusting the model parameters based on the similarity between the images, the image features can better reflect the semantic information in the text description. This alignment strategy ensures that the model can accurately capture the macroscopic interaction between the hand and the object when processing the whole image, laying the foundation for subsequent detailed analysis.

[0028] Optionally, local alignment specifically includes: processing image features through multi-head attention to obtain processed image features; fusing local text features with processed image features to obtain a local fused feature map; and aligning the local fused feature map with the globally aligned multi-scale fused features to obtain locally aligned multi-scale fused features.

[0029] Specifically, at the local joint level, we focus on feature learning for hand joints. By aligning the textual features of hand joints with the corresponding joint features of the multi-scale fused features, the model can more accurately identify and locate hand joints. Specifically, we generate corresponding textual features for each hand joint and extract local features of the joint region from the image features and fuse them into the image features. Next, we calculate the similarity between the fused image features and the multi-scale fused features and optimize it using a contrastive loss function. This allows the model to deeply understand the local features of hand joints, accurately capturing subtle changes in hand posture even in complex hand-object interaction scenarios.

[0030] Optionally, pixel alignment specifically includes: extracting a joint heatmap from image features; fusing the joint heatmap with local text features to obtain fused joint text features; fusing the fused joint text features with image features to obtain a joint fusion feature map; and aligning the locally aligned multi-scale fusion features with the locally aligned multi-scale fusion features to obtain pixel-aligned multi-scale fusion features, i.e., aligned multi-scale fusion features.

[0031] Specifically, at the pixel level, we use 2D label heatmaps to further refine the model's ability to locate image features. Each hand joint has a corresponding heatmap ( Figure 6 (The right joint position is highlighted). By learning these heatmaps, the model can accurately determine the specific location of each joint in the image. Specifically, we combine heatmap information with image features, utilizing a multi-head self-attention mechanism to enhance the expressive power of the features. Then, we calculate the matching degree between the multi-scale fused features and the joint text features in the image features in the embedding space, and compare it with the target heatmap to optimize the model parameters. In this way, the model can accurately predict the attributes and position of each pixel at the pixel level, thereby achieving high-precision estimation of hand-object pose. The specific process is as follows... Figure 6 The diagram shown is a schematic of the text supervision module.

[0032] S105: In the feature output layer, the aligned multi-scale fused features are processed by the 3D field learning module to learn the signed distance field information of the hand and the object. After passing through the field-guided pose regression module, the final 3D pose parameters of the hand and the object are obtained.

[0033] Optionally, in the feature output layer, the attributes and position of each pixel in the image are output based on the matching degree between the enhanced image features and the local alignment features.

[0034] This invention first proposes a novel image-text fusion method, successfully integrating textual semantic information with image features in a deep and precise manner. By introducing a dynamic parameter modulation mechanism, it not only enhances the model's focus on key areas of hand-object interaction in the image but also effectively suppresses interference from irrelevant background information. In the hand-object pose estimation task, this method enables the model to more accurately capture the occlusion of hand joints and the pose details of objects, thereby significantly improving the accuracy and robustness of hand-object pose estimation, especially in complex scenes and under occlusion conditions.

[0035] Furthermore, this invention uses global hand-object interaction text, local joint label text, and 2D heatmaps to supervise the fused image features, achieving multi-level feature learning from the overall to the detailed, and from semantic understanding to precise localization. This not only significantly improves the accuracy and robustness of hand-object pose estimation, but also, in complex scenes and under occlusion conditions, can more accurately capture the occlusion of hand joints and the pose details of objects, thus achieving higher-quality hand-object pose estimation.

[0036] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A method for estimating hand-object interaction pose based on text prompts, characterized in that, include: The system acquires 2D images of hand-object interaction, global hand-object interaction relationship text prompts, and local finger joint occlusion text prompts, and inputs them into a hand-object interaction posture prediction model. The hand-object interaction posture prediction model includes a feature extraction layer, a feature fusion layer, a feature supervision layer, and a feature output layer. The global hand-object interaction relationship text prompt is the contact state between the hand and the object; The text prompt for partial finger joint occlusion is the occlusion information of the finger joint, including: self-occluded joints, object-occluded joints, and visible joints; In the feature extraction layer, the 2D image of hand-object interaction is input into the trained ResNet-50 network to obtain image features; In the feature fusion layer, the global hand-object interaction text prompt is input to the global text encoder for encoding to obtain global text features, which are then input to the multilayer perceptron and mapped to global text guidance dynamic parameters. Based on the global text guidance dynamic parameters, the global text features are fused with image features to obtain global fused features. The text prompt for local finger joint occlusion is input to the local text encoder for encoding to obtain local text features. The local text features are input to the multilayer perceptron and mapped to local text guidance dynamic parameters. Through the local text guidance dynamic parameters, the local text features are fused with the global fused features to obtain multi-scale fused features. The global text guidance dynamic parameters and the local text guidance dynamic parameters are used to adjust the scale and offset of the image features and the global fused features, respectively. In the feature supervision layer, the global text features are fused with the image features and then globally aligned with the multi-scale fused features. The local text features are fused with the image features and then locally and pixelally aligned with the globally aligned multi-scale fused features to obtain aligned multi-scale fused features. The global alignment includes: fusing global text features with image features to obtain a global fused feature map; aligning the multi-scale fused features with the global fused feature map to obtain globally aligned multi-scale fused features. The local alignment includes: processing the image features through multi-head attention to obtain a processed image. Features; fusing local text features with processed image features to obtain a local fused feature map; aligning the local fused feature map with globally aligned multi-scale fused features to obtain locally aligned multi-scale fused features; the pixel alignment includes: extracting joint heatmaps from image features; fusing the joint heatmaps with local text features to obtain fused joint text features; fusing the fused joint text features with image features to obtain a joint fused feature map; aligning the locally aligned multi-scale fused features with the joint fused feature map to obtain pixel-aligned multi-scale fused features, i.e., aligned multi-scale fused features; In the feature output layer, the aligned multi-scale fused features are processed by the 3D field learning module to learn the signed distance field information of the hand and the object. After passing through the field-guided pose regression module, the final 3D pose parameters of the hand and the object are obtained.

2. The hand-object interaction pose estimation method based on text prompts as described in claim 1, characterized in that, The step of inputting the 2D image of the hand-object interaction into the trained ResNet-50 network to obtain image features specifically includes: Input the 2D image of the hand-object interaction into the ResNet-50 network; The ResNet-50 network is used to process 2D images of hand-object interaction to obtain the features of the 2D images of hand-object interaction. The 2D image features of hand-object interaction are input into spatial attention mechanism and channel attention mechanism, and the image features are obtained by combining the two.

3. The hand-object interaction pose estimation method based on text prompts as described in claim 1, characterized in that, The weights within the local text encoder remain unchanged during the training of the hand-object interaction pose prediction model.

Images