Hand-object estimation method and system, electronic equipment and storage medium

Abstract

The invention belongs to the technical field of computer vision, and discloses a hand-object estimation method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining a hand-object interaction image, and extracting a hand and object deep feature map; predicting a hand vertex 2D coordinate and an object 2D coordinate; performing posture alignment on the 2D coordinate of the vertex of the hand and the deep feature map of the hand and the 2D coordinate of the object and the deep feature map of the object to obtain posture alignment features of the hand and the object, and mapping the posture alignment features to a 3D space to obtain 3D space features of the hand and the object; capturing fine-grained interactivity of 3D spatial features of the hand and the object through a cross attention mechanism, and generating enhanced features of the hand and the object; enhancing the similarity between hand vertexes and the difference between the hand vertexes and environment voxels by using comparative learning to obtain hand space optimization features; and finally, decoding the object enhancement feature and the hand space optimization feature, and outputting a hand-object estimation result. According to the method, the performance of the 3D interactive hand-object attitude estimation method can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a hand-object estimation method, system, electronic device and storage medium based on three-dimensional interactive modeling and vertex contrast learning. Background Art

[0002] 3D hand-object pose estimation is one of the hot research directions in the field of computer vision, and plays a vital role in virtual reality, human-computer interaction and other fields. This task is usually to estimate or reconstruct the hand and the manipulated object in the image. In recent years, with the rapid development of deep neural network technology and the emergence of large-scale hand-object interaction datasets, many breakthroughs have been made in the research of hand and object poses. At present, estimating the pose of a single hand and a single object from a monocular image has achieved relatively good results. However, due to the high degree of freedom of the hand and the mutual occlusion caused by the frequent contact between the hand and the object, estimating the interactive hand-object pose from a monocular image is more challenging.

[0003] At present, interactive hand-object pose estimation methods have achieved relatively satisfactory results, but there are still some problems: (1) The core of the 3D interactive hand-object pose estimation task is to accurately model the interactive relationship between the hand and the object. In recent years, although the basic position and posture characteristics of the hand and the object can be captured, there are deficiencies in the details of the hand-object interaction, especially the interactive contact between the hand and the object mainly occurs on the surface, which does not make good use of the surface contour information of the hand and the object. (2) Joint estimation of hand-object pose in complex backgrounds, especially modeling of small objects, is challenging. Because small objects occupy a small area in the image, less effective information is extracted from the image. In addition, in complex backgrounds, the image may contain multiple objects, which may be similar to the object being operated in color or shape. However, the current mainstream methods do not adequately capture the features of these small objects or similar objects. Therefore, the above two problems limit the performance of 3D interactive hand-object pose estimation methods. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a hand-object estimation method, system, electronic device and storage medium, aiming to improve the performance of a three-dimensional interactive hand-object posture estimation method. On the one hand, through a three-dimensional interactive modeling method, a mapping of multi-scale two-dimensional to three-dimensional spatial features is achieved, while capturing the fine-grained interaction correlation between the hand and the object in three-dimensional space, and enhancing the representation of interaction details; on the other hand, through comparative learning of positive and negative samples between hand vertices in three-dimensional space, the similarity and difference between hand mesh vertices and environmental voxels in complex backgrounds are enhanced, thereby enhancing the model's ability to handle hand-object posture estimation in complex scenes.

[0005] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a hand-object estimation method, the method comprising: Acquire hand-object interaction images; Performing feature extraction on the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively; Predicting the hand joints on the hand deep feature map to obtain the 2D coordinates of the hand vertices; encoding the object heat map on the object deep feature map to obtain the 2D coordinates of the object; Performing posture alignment of the hand vertex 2D coordinates and the hand deep feature map to obtain a hand posture alignment feature; performing posture alignment of the object 2D coordinates and the object deep feature map to obtain an object posture alignment feature; Performing 3D feature mapping on the hand posture alignment feature and the object posture alignment feature to obtain hand 3D spatial features and object 3D spatial features respectively; According to the cross attention mechanism, the fine-grained interaction relationship between the 3D spatial features of the hand and the 3D spatial features of the object is captured to obtain hand enhancement features and object enhancement features; By using contrastive learning of positive and negative samples, the similarity between hand vertices in the hand enhancement feature is enhanced, and the difference between the hand vertices in the hand enhancement feature and the environment voxels is enhanced, so as to obtain a hand space optimization feature; The object enhancement feature and the hand space optimization feature are decoded to obtain a hand-object estimation result.

[0006] As an optional implementation manner of the first aspect of the present application, the steps of performing feature extraction on the hand-object interaction image to obtain a deep feature map of the hand and a deep feature map of the object respectively include: using a Restnet50-FPN single- and double-branch hybrid network to perform shallow feature extraction on the hand-object interaction image to obtain shallow features of the hand and shallow features of the object; using a multi-scale backbone to perform multi-scale feature extraction on the shallow features of the hand and the shallow features of the object respectively to obtain a deep feature map of the hand and a deep feature map of the object.

[0007] As an optional implementation of the first aspect of the present application, the step of predicting hand joint points on the deep hand feature map to obtain 2D coordinates of hand vertices includes: extracting hand features from the deep hand feature map through an hourglass module, introducing residual learning through residual connections, and processing hand features through basic blocks to obtain three different feature vectors, wherein the three different feature vectors are a two-dimensional heat map, a 2D joint point coordinates, and a latent feature map.

[0008] As an optional implementation of the first aspect of the present application, the step of performing object heat map encoding on the deep feature map of the object to obtain the 2D coordinates of the object includes: passing the deep feature map of the object through a convolution block and an activation function to obtain a two-dimensional heat map representation of the object, wherein the two-dimensional heat map representation of the object includes a dimension height H and a dimension width W; averaging the dimension height H and the dimension width W to quantize the two-dimensional heat map coordinates, and converting them into corresponding one-dimensional vectors respectively; assuming W and H The tensors of dimensions represent heat map indexes on dimension width W and dimension height H respectively; the normalized one-dimensional vector is multiplied by the corresponding heat map index to obtain the 2D coordinates of the object.

[0009] As an optional implementation of the first aspect of the present application, before the step of performing 3D feature mapping on the hand posture alignment features and the object posture alignment features, the step also includes: preprocessing the hand posture alignment features and the object posture alignment features, the preprocessing including: downsampling the hand posture alignment features by four times and simplifying the number of hand vertices; and simplifying the number of object vertices of the object posture alignment features.

[0010] As an optional implementation of the first aspect of the present application, according to the cross-attention mechanism, the steps of capturing the fine-grained interaction between the hand 3D spatial features and the object 3D spatial features to obtain hand enhancement features and object enhancement features include: using a one-dimensional convolutional layer to respectively extract the query value and feature value of the hand 3D spatial features and the object 3D spatial features; obtaining the corresponding attention weighted features according to the query value, the feature value and the sum of the feature dimensions; using a one-dimensional convolutional layer to respectively fuse the hand 3D spatial features and the object 3D spatial features with the corresponding attention weighted features to respectively obtain hand enhancement features and object enhancement features.

[0011] As an optional implementation of the first aspect of the present application, the similarity between hand vertices in the hand enhancement feature and the difference between hand vertices and environmental voxels in the hand enhancement feature are enhanced by contrast learning of positive and negative samples, and the step of obtaining the hand space optimization feature includes: using environmental voxels to form an embedded representation of the vertex coordinates, wherein each environmental voxel represents the feature information of a local area in the hand enhancement feature to capture the similarities and differences between the data, for each hand vertex, the positive sample is the predicted hand vertex coordinate value, and the negative sample is the environmental voxel; based on the hand vertices and the environmental voxels, the hand InfoNCE loss function is used to obtain the hand space optimization feature, and the hand InfoNCE loss function includes a first contrast loss function between hand vertices and a second contrast loss function between hand vertices and environmental voxels.

[0012] In a second aspect, an embodiment of the present application provides a hand-object estimation system, the system comprising: An image acquisition module is used to acquire a hand-object interaction image; A three-dimensional dense modeling module is used to extract features from the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively; predict hand joint points on the hand deep feature map to obtain 2D coordinates of hand vertices; perform object heat map encoding on the object deep feature map to obtain object 2D coordinates; perform posture alignment on the hand vertex 2D coordinates and the hand deep feature map to obtain hand posture alignment features; perform posture alignment on the object 2D coordinates and the object deep feature map to obtain object posture alignment features; perform 3D feature mapping on the hand posture alignment features and the object posture alignment features to obtain hand 3D spatial features and object 3D spatial features respectively; and capture the fine-grained interaction relationship between the hand 3D spatial features and the object 3D spatial features according to the cross-attention mechanism to obtain hand enhancement features and object enhancement features; A vertex contrast learning module, used for enhancing the similarity between hand vertices in the hand enhancement feature and enhancing the difference between hand vertices and environmental voxels in the hand enhancement feature by contrast learning of positive and negative samples, so as to obtain a hand space optimization feature; The output image module is used to decode the object enhancement features and the hand space optimization features to obtain a hand-object estimation result.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] Compared with the prior art, the present invention proposes a hand-object estimation method: by acquiring the hand-object interaction image and extracting the deep feature maps of the hand and the object respectively, the key visual information of the hand and the object can be captured, providing a data basis for subsequent processing; the hand joint points are predicted on the hand deep feature map to obtain the 2D coordinates of the hand vertices, and the object deep feature map is encoded with the object heat map to obtain the object 2D coordinates, which can accurately characterize the spatial position information of the hand and the object; the hand vertex 2D coordinates are aligned with the hand deep feature map to obtain the hand posture alignment feature, and the object 2D coordinates are aligned with the object deep feature map to obtain the object posture alignment feature, so as to ensure the consistency of the hand and the object in spatial position and reduce the estimation error; the hand and the object The posture alignment features are used for 3D feature mapping to obtain the 3D spatial features of the hand and the 3D spatial features of the object, respectively, to provide three-dimensional spatial information for capturing the interaction between the hand and the object; the fine-grained interaction between the 3D spatial features of the hand and the 3D spatial features of the object is captured through the cross-attention mechanism to obtain the hand enhancement features and the object enhancement features, which can more accurately characterize the interaction details between the hand and the object; the comparative learning of positive and negative samples is used to enhance the similarity between the hand vertices in the hand enhancement features and the difference between the hand vertices and the environmental voxels to obtain the hand space optimization features, which further improves the representation ability of the hand features and enhances the robustness of the estimation; finally, the object enhancement features and the hand space optimization features are decoded to achieve accurate estimation of the hand-object interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a hand-object estimation method provided by the first embodiment of the present invention; Figure 2 is a process division diagram of a hand-object estimation method in a first embodiment of the present invention; Figure 3 is a diagram illustrating a 2D-3D vertex mapping process in a first embodiment of the present invention; Figure 4 is a diagram of a 3D hand-object cross-attention method in a first embodiment of the present invention; Figure 5 is a diagram of a mesh vertex comparison learning method in a first embodiment of the present invention; Figure 6 It is a structural diagram of a hand-object estimation system provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0019] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0020] Example 1 See also Figure 1 , which is a flow chart of a hand-object estimation method proposed in the first embodiment of the present application, please refer to Figure 2 , which is a process division diagram of a hand-object estimation method in the first embodiment of the present application, and the proposed method steps are as follows.

[0021] Step S01: Acquire a hand-object interaction image.

[0022] Step S02: extracting features from the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively.

[0023] Step S03: Predict the hand joints on the hand deep feature map to obtain the 2D coordinates of the hand vertices; and perform object heat map encoding on the object deep feature map to obtain the object 2D coordinates.

[0024] Step S04: align the hand vertex 2D coordinates with the hand deep feature map to obtain the hand posture alignment feature; align the object 2D coordinates with the object deep feature map to obtain the object posture alignment feature.

[0025] Step S05: Perform 3D feature mapping on the hand posture alignment features and the object posture alignment features to obtain hand 3D spatial features and object 3D spatial features respectively.

[0026] Step S06: According to the cross-attention mechanism, the fine-grained interaction relationship between the hand 3D spatial features and the object 3D spatial features is captured to obtain the hand enhancement features and the object enhancement features.

[0027] It should be noted that, in this embodiment, the process from step S02 to step S06 is a 3D interaction modeling (DIM) process.

[0028] For step S02, in some embodiments, a Restnet50-FPN single- and double-branch hybrid network is used to perform shallow feature extraction on the hand-object interaction image to obtain shallow hand features and shallow object features; a multi-scale backbone is used to perform multi-scale feature extraction on shallow hand features and shallow object features, respectively, to obtain deep hand feature maps and deep object feature maps.

[0029] Specifically, modifications are made to the residual network Restnet50, dividing it into 5 levels: level 0 to level 4, where level 0 and level 1 are still single-branch structures, and the hand and object are taken as a whole foreground for feature learning in the early stage. In the intermediate stage, levels 2 and 3 are dual-branch structures, which learn the hand and object features respectively. Level 4 is a single-branch structure, which unifies the feature space of the hand and object for subsequent interactive operations. Then, two pyramid structures (FPN) are used to continue the dual-branch output. The first branch performs feature fusion operations on the feature maps output by levels 1-4, and the second branch obtains feature maps at different scales to strengthen the feature representation of hands and objects. The entire multi-scale output backbone network outputs deep feature map representations of hands and objects. , , and a set of multi-scale features of the hand and the object respectively. is the deep feature map representation of the hand, It is the deep feature map representation of the object, and H and W are the height and width of the input image respectively.

[0030] For step S03, in some embodiments, the hand deep feature map is subjected to hand feature extraction through an hourglass module, residual learning is introduced through residual connection, and hand features are processed through basic blocks to obtain three different feature vectors, which are a two-dimensional heat map, a 2D joint point coordinates, and a latent feature map.

[0031] Specifically, for the 2D joint coordinates of the hand , the deep hand feature map output by the backbone network , after the hourglass module, which focuses on the extraction of hand features. Subsequently, residual learning is introduced through residual connections to enhance the representation ability of the network. Finally, the features are further processed by the basic blocks and three different feature vectors are output: a two-dimensional heat map , 2D joint point coordinates , here and , and the latent feature map .

[0032] For step S03, in some embodiments, the deep feature map of the object is passed through a convolution block and an activation function to obtain a two-dimensional heat map representation of the object, where the two-dimensional heat map representation of the object includes a dimension height H and a dimension width W; the average is calculated on the dimension height H and the dimension width W to quantize the two-dimensional heat map coordinates and convert them into corresponding one-dimensional vectors respectively; let W and H The tensors of dimensions represent the heat map indexes on the dimension width W and dimension height H respectively; multiply the normalized one-dimensional vector with the corresponding heat map index to obtain the 2D coordinates of the object.

[0033] Specifically, the deep feature map of the object , pass it through a convolution block composed of multiple convolutions, and finally use the corresponding activation function to get a two-dimensional heat map representation of the object .in , represents the minimum number of vertices of the object, B is the training batch, H and W are the height and width of the input image respectively. Specifically, define the two-dimensional pixel projection position coordinates of the object , and The coordinates represent the horizontal and vertical values ​​respectively. , calculate the average of its width and height dimensions, quantize the two-dimensional heat map coordinates, and convert them into two independent one-dimensional vectors , .set up For a W and H A tensor of dimension , representing the heat map index. Multiply the normalized one-dimensional vector with the heat map index to get the 2D coordinate representation of the object. As shown in the formula: in, represents element-wise multiplication, Represents the 2D coordinates of an object.

[0034] For step S04, specifically, after obtaining the 2D coordinates of the hand and the object respectively, 2D posture alignment is applied to the hand-object latent feature maps of different scales respectively, as shown below: in, represents the feature representation after alignment, Represents hand-object feature maps at different scales, Represents the 2D coordinates of a hand or object.

[0035] In step S05, the hand posture alignment feature and the object posture alignment feature are preprocessed, and the preprocessing includes: downsampling the hand posture alignment feature by four times and simplifying the number of hand vertices; and simplifying the number of object vertices of the object posture alignment feature.

[0036] Specifically, Figure 3 As shown, a learnable parameter is introduced to realize the feature conversion from two-dimensional posture space to 3D mesh vertex space. However, for the hand model, directly mapping to a complete set of mesh vertices (such as 778 mesh vertices of the MANO hand model) often faces the problem of large amount of calculation and complex calculation. The present invention uses a downsampling strategy. Specifically, the hand model is downsampled four times, and the original 778 mesh vertices are reduced to 49. For the object model, the minimum mesh vertices are represented as 21. Through experiments, the present invention sets the object mesh vertices to 63. This strategy not only simplifies the problem, but also can design a more compact and efficient mapping matrix M i , mapping 2D features into 3D space. It can be expressed as: in, express j Mesh vertex feature representation in 3D space, express j Feature representation after alignment, j Indicates hand h or object o , is the number of key points, and the number of key points of hands and objects is 21.

[0037] For step S06, in some embodiments, a one-dimensional convolution layer is used to respectively extract query values ​​and feature values ​​of the hand 3D spatial features and the object 3D spatial features; the corresponding attention weighted features are obtained according to the sum of the query values, feature values ​​and feature dimensions; and a one-dimensional convolution layer is used to respectively fuse the hand 3D spatial features and the object 3D spatial features with the corresponding attention weighted features to obtain hand enhancement features and object enhancement features, respectively.

[0038] Specifically, when hands interact with objects, there is a correlation between them. For example, when picking up a pair of scissors or a plate, the hand postures are clearly different. However, previous work often models the interaction relationship between hands and objects based on sparse predefined key points or joints. The present invention uses a cross-attention mechanism to aggregate the fine-grained interaction features between hands and objects and enhance the feature expression of hands and objects. Specifically, Figure 4 As shown, the mesh vertex features of the hand and object in three-dimensional space are and , in the following text and Indicates that a one-dimensional convolutional layer is used to extract their respective query values ​​(query value), feature values ​​(key value) and key values ​​(value value). For the hand model, that is, and , which represents the query, key and value of a specific mesh vertex of the hand. At the same time, the query, key and value of the object model can also be calculated and represented. , and ,in is the sum of the hand or object feature dimensions. For each hand mesh vertex, all object vertex features are aggregated, and each object vertex is similar. The respective attention weights are calculated using the formula and normalized, as shown in the following formula: Here the hand feature query is calculated Key with object features The similarity between them is calculated, and the attention weight of each object feature for the hand feature is obtained through the normalized exponential function. .in, Indicates the degree of attention of each element in the hand feature set to all elements in the object feature set. Similarly, the object feature query is calculated With hand feature key The similarity between them is calculated, and the attention weight of the hand feature for the object feature is obtained. .and Indicates the degree of attention each element in the object feature set pays to all elements in the hand feature set.

[0039] Finally, the original features (i.e., the 3D spatial features of the hand and the 3D spatial features of the object) and the attention weighted features are concatenated together and fused using a one-dimensional convolutional layer, as shown in the following formula: in, Represents the final three-dimensional space i Features, here i Indicates hand features or object features , Indicates the above or , Indicates the above or , Indicates a connection operation. Represents a fusion operation. Using three-dimensional interactive modeling DIM, the model's ability to understand and process the fine-grained interactive relationship between the two in 3D space is enhanced.

[0040] Step S07: using comparative learning of positive and negative samples, the similarity between hand vertices in the hand enhancement feature is enhanced, and the difference between the hand vertices and the environment voxels in the hand enhancement feature is enhanced to obtain the hand space optimization feature.

[0041] It should be noted that, in this embodiment, the process of step S07 is a Vertex Contrastive Learning Network (VCL-Net) process.

[0042] In some embodiments, environmental voxels are used to form an embedded representation of vertex coordinates, where each environmental voxel represents feature information of a local area in a hand enhanced feature to capture similarities and differences between data. For each hand vertex, the positive sample is the predicted hand vertex coordinate value, and the negative sample is the environmental voxel. Based on the hand vertices and environmental voxels, a hand InfoNCE loss function is used to obtain a hand space optimization feature. The hand InfoNCE loss function includes a first contrast loss function between hand vertices and a second contrast loss function between hand vertices and environmental voxels.

[0043] Specifically, the goal of contrastive learning is usually to learn an effective data representation by comparing the similarities and differences between data samples, so that the representation of similar data in the feature space is closer, while the representation of different types of data in the feature space is more dispersed. The contrastive learning process is set using the InfoNCE function, as shown in the formula: in, represents the InfoNCE loss, Represents a logarithmic function The input value of represents the exponential function, represents the aiming point image, for The embedding space representation of represents positive sample embedding, Represented as a negative embedding, is the set of negative samples. All embeddings are normalized. represents the inner product, is the temperature hyperparameter that controls the smoothness of the distribution. The larger it is, the flatter the distribution of the softmax function output is, and the model's ability to distinguish the similarities of different samples is reduced; The smaller it is, the stronger the model's ability to distinguish different sample pairs.

[0044] Molecular part query With positive samples The similarity between the temperature parameters Used to control the smoothness of the similarity distribution, where The function converts the obtained similarity into a non-negative value and amplifies the similarity score of the positive sample. The denominator represents the sum of the similarity score of the positive sample and the similarity score of all negative samples. Overall, the model pays more attention to positive samples during training while minimizing the similarity of negative samples.

[0045] The present invention believes that due to the severe occlusion between the hand and the objects in the multi-object background, using the 3D vertex annotation information of the hand may not provide sufficiently accurate supervision. Specifically, during feature learning, the area occupied by the hand mesh vertices is relatively small, and the vast majority are background elements, such as the objects being operated. These background elements occlude the hand, greatly affecting the learning and representation of hand features. To address this challenge, Figure 5 A mesh vertex contrastive learning method is described, which provides global supervision for the 3D feature information of the hand. For the interaction process between the hand and the object, the representation of the hand and the object in the feature space is more dispersed, and the interference of background elements such as objects on the hand feature learning is reduced to achieve accurate hand pose estimation.

[0046] First, valid keypoints are formed. Anchor indices are randomly selected from a specific set based on the predefined number of anchors, and then the consistency of the anchor index with the vertex mask is determined to obtain the label mask. Based on the mask and the number of instances to be sampled, the index instance of the corresponding instance is obtained. . Let the set represents all possible vertex indices, where is the batch size, is the number of mesh vertices. From the set A subset M is randomly selected from as the set of anchor indexes, where For each anchor index m∈M , there is a corresponding label The instance index solution is shown in the equation. : in, represents the positive instance index set, Represents a set of negative instance indices. Finally, valid key points can be obtained through instance indices in 3D space.

[0047] Secondly, the optimization of hand posture. There is correlation between hand mesh vertices of the same type, and mesh vertex contrast learning is used to strengthen the connection between them, which enables the model to learn better vertex feature representations and thus improve the accuracy of vertex coordinate predictions. In addition, in the 3D feature space, the hand mesh vertices are also compared with other environmental voxels to further strengthen the feature representation by learning differences. After obtaining the 3D feature space of the hand, the environmental voxel embedding is used to form an embedded representation of the vertex coordinates, where each environmental voxel embedding represents the feature information of a local area in the feature space, ensuring that the feature space can fully capture the similarities and differences between the data. For each hand mesh vertex, the positive sample is the predicted mesh vertex coordinate value, and the negative sample is some environmental voxels.

[0048] Based on the key points (i.e., the hand vertices of the present invention) and the 3D embedded representation (i.e., the environment voxels of the present invention), the present invention performs optimization operations in the 3D hand space and uses the hand InfoNCE loss function to obtain more accurate hand features. Specifically, for the hand-environment contrast learning optimization, the optimal parameter settings are selected based on multiple experimental comparisons. The detailed definition of the second contrast loss function is shown in the formula: in, represents the second contrast loss between hand vertices and environment voxels, is the mesh vertex embedding set of positive samples, represents the positive sample, represents negative samples, i.e., environmental voxels, and C represents the logarithmic function The input value of Represents the hand vertex coordinates.

[0049] For hand-hand pose contrast learning, the number of selected anchor points is consistent with that of hand-environment contrast learning. The specific first contrast loss function is defined as shown in the formula: in, represents the first contrast loss between hand vertices, D Represents a logarithmic function The input value of .

[0050] Step S08: Decode the object enhancement features and the hand space optimization features to obtain the hand-object estimation result.

[0051] In this embodiment, it should be noted that the present invention is experimented on a commonly used benchmark.

[0052] For experimental datasets, HO3D v2 is a 3D hand and object dataset consisting of 77K images from 68 video sequences. It includes 10 objects and 10 subjects, with 66,034 training samples and 11,524 evaluation samples. This patent uses the official training set and test set segmentation protocol, and submits the test results to the official website for test evaluation. Dex-YCB is a recently launched large-scale dataset, in which 582K RGB-D frames are recorded from 8 different perspectives for more than 1,000 sequences of 10 objects grasping 20 different objects. It can be applied to multiple tasks such as 2D hand and key point detection, object pose estimation, and 3D hand pose estimation. This patent mainly applies the 3D hand pose estimation problem, and then gives the results according to the official s0 segmentation protocol.

[0053] For the hand posture evaluation index, PA-MPJPE / MPVPE measures the average value of the position error of each mesh vertex through the Euclidean distance (mm) between the hand coordinates estimated by the model and the real coordinates, including alignment and non-alignment analysis. F-Score is the harmonic mean of precision and recall. In the 3D posture estimation task, F-Score can be used to evaluate the similarity between two meshes. At a given distance threshold , i.e. 5mm / 15mm, compare the distance between two mesh vertices. By calculating the F-Score under different thresholds, the performance of the algorithm under different accuracy requirements can be evaluated.

[0054] For the evaluation index of object pose, ADD-0.1D is a common evaluation index in the task of object pose estimation, which is the average mesh vertex error within 10% of the object diameter. Here, ADD-0.1D combines the average mesh vertex error and a fixed ratio (10%) of the object diameter to quantify the similarity between the predicted pose and the true pose of the object.

[0055] For experimental details and parameter settings, the entire model (DIM and DIVCL-Net) of this invention is optimized using the Adam optimizer, with a weight decay of 5e-4 and a batch size of 64. In training, the model is trained for 70 rounds. The initial learning rate is 1e-4, and decay is performed every 10 rounds. For hyperparameters, this paper sets the input resolution size to . All implementations are based on PyTorch.

[0056] Example 2 See also Figure 6 , which is a schematic diagram of the structure of a hand-object estimation system proposed in the second embodiment of the present application, wherein the system comprises: An image acquisition module 100 is used to acquire a hand-object interaction image; The three-dimensional dense modeling module 200 is used to extract features from the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively; predict hand joint points on the hand deep feature map to obtain 2D coordinates of hand vertices; perform object heat map encoding on the object deep feature map to obtain object 2D coordinates; perform posture alignment on the hand vertex 2D coordinates and the hand deep feature map to obtain hand posture alignment features; perform posture alignment on the object 2D coordinates and the object deep feature map to obtain object posture alignment features; perform 3D feature mapping on the hand posture alignment features and the object posture alignment features to obtain hand 3D spatial features and object 3D spatial features respectively; and capture the fine-grained interaction relationship between the hand 3D spatial features and the object 3D spatial features according to the cross-attention mechanism to obtain hand enhancement features and object enhancement features; A vertex contrast learning module 300 is used to enhance the similarity between hand vertices in the hand enhancement feature and enhance the difference between hand vertices and environmental voxels in the hand enhancement feature by contrast learning of positive and negative samples, so as to obtain a hand space optimization feature; The output image module 400 is used to decode the object enhancement features and the hand space optimization features to obtain a hand-object estimation result.

[0057] A hand-object estimation system in an embodiment of the present application may be a device, or a component, integrated circuit, or chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), etc., which is not specifically limited in the embodiment of the present application.

[0058] A hand-object estimation system in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0059] The hand-object estimation system provided in the embodiment of the present application can realize Figure 1 The various processes of implementing a hand-object estimation method in the method embodiment will not be described again here to avoid repetition.

[0060] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned hand-object estimation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0061] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the embodiment of the above-mentioned hand-object estimation method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0062] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0063] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0065] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A hand-object estimation method, characterized in that: include: Acquire hand-object interaction images; Performing feature extraction on the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively; Predicting the hand joints on the hand deep feature map to obtain the 2D coordinates of the hand vertices; encoding the object heat map on the object deep feature map to obtain the 2D coordinates of the object; Performing posture alignment on the hand vertex 2D coordinates and the hand deep feature map to obtain a hand posture alignment feature; Performing posture alignment on the object 2D coordinates and the object deep feature map to obtain an object posture alignment feature; Performing 3D feature mapping on the hand posture alignment feature and the object posture alignment feature to obtain hand 3D spatial features and object 3D spatial features respectively; According to the cross attention mechanism, the fine-grained interaction relationship between the 3D spatial features of the hand and the 3D spatial features of the object is captured to obtain hand enhancement features and object enhancement features; By using contrastive learning of positive and negative samples, the similarity between hand vertices in the hand enhancement feature is enhanced, and the difference between the hand vertices in the hand enhancement feature and the environment voxels is enhanced, so as to obtain a hand space optimization feature; The object enhancement feature and the hand space optimization feature are decoded to obtain a hand-object estimation result.

2. A hand-object estimation method according to claim 1, characterized in that: The steps of extracting features from the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively include: Using a Restnet50-FPN single- and double-branch hybrid network to extract shallow features of the hand-object interaction image, to obtain shallow features of the hand and shallow features of the object; The multi-scale backbone is used to perform multi-scale feature extraction on the shallow features of the hand and the shallow features of the object respectively, to obtain a deep feature map of the hand and a deep feature map of the object.

3. A hand-object estimation method according to claim 1, characterized in that: The step of predicting the hand joint points on the hand deep feature map to obtain the 2D coordinates of the hand vertices includes: The hand deep feature map is used to extract hand features through the hourglass module, residual learning is introduced through residual connection, and the hand features are processed through basic blocks to obtain three different feature vectors, which are a two-dimensional heat map, a 2D joint point coordinates, and a latent feature map.

4. The hand-object estimation method according to claim 1, characterized in that: The step of performing object heat map encoding on the object deep feature map to obtain the object 2D coordinates includes: The deep feature map of the object is passed through a convolution block and an activation function to obtain a two-dimensional heat map representation of the object, wherein the two-dimensional heat map representation of the object includes a dimension height H and a dimension width W; Calculate the average value on the dimension height H and the dimension width W to quantize the two-dimensional heat map coordinates and convert them into corresponding one-dimensional vectors respectively; Let W and H The tensors of dimensions represent the heat map indexes on the dimension width W and dimension height H respectively; The normalized one-dimensional vector is multiplied by the corresponding heat map index to obtain the object 2D coordinates.

5. The hand-object estimation method according to claim 1, characterized in that: Before the step of performing 3D feature mapping on the hand posture alignment feature and the object posture alignment feature, the method further includes: Preprocessing the hand posture alignment features and the object posture alignment features includes: Downsampling the hand pose alignment feature by a factor of four and reducing the number of hand vertices; The number of object vertices of the object pose alignment feature is reduced.

6. A hand-object estimation method according to claim 1, characterized in that: The steps of capturing the fine-grained interaction relationship between the hand 3D spatial features and the object 3D spatial features according to the cross attention mechanism to obtain the hand enhancement features and the object enhancement features include: Use a one-dimensional convolutional layer to extract the query value and feature value of the hand 3D spatial features and the object 3D spatial features respectively; Obtaining a corresponding attention weighted feature according to the query value, the feature value and the sum of the feature dimensions; A one-dimensional convolutional layer is used to fuse the hand 3D spatial features and the object 3D spatial features with the corresponding attention weighted features to obtain hand enhancement features and object enhancement features, respectively.

7. The hand-object estimation method according to claim 1, characterized in that: The steps of enhancing the similarity between hand vertices in the hand enhancement feature and enhancing the difference between the hand vertices and the environment voxels in the hand enhancement feature by using the contrast learning of positive and negative samples to obtain the hand space optimization feature include: Use environmental voxels to form an embedded representation of vertex coordinates, where each environmental voxel represents the feature information of a local area in the hand enhancement feature to capture the similarities and differences between the data. For each hand vertex, the positive sample is the predicted hand vertex coordinate value, and the negative sample is the environmental voxel. Based on the hand vertices and the environment voxels, a hand space optimization feature is obtained using a hand InfoNCE loss function, where the hand InfoNCE loss function includes a first contrast loss function between hand vertices and a second contrast loss function between hand vertices and environment voxels.

8. A hand-object estimation system, characterized in that: The system comprises: An image acquisition module is used to acquire a hand-object interaction image; A three-dimensional dense modeling module is used to extract features from the hand-object interaction image to obtain a hand deep feature map and an object deep feature map respectively; predict hand joint points on the hand deep feature map to obtain 2D coordinates of hand vertices; perform object heat map encoding on the object deep feature map to obtain object 2D coordinates; perform posture alignment on the hand vertex 2D coordinates and the hand deep feature map to obtain hand posture alignment features; perform posture alignment on the object 2D coordinates and the object deep feature map to obtain object posture alignment features; perform 3D feature mapping on the hand posture alignment features and the object posture alignment features to obtain hand 3D spatial features and object 3D spatial features respectively; and capture the fine-grained interaction relationship between the hand 3D spatial features and the object 3D spatial features according to the cross-attention mechanism to obtain hand enhancement features and object enhancement features; A vertex contrast learning module, used for enhancing the similarity between hand vertices in the hand enhancement feature and enhancing the difference between hand vertices and environmental voxels in the hand enhancement feature by contrast learning of positive and negative samples, so as to obtain a hand space optimization feature; The output image module is used to decode the object enhancement features and the hand space optimization features to obtain a hand-object estimation result.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of a hand-object estimation method as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of a hand-object estimation method as described in any one of claims 1-7 are implemented.

Images