3D human-object interaction reconstruction method based on multi-scale correction and frequency domain adaptive graph convolution

Abstract

This invention discloses a 3D human-object interaction reconstruction method based on multi-scale correction and frequency-domain adaptive graph convolution, belonging to the field of computer vision and 3D reconstruction technology. By extracting features from the input data, a multi-scale correction mechanism is used to capture human-object interaction features at different scales, optimizing the completeness of feature representation. A frequency-domain adaptive graph convolution module is introduced to adaptively learn frequency-domain feature dependencies, effectively suppressing high-frequency noise interference. Through feature fusion and reconstruction, an accurate 3D human-object interaction model is output. This invention, through the synergistic effect of multi-scale correction and frequency-domain adaptive graph convolution, improves the reconstruction stability and geometric accuracy in complex interaction scenarios, and can be widely applied in virtual reality, augmented reality, film and television production, etc., possessing good practicality and promotional value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, specifically to a 3D human-object interaction reconstruction method based on multi-scale correction and frequency domain adaptive graph convolution. Background Technology

[0002] As a key bridge connecting two-dimensional visual perception and three-dimensional spatial modeling, 3D human-object interaction reconstruction technology can recover the three-dimensional form and interaction relationship of the human body and objects from a single RGB image, and has irreplaceable application value in fields such as intelligent interaction, virtual try-on, and film and television production. With the deep penetration of deep learning in the field of computer vision, single-image 3D reconstruction technology has made significant progress. However, due to the inherent lack of depth information in two-dimensional images and the complexity of occlusion in interactive scenes, how to achieve fine feature fusion and physical consistency reconstruction of the human body and objects remains the core bottleneck restricting the practical application of the technology.

[0003] Existing single-image 3D human-object interaction reconstruction methods generally suffer from problems such as insufficient feature representation and coarse interaction relationship modeling. Traditional methods often use single-scale features for parameter regression, which makes it difficult to accurately model both the global human posture and local interaction details. At the same time, the feature learning of the human body and the object is independent of each other, lacking in-depth exploration of the interaction correlation between the two. This leads to problems such as contact area penetration, object shape distortion, and poor adaptation between human posture and object in the reconstruction results, which cannot meet the needs of high-precision application scenarios.

[0004] Chinese patent (publication number: CN110728219B) proposes a 3D face generation method based on a multi-column, multi-scale graph convolutional neural network. This patent addresses the problems of single feature scale and insufficient detail representation in traditional 3D reconstruction by innovatively employing a multi-scale graph convolutional encoder architecture. It extracts geometric features at different scales through a multi-column network and selectively fuses them, while introducing a self-attention mechanism to enhance the feature representation of key regions, effectively improving the accuracy of 3D morphological detail reconstruction. However, this technology still has significant shortcomings: its core focuses on generating a single target—the face—without involving modeling the interaction relationship between the human body and objects, lacking a dedicated object context feature extraction mechanism, and failing to capture the spatial constraints and interactive relationships between the human body and objects; furthermore, it only performs feature aggregation in the spatial domain, without considering the impact of frequency domain noise on reconstruction accuracy, making it prone to feature distortion in complex interactive scenarios.

[0005] To address the limitations of insufficient multi-scale feature modeling and single-target reconstruction, Chinese patent (publication number: CN109242950A) proposes a multi-view dynamic 3D reconstruction method for human bodies in close-interaction scenarios involving multiple users. This patent addresses the occlusion problem in such scenarios by using multi-view video acquisition and spatiotemporal joint pose tracking to achieve dynamic reconstruction of the human body's 3D model under close interaction, thus overcoming the limitations of single-target reconstruction to some extent and strengthening the correlation modeling between multiple targets. However, this technology still has shortcomings: it relies on multi-view image input, resulting in high equipment deployment costs and making it unsuitable for the lightweight requirements of single-image reconstruction; furthermore, it does not employ a feature optimization architecture combining graph convolution and Transformer, lacking refined processing of mixed features of human bodies and objects, leading to poor feature fusion in interactive areas and difficulty in achieving high-precision collaborative reconstruction of human bodies and objects under single-image input.

[0006] Therefore, there is an urgent need to achieve accurate reconstruction of the three-dimensional form and interaction relationship between the human body and objects, and to provide a more efficient and universal technical solution for single-image 3D human-object interaction reconstruction. Summary of the Invention

[0007] To address the aforementioned technical problems, this application discloses a 3D human-object interaction reconstruction method based on multi-scale correction and frequency-domain adaptive graph convolution, including:

[0008] Acquire a single RGB image and the target object ID, extract image features through a backbone network, predict the initial parameters of the human body and the object based on the initial regressor, and generate the initial three-dimensional vertices of the human body and the object by combining the parameterized model and the object template.

[0009] An inverted feature pyramid is constructed, the initial 3D vertices are projected onto the feature map and the features are sampled and fused. The parameter residuals are predicted through a fully connected network with a soft truncation mechanism to correct the initial parameters.

[0010] Based on the corrected parameters, a hybrid token sequence is constructed, the body context vector is extracted, and the hybrid token sequence is feature-optimized by a frequency domain adaptive graph convolution Transformer.

[0011] By outputting the coordinates of 3D joint points and the coordinates of human and object mesh vertices through the regression head, 3D human-object interaction reconstruction is completed.

[0012] Preferably, the process of constructing the inverted feature pyramid and performing feature sampling fusion specifically includes:

[0013] Backbone network outputs deep features First, use a single-channel convolutional layer to... Channel compression is performed to obtain basic features. The feature pyramid is formed by sequentially generating feature maps of different resolutions through three cascaded transposed convolutional layers. The resolution of each feature map is increased step by step according to a preset ratio;

[0014] For the initial three-dimensional vertices of the human body and the three-dimensional vertices of the object any vertex in Using the camera intrinsic parameter matrix and external references ( , Calculate its two-dimensional projection coordinates on each feature map. ,in For the camera rotation matrix, Let be the camera translation vector, calculated using the following formula: , , ;in , , As vertex The projected homogeneous coordinate components are interpolated using a bilinear interpolation function. Extract the feature vector of each projected coordinate on the corresponding feature map, and concatenate and fuse features of different scales to obtain the multi-scale fused features of the vertex. ,in Using two-dimensional projected coordinates, the final result is the aggregation of all vertex features to form a global fused feature. .

[0015] Preferably, the parameter residual prediction and parameter correction process of the soft truncation mechanism is as follows:

[0016] Global fusion features Inputting the fully connected network, it predicts the parameter residuals for human pose, shape, and object pose, respectively. The prediction formula is as follows: , , , ,in For human posture parameter residuals, For human body shape parameter residuals, The residual of the object's rotation matrix. The residual of the object's translation vector. , , , These are the preset scaling factors, and tanh(·) is the hyperbolic tangent activation function, which compresses the output to the (-1,1) interval to ensure that the correction amount is limited. Within the range;

[0017] The initial parameters are corrected based on the predicted parameter residuals, and the corrected parameters are: , , , ,in , These are the initial posture and shape parameters of the human body. , Let the initial rotation matrix and translation vector of the object be... , To correct the human body posture and shape parameters, , These are the rotation matrix and translation vector of the corrected object.

[0018] Preferably, the process of constructing the hybrid token sequence includes:

[0019] Based on the corrected parameters , Generate corrected 3D human body vertices ,based on , Generate corrected 3D vertices of the object Extract human joint features respectively Human Vertebrae and object vertex features Through a fully connected layer , , Mapped to a unified dimension, forming a hybrid sequence containing both geometric and semantic information. This sequence simultaneously covers key interaction features between the human body and objects, providing a data foundation for subsequent global interaction perception.

[0020] Preferably, the process of extracting and applying the object context vector is as follows:

[0021] Vertex features of objects in a mixed token sequence Perform global average pooling to extract the entity context vector. The extraction formula is: ,in The total number of vertices of the object. For the first Feature vectors of the vertices of an object; It represents the global state of an object, including its spatial location, category attributes, and geometric features.

[0022] Preferably, the frequency domain adaptive graph residual block processing process in the frequency domain adaptive graph convolution Transformer includes:

[0023] Let the features of the input graph nodes be... , The matrix is ​​composed of the feature vectors of all nodes in the graph. The features are transformed from the spatial domain to the frequency domain by a one-dimensional discrete cosine transform (DCT) to obtain the spectral coefficients. ,in The transformation matrix is ​​a DCT; a learnable gated mask is used. Adaptive frequency-selective filtering is applied to the spectral coefficients, and the filtering formula is as follows: ,in This represents element-wise multiplication. The Sigmoid activation function is used to automatically learn weights through training, suppressing high-frequency noise components and preserving low-frequency components that characterize the smoothness of object or human body structures.

[0024] The filtered spectral coefficients The features are recovered to the spatial domain by inverse discrete cosine transform (IDCT) and smoothed features are obtained. ,in Let A be the transpose of the DCT transformation matrix; let the object context vector be... and Perform feature splicing to obtain Topological neighborhood information is aggregated through graph convolution operations. The graph convolution update formula is: ,in Let be the adjacency matrix of the graph. Let be the degree matrix of the graph. Let be the learnable weight parameters of the graph convolutional layer, and ReLU(·) be the linear rectified activation function.

[0025] Preferably, in the graph convolution operation, the construction process of the adjacency matrix A and the degree matrix D is as follows:

[0026] The adjacency matrix A is dynamically constructed based on the geometric distance between nodes. For any two nodes i and j in the graph, the Euclidean distance between them is calculated. ,in , They are nodes , 3D coordinates; preset distance threshold ,when At that time, determine the node and There are topological connections, and the corresponding elements in the adjacency matrix A are... =1; when When the two are determined to have no topological connection, =0, and =0, the node itself has no connections;

[0027] Degree matrix It is a diagonal matrix, and its diagonal elements ,in Let be the total number of nodes in the graph. This represents the sum of the elements in the i-th row of the adjacency matrix A, which is the degree of connection of node i.

[0028] Preferably, the human interaction-aware branch included in the frequency-domain adaptive graph convolution Transformer is specifically: the branch adopts interaction-aware adaptive frequency-domain graph convolution, and the features of the human branch are extracted as context vectors. And inject it into human characteristics Gated networks; using graph filtering principles to... Decoupling is achieved through low-frequency smoothing features and high-frequency sharpening features. The low-frequency path aggregates features using an adjacency matrix A, as shown in the formula: ,in The low-frequency pathway uses learnable weights; the high-frequency pathway extracts high-frequency details by subtracting the smoothing component from the original features, as shown in the formula: ,in These are the learnable weights for high-frequency pathways;

[0029] Using object context vectors Generate adaptive gating coefficients ,in It is the Sigmoid activation function. Indicates human body characteristics With object context vector splicing It is a multilayer perceptron; through gating coefficients Dynamically fuse low-frequency smoothing features and high-frequency sharpening features to output fused features. When the human body mesh is close to an object, Adaptive reduction, preserving more high-frequency contact details, when moving away from the object It adaptively enlarges to maintain the smoothness of the human body surface.

[0030] Preferably, the object rigidity constraint branch included in the frequency domain adaptive graph convolution Transformer specifically employs a hybrid modeling approach of local smoothing and global rigidity to adapt to the rigid structural characteristics of the object, utilizing a dynamically constructed sparse graph structure of the object. Perform feature smoothing to eliminate reconstruction noise. The dynamic adjacency matrix of the object nodes; local smoothing features are expressed by the formula. Calculation, where The feature matrix of the object nodes, These are the learnable weights for local smoothing pathways;

[0031] The global mean feature is obtained by averaging the features of all nodes of the object. The features are then input into a multilayer perceptron (MLP) for feature transformation, and then the global features are expanded to the same level using a broadcast operation. With the same dimension, global rigid features are obtained. Features enforce the consistency of the overall structure of an object; the network, through... Automatically learn rigid consistency gating coefficients ,in Use the Sigmoid activation function; outputs object features. The formula is: .

[0032] Preferably, the iterative optimization of the frequency domain adaptive graph convolution Transformer is as follows: a staged encoder structure is adopted, the number of feature channels is gradually reduced according to a preset ratio, simulating the cognitive process from global semantic understanding to local geometric refinement, and each stage encoder focuses on feature learning at different scales;

[0033] Input mixed token sequence Global interaction features are extracted using a multi-head self-attention mechanism to capture long-distance dependencies between the human body and objects, different parts of the human body, and different parts of objects. The resulting sequence is then decomposed into human tokens and object tokens, which are fed into the human interaction perception branch and the object rigidity constraint branch respectively for differentiated processing. The human token focuses on optimizing interaction details, while the object token focuses on preserving rigid structures. The processed human and object features are then concatenated and fused before being fed into the next encoder stage, forming a closed-loop iterative optimization process. Each iteration optimizes the feature representation based on the results of the previous iteration. After multiple alternating processing layers, the 3D joint coordinates are directly predicted by the regression head. And the coordinates of the vertices of the human body and object mesh, where It is a set of three-dimensional spatial coordinates of key human joints, and outputs the 3D human-object interaction reconstruction results.

[0034] Compared with the prior art, the technical solution of this application has the following technical effects:

[0035] The construction of the hybrid token sequence in this invention effectively breaks through the limitations of independent modeling of human and object features. By integrating the geometric and semantic features of human joints, human vertices and object vertices, it achieves a unified representation of the key interactive information of the two. The sequence covers human posture details, object morphological features and spatial correlation information between the two, providing a comprehensive data foundation for interactive perception modeling and improving the completeness and relevance of feature expression.

[0036] The extraction and application of object context vectors in this invention constructs a dynamic guidance mechanism for human beings and objects, so that the updating of human features is no longer limited to its own topological neighborhood information, but is subject to real-time control of the global state of the object. The interactive perception design allows the human body mesh to accurately fit the shape of the object in the contact area, effectively avoiding non-physical phenomena such as suspension or clipping, and improving the physical consistency and visual realism of human-object interaction reconstruction.

[0037] This invention, Frequency Domain Adaptive Graph Convolution Transformer, innovatively integrates the advantages of local structure modeling of graph convolution with the global interactive perception capability of Transformer. Through frequency domain filtering mechanism, it adaptively suppresses high-frequency noise, retains effective features, enhances the smoothness of human body surface and the rigidity of object structure, captures fine-grained details of contact area, and achieves dual optimization of global structural stability and local detail refinement.

[0038] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings.

[0039] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0041] Based on the description of the figures and their corresponding technical content in the document, the titles of the figures are as follows:

[0042] Figure 1 : Overall flowchart of a 3D human-object interaction reconstruction method based on multi-scale correction and frequency domain adaptive graph convolution;

[0043] Figure 2 : Transformer architecture diagram for multi-scale feature generation, grid sampling and feature projection, and frequency domain adaptive graph convolution;

[0044] Figure 3Schematic diagram of the structure of human branch interaction perception adaptive frequency domain graph convolutional residual block and dynamic correction controller;

[0045] Figure 4 : Schematic diagram of object branch rigid constraint graph convolution residual block and local smoothing and global rigidity fusion structure;

[0046] Figure 5 The diagram illustrates the existing problems of suspension and clipping, and the contact reconstruction optimized by this invention.

[0047] Figure 6 : A schematic diagram of the multi-view visualization and reconstruction results of the manipulated object scene;

[0048] Figure 7 A schematic diagram comparing the 3D reconstruction effects of the present invention's MRG-HOI method with the existing advanced technology HOI-TG in complex human-object interaction scenarios. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the following description, specific details such as specific configurations and components are provided merely to help fully understand the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. In addition, for clarity and brevity, descriptions of known functions and structures are omitted in the embodiments.

[0050] It should be understood that the phrase "an embodiment" or "this embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "an embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0051] Furthermore, reference numerals and / or letters may be repeated in different examples within this application. Such repetition is for the purpose of simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0052] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" in this article describes another type of relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the related objects before and after it are in an "or" relationship.

[0053] In this article, the term "at least one" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, "at least one of A and B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0054] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion.

[0055] Example 1

[0056] This embodiment mainly describes a refined 3D human-object interaction reconstruction method based on multi-scale parameter correction and frequency domain adaptive graph convolution Transformer, such as... Figure 1 As shown, it specifically includes:

[0057] Acquire a single RGB image and the target object ID, extract image features through a backbone network, predict the initial parameters of the human body and the object based on the initial regressor, and generate the initial three-dimensional vertices of the human body and the object by combining the parameterized model and the object template.

[0058] An inverted feature pyramid is constructed, the initial 3D vertices are projected onto the feature map and the features are sampled and fused. The parameter residuals are predicted through a fully connected network with a soft truncation mechanism to correct the initial parameters.

[0059] Based on the corrected parameters, a hybrid token sequence is constructed, the body context vector is extracted, and the hybrid token sequence is feature-optimized by a frequency domain adaptive graph convolution Transformer.

[0060] The final 3D joint coordinates and the vertex coordinates of the human and object meshes are output by the regression head to complete the 3D human-object interaction reconstruction.

[0061] Furthermore, the input is a single RGB image. (in Image width, Image features are extracted using a backbone network, taking the image height and the target object ID as inputs. (in The number of feature channels, , Let the width and height of the feature map be (represented by the initial regressor), and let the initial parameters of the human body and object be predicted. Let the initial human pose parameters be... The initial human body shape parameters are The initial object rotation matrix is The initial object translation vector is .

[0062] Based on SMPL-H parameter model Generate the initial three-dimensional vertices of the human body. ,in The number of vertices in the human body is given by the following formula: Similarly, the initial 3D vertices of the object are generated using an object template. , where N_O is the number of vertices of the object.

[0063] Furthermore, Multi-Scale Parameter Correction (MSPR) is implemented by constructing an inverted feature pyramid and refining the features extracted by the backbone network. Through single-channel convolutional layers Channel compression is performed to obtain basic features. The channel compression formula is: Subsequently, a feature pyramid is formed by sequentially generating feature maps of different resolutions through three cascaded transposed convolutional layers (Deconvolution). The resolution of each feature map is increased progressively according to a preset ratio, with the output feature maps of the three-level transposed convolutional layers being as follows: , , ,in, Relative to Feature maps with 8x improved resolution. Low-resolution features. Contains strong semantic information for correcting global rotation; high-resolution features Includes high-frequency textures for correcting local contacts; the feature pyramid construction process corresponds to... Figure 2 The core functionality of the MultiScaleFeatGenraterV2 module.

[0064] Projection-based feature sampling for initial human 3D vertices and the three-dimensional vertices of the object any vertex in Using the camera intrinsic parameter matrix and external references ( , Calculate its two-dimensional projection coordinates on each feature map. ,in This is the camera rotation matrix, used to characterize the camera's spatial rotation state. Let be the camera translation vector, used to characterize the spatial position offset of the camera. The projection calculation formula is: , , ,in , , As vertex Homogeneous coordinate components after projection.

[0065] For each layer of feature map in the pyramid Using bilinear interpolation function Extract the feature vectors at the corresponding coordinates, and concatenate and fuse features from different scales to obtain the multi-scale fused features of the vertices. ,in It uses two-dimensional projected coordinates; it aggregates the features of all human and object vertices to form a global fused feature vector. ;correspond Figure 2 The collaborative workflow of the Grid Sampling and Feature Projection modules enables geometric alignment and feature fusion between 3D vertices and 2D feature maps.

[0066] Furthermore, for soft-truncation residual prediction and parameter correction, to prevent excessive correction from causing the collapse of human anatomical structures, a soft-truncation mechanism based on the hyperbolic tangent function (Tanh) is introduced to incorporate globally fused features. Inputting the fully connected network (MLP), it predicts the parameter residuals for human pose, shape, and object pose, respectively. The prediction formula is as follows:

[0067]

[0068]

[0069]

[0070]

[0071] For human posture parameter residuals, For human body shape parameter residuals, The residual of the object's rotation matrix. The residual of the object's translation vector. , , , These are the preset scaling factors, and tanh(·) is the hyperbolic tangent activation function, which compresses the output to the (-1,1) interval to ensure that the correction amount is limited. Within the range, the continuity of the gradient at the boundary and the stability of backpropagation are guaranteed.

[0072] The initial parameters are corrected based on the predicted parameter residuals, and the corrected parameters are: , , , ,in , These are the initial posture and shape parameters of the human body. , Let the initial rotation matrix and translation vector of the object be... , To correct the human body posture and shape parameters, , The corrected object's rotation matrix and translation vector; the correction process corresponds to... Figure 3 The Dynamic CorrectionControler module in the middle class solves the problem of accurate global pose but misaligned local details in the initial reconstruction by finely adjusting the parameter space.

[0073] Furthermore, frequency-domain adaptive graph convolutional Transformer (TGformer) optimization, for hybrid token construction and object context extraction, is based on the corrected parameters. , Generate corrected 3D human body vertices ,based on , Generate corrected 3D vertices of the object Extract human joint features separately Human Vertebrae and object vertex features ,in This is the set of feature vectors for human joints. Depend on The geometric coordinate information is encoded to obtain, Depend on The geometric coordinate information is encoded; it is then processed through a fully connected layer. , , Mapped to a unified dimension, forming a hybrid sequence containing both geometric and semantic information. This sequence simultaneously covers key interaction features between the human body and objects, providing a data foundation for subsequent global interaction perception.

[0074] To explicitly model the guiding effect of objects on human pose, the vertex features of objects in the mixed token sequence are analyzed. Perform global average pooling to extract the entity context vector. The extraction formula is: ,in The total number of vertices of the object. For the first Feature vectors of the vertices of an object; It represents the global state of an object, including its spatial location, category attributes, and geometric features, which will be injected as conditions into the human feature update process.

[0075] Frequency domain adaptive graph residual block processing, assuming the input graph node features are... ( The matrix consisting of the feature vectors of all nodes in the graph is used to transform the features from the spatial domain to the frequency domain through a one-dimensional discrete cosine transform (DCT) to obtain the spectral coefficients. ,in This is a DCT transform matrix used to convert spatial domain features to frequency domain features; it utilizes a learnable gated mask. Adaptive frequency-selective filtering is applied to the spectral coefficients, and the filtering formula is as follows: , where ⨀ represents element-wise multiplication, σ(·) is the Sigmoid activation function, ensuring that the weight values ​​are between (0,1). The network automatically learns the weights through training, suppressing high-frequency noise components and retaining low-frequency components that characterize the smoothness of objects or human structures.

[0076] The filtered spectral coefficients The smoothed features are obtained by restoring the spatial domain through inverse discrete cosine transform (IDCT). ,in Let A be the transpose of the DCT transformation matrix; let the object context vector be... and Perform feature splicing to obtain Topological neighborhood information is aggregated through graph convolution operations. The graph convolution update formula is as follows: ,in Let be the adjacency matrix of the graph, used to represent the topological connections between nodes. Let be the degree matrix of the graph. The learnable weight parameters for the graph convolutional layer are defined by ReLU(·), a linear rectified activation function. A residual connection X is added at the end to prevent the vanishing gradient problem during network training. This frequency-domain adaptive graph residual block is... Figure 2 The core component of the Graph Residual Block enables frequency domain noise reduction and spatial domain topology aggregation of features.

[0077] The adjacency matrix and degree matrix are constructed as follows: the adjacency matrix A is dynamically constructed based on the geometric distance between nodes. For any two nodes in the graph... and Calculate the Euclidean distance between the two. ,in , They are nodes , 3D coordinates; preset distance threshold ,when When determining that nodes i and j have a topological connection, the corresponding elements in the adjacency matrix A are... =1; when When the two are determined to have no topological connection, =0, and =0 (the node itself has no connections);

[0078] The degree matrix D is a diagonal matrix, and its diagonal elements ,in Let be the total number of nodes in the graph. Let represent the sum of the elements in the i-th row of the adjacency matrix A, i.e., the connectivity degree of node i. The adjacency matrix A and degree matrix D constructed in this way accurately reflect the geometric relationships between nodes, providing a topological foundation that fits the physical structure for graph convolution operations. This construction logic permeates... Figure 4 The process of graph convolution operation between human body branches and object branches.

[0079] The human interaction perception branch is processed using interactive perception adaptive frequency domain graph convolution (IA-AFS-GCN), corresponding to... Figure 3 The architecture of Human Branch: Interaction-Aware AFS-GCN Residual Block extracts the features of human branches as context vectors. And inject it into human characteristics ( A gated network (where the feature matrix of the human node is used);

[0080] Using graph filtering principles Decoupling is achieved through low-frequency smoothing features and high-frequency sharpening features. The low-frequency path aggregates features using the adjacency matrix A, as shown in the formula. ,in The low-frequency pathway uses learnable weights to capture the continuous structural morphology of the human body surface; the high-frequency pathway extracts high-frequency details by subtracting the smoothing component from the original features, as shown in the formula. ,in Learnable weights for high-frequency pathways are used to capture details of local abrupt changes such as joint bending and object contact.

[0081] Using object context vectors Generate adaptive gating coefficients ,in It is the Sigmoid activation function. Indicates human body characteristics With object context vector The MLP (Multilayer Perceptron) is used to learn the mapping relationship of gating coefficients.

[0082] By gating coefficient Dynamically fuse low-frequency smoothing features and high-frequency sharpening features to output fused features. When the human body mesh is close to an object, Adaptive reduction, preserving more high-frequency contact details, when moving away from the object Adaptively enlarges to maintain the smoothness of the human body surface. This branch... Figure 3 The synergistic effect of layers such as BertLayerNorm and GraphLinear enables refined interactive perception of human features.

[0083] The rigidity constraint branch of the object is handled by a hybrid modeling approach that combines local smoothing and global rigidity. Figure 4 The Object Branch: Rigid-Constraint GCN Residual Block architecture is designed to adapt to the rigid structural characteristics of objects. It first utilizes a dynamically constructed sparse graph structure of objects. ( The dynamic adjacency matrix of the object nodes (constructed based on the geometric distance between the object nodes) is used for feature smoothing to eliminate reconstruction noise;

[0084] Local smoothing features are expressed by formula The calculation yielded, where The feature matrix of the object nodes, Learnable weights for local smoothing paths, used to preserve local details of the object's surface;

[0085] The global mean feature is obtained by averaging the features of all nodes of the object. The features are then input into a multilayer perceptron (MLP) for feature transformation, and then the global features are expanded to the same level using a broadcast operation. With the same dimension, global rigid features are obtained. This feature enforces the consistency of the overall structure of the object;

[0086] Network through Automatically learn rigid consistency gating coefficients ,in This is the Sigmoid activation function, used to balance local details with global rigidity;

[0087] Final output object features ,when When the value approaches 1, object feature updates rely more on the global mean, effectively suppressing non-physical surface depressions, distortions, and other deformities; when... As the value approaches 0, the focus shifts to preserving local details. This branch... Figure 4 The combination of Mean Pooling and Consistency Gate modules enables rigid constraint optimization of object features.

[0088] Furthermore, the frequency-domain adaptive graph convolutional Transformer employs a staged encoder structure, where the number of feature channels decreases progressively according to a preset ratio. This simulates the cognitive process from global semantic understanding to local geometric refinement, with each stage encoder focusing on feature learning at different scales. Figure 2 The hierarchical architecture design of TGformer;

[0089] Input mixed token sequence Global interaction features are extracted through a multi-head self-attention mechanism to capture long-distance dependencies between the human body and objects, different parts of the human body, and different parts of objects.

[0090] The sequence after interaction is then decomposed into human tokens and object tokens, which are then fed into the human interaction perception branch and the object rigid constraint branch for differentiated processing. The human token focuses on optimizing interaction details, while the object token focuses on maintaining rigid structure.

[0091] The processed human and object features are spliced ​​and fused again and sent to the next level encoder to form a closed-loop iterative optimization process. Each iteration optimizes the feature expression based on the results of the previous iteration.

[0092] After multiple alternating processing steps, the 3D joint coordinates J_{3D} are directly predicted using the regression head. This involves using a set of 3D spatial coordinates of key human joints and the vertex coordinates of human and object meshes to achieve a refined 3D human-object interaction reconstruction that meets physical consistency and geometric accuracy requirements. The entire iterative optimization process is underway. Figure 2 A closed loop is formed in the overall pipeline, realizing a gradual improvement from initial coarse reconstruction to fine reconstruction.

[0093] This detailed description of the implementation optimizes the feature representation and noise suppression in the 3D human-object interaction reconstruction process by combining multi-scale correction with frequency domain adaptive graph convolution. It improves the geometric consistency and morphological stability of the reconstruction model in complex interaction scenarios, avoids common artifacts such as local structural jitter and abnormal joint morphology, and improves the reconstruction results in both global structural integrity and local detail accuracy.

[0094] Based on Example 1, this example comprehensively verifies the technical advantages of the MRG-HOI network of this invention in monocular 3D human-object interaction reconstruction tasks. Systematic quantitative and qualitative verification was conducted on the publicly available BEHAVE dataset. All experiments were based on a unified hardware environment and software framework to ensure the objectivity and comparability of the evaluation results. For quantitative evaluation, ChamferDistance (CD) was selected as the core indicator of geometric reconstruction accuracy. This indicator calculates the average Euclidean distance between the reconstructed mesh and the real mesh vertices, in millimeters; a lower value indicates a smaller geometric error. Simultaneously, ContactPrecision and ContactRecall were selected to evaluate the quality of interaction detail recovery. Both are dimensionless indicators in the [0,1] interval; values ​​closer to 1 indicate more accurate capture of human-object contact relationships. The comparison objects include the classic optimization-based method Phosa, the implicit neural field-based Chore, the initialization-refinement pipeline design CONTHO, and the current state-of-the-art HOI-TG model. All models used the same input data format (256×256 resolution RGB images) and evaluation process.

[0095] The quantitative evaluation results are summarized in the table below, clearly demonstrating the significant advantages of this invention in various core indicators. METRO and Graphormer, as early foundational methods, have significant shortcomings in geometric reconstruction accuracy and contact modeling capabilities, with METRO's CD... human Reaching 46.82, CD object The value is 59.13, Contact p With a value of only 0.084, it lags significantly behind existing state-of-the-art methods; Graphormer's CD human It dropped to 42.40, but Contact p With a score of only 0.047, the contact relationship capture effect is poor. PHOSA uses a physics-based optimization path to improve the CD... human Optimized to version 12.17, CD object Dropped to 26.62, Contact p Improved to 0.393, but limited by preset physical priors, Contact rWith a recall of only 0.266, its contact recall performance is poor. CHORE employs unsigned distance field implicit representation and implicit learning methods, CD human Further decreased to 5.58, CD object 10.66, Contact p With Contact r These figures were 0.587 and 0.472 respectively, representing a significant performance improvement; CONTHO, with its initialization-refinement pipeline design, CD human Optimized to version 4.99, CD object 8.42, Contact p With Contact r The accuracy and contact modeling performance have been improved to 0.628 and 0.496, respectively, further optimizing geometric accuracy and contact modeling performance. HOI-TG, as a leading model currently available, [is used in conjunction with CD / DVD]. human Reaching 4.59, CD object 8.00, Contact p With Contact r The values ​​are 0.662 and 0.554 respectively, demonstrating outstanding performance in global interactive capture. This invention, through the innovative fusion of multi-scale parameter correction and frequency-domain adaptive graph convolution Transformer, achieves high performance in CD. human Further optimized to 4.57, a 0.44% reduction compared to HOI-TG; CD object The accuracy dropped to 7.09, a decrease of 11.38% compared to HOI-TG's 8.00, indicating a significant improvement in geometric reconstruction accuracy; Contact p Improved to 0.682, a 3.02% increase compared to HOI-TG; Contact r The accuracy reached 0.589, a 6.32% improvement over HOI-TG's 0.554, representing a further breakthrough in the accurate capture of contact relationships and fully demonstrating the dual technical advantages of this invention in geometric accuracy and interactive detail modeling.

[0096] Table 1: Qualitative experimental results on the BEHAVE dataset.

[0097] In the qualitative verification, the solution to the core pain points is first presented intuitively through a detailed schematic diagram of the contact reconstruction principle, such as... Figure 5 As shown, it contains four subgraphs. Figure 5 (a) Corresponding to the core content of the principle of refined contact reconstruction, it presents the suspension problem caused by the excessive smoothing of existing technology, resulting in an average gap of 3.2mm and a maximum of 5.7mm between the hand and the cup. Figure 5 (b) Demonstrates the penetrating effect of a finger penetrating a cup to a depth of 2.8mm-4.1mm caused by an uncontrolled high-frequency signal; Figure 5(c) As per the technology of this invention, the palm fits snugly against the cup body without gaps or gaps, presenting a natural gripping interaction. Figure 5 (d) Presents the precise features processed by the present invention (with orderly geometric relationships represented by blue regular topological lines), which visually metaphorically visualizes the optimization process from noise interference to precise feature expression.

[0098] Multi-view visualization of the reconstruction results further demonstrates the performance in complex scenarios, such as Figure 6 As shown, the figure contains an interactive scene sub-figure, presenting the left and right side views. It can be clearly seen that in the handheld object scene, the overlap rate of the finger and object contact points reconstructed by the present invention reaches 92.7%, the average gap between the human torso and the object surface is small, the object contour error reconstructed in the operation object scene is low, and the human mesh reconstructed by the present invention has natural joint curvature, complete object contour without distortion, and tight fit between human and object contact areas.

[0099] We explored the impact of different design choices in the multi-scale refiner module. As shown in Table 2, we compared simple feature stitching, using only mutual attention, and the complete geometry-guided scheme.

[0100] Table 2: Fusion strategy ablation for multi-scale refiners.

[0101] The results show that: (1) Introducing only the mutual attention mechanism (ID 2) improves ContactF1 compared to simple concatenation (ID 1), proving that active feature transfer helps focus the contact area; (2) Explicitly injecting the relative position vector (ID 3) significantly reduces This confirms the importance of geometric priors for spatial alignment; (3) The combination of the two (ID 4) achieves the best performance, indicating that semantic features and geometric positions are highly complementary in the correction process.

[0102] Table 3: Comparison of different GCN architectures.

[0103] As shown in Table 3, the standard GCN commonly used in existing technologies tends to lead to over-smoothing, while simply introducing high-frequency signals can cause clipping artifacts. The frequency-domain adaptive mechanism proposed in this invention can dynamically balance low-frequency smoothing and high-frequency details, effectively suppressing noise while preserving realistic contact features, resulting in reconstruction accuracy superior to single-frequency processing methods.

[0104] Figure 7This paper compares the 3D reconstruction performance of the proposed method (MRG-HOI) with existing advanced technology (HOI-TG) in complex human-object interaction scenarios. As shown in the two columns on the right and the red markers in the figure, the existing technology HOI-TG frequently exhibits geometric clipping phenomena that violate physical laws when handling close contact between the human body and the object or spatial occlusion. Specifically, this manifests as human limbs penetrating table legs, object edges cutting into the abdomen, and non-physical collapse of the mesh at the contact surface. These defects are attributed to the lack of fine perception of the object's topology and effective noise suppression mechanisms in existing methods, leading to spatial ambiguity in pose prediction under symmetrical or complex geometric conditions.

[0105] In contrast, this invention (middle two columns) demonstrates superior physical consistency and geometric robustness by incorporating a frequency-domain adaptive graph convolutional Transformer and a multi-scale parameter correction mechanism. By adaptively filtering high-frequency noise interference through a frequency-domain gating mechanism and enhancing the topological stability of the structure using rigid constraint branches of the object, this invention effectively corrects local mesh distortion and jitter. Experimental results show that, regardless of whether the object is being transported at a significant angle or in a static seated scenario, the reconstructed human body and object maintain clear and accurate physical boundaries, achieving a tight fit of the contact area without clipping, fully validating the technical advantages of implicit interaction modeling strategies in improving the reconstruction accuracy of complex scenes.

[0106] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any changes, modifications, substitutions, integrations, and parameter changes made to these embodiments within the spirit and principles of the present invention, without departing from the principles and spirit of the present invention, through conventional substitutions or to achieve the same function, fall within the scope of protection of the present invention.

Claims

1. A 3D human-object interaction reconstruction method based on multi-scale correction and frequency domain adaptive graph convolution, characterized in that, include: Acquire a single RGB image and the target object ID, extract image features through a backbone network, predict the initial parameters of the human body and the object based on the initial regressor, and generate the initial three-dimensional vertices of the human body and the object by combining the parameterized model and the object template. An inverted feature pyramid is constructed, the initial 3D vertices are projected onto the feature map and the features are sampled and fused. The parameter residuals are predicted through a fully connected network with a soft truncation mechanism to correct the initial parameters. Based on the corrected parameters, a hybrid token sequence is constructed, the body context vector is extracted, and the hybrid token sequence is feature-optimized by a frequency domain adaptive graph convolution Transformer. By outputting the coordinates of 3D joint points and the coordinates of human and object mesh vertices through the regression head, 3D human-object interaction reconstruction is completed.

2. The method according to claim 1, characterized in that, The process of constructing the inverted feature pyramid and performing feature sampling and fusion specifically includes: Backbone network outputs deep features First, use a single-channel convolutional layer to... Channel compression is performed to obtain basic features. The feature pyramid is formed by sequentially generating feature maps of different resolutions through three cascaded transposed convolutional layers. The resolution of each feature map is increased step by step according to a preset ratio; For the initial three-dimensional vertices of the human body and the three-dimensional vertices of the object any vertex in Using the camera intrinsic parameter matrix and external references ( , Calculate its two-dimensional projection coordinates on each feature map. ,in For the camera rotation matrix, Let be the camera translation vector, calculated using the following formula: , , ;in , , As vertex The projected homogeneous coordinate components are interpolated using a bilinear interpolation function. Extract the feature vector of each projected coordinate on the corresponding feature map, and concatenate and fuse features of different scales to obtain the multi-scale fused features of the vertex. ,in Using two-dimensional projected coordinates, the final result is the aggregation of all vertex features to form a global fused feature. .

3. The method according to claim 2, characterized in that, The parameter residual prediction and parameter correction process of the soft truncation mechanism is as follows: Global fusion features Inputting the fully connected network, it predicts the parameter residuals for human pose, shape, and object pose, respectively. The prediction formula is as follows: , , , ,in For human posture parameter residuals, For human body shape parameter residuals, The residual of the object's rotation matrix. The residual of the object's translation vector. , , , These are the preset scaling factors, and tanh(·) is the hyperbolic tangent activation function, which compresses the output to the (-1,1) interval to ensure that the correction amount is limited. Within the range; The initial parameters are corrected based on the predicted parameter residuals, and the corrected parameters are: , , , ,in , These are the initial posture and shape parameters of the human body. , Let the initial rotation matrix and translation vector of the object be... , To correct the human body posture and shape parameters, , These are the rotation matrix and translation vector of the corrected object.

4. The method according to claim 3, characterized in that, The process of constructing the hybrid token sequence includes: Based on the corrected parameters , Generate corrected 3D human body vertices ,based on , Generate corrected 3D vertices of the object Extract human joint features respectively Human Vertebrae and object vertex features Through a fully connected layer , , Mapped to a unified dimension, forming a hybrid sequence containing both geometric and semantic information. This sequence simultaneously covers key interaction features between the human body and objects, providing a data foundation for subsequent global interaction perception.

5. The method according to claim 4, characterized in that, The process of extracting and applying the object context vector is as follows: Vertex features of objects in a mixed token sequence Perform global average pooling to extract the entity context vector. The extraction formula is: ,in The total number of vertices of the object. For the first Feature vectors of the vertices of an object; It represents the global state of an object, including its spatial location, category attributes, and geometric features.

6. The method according to claim 1, characterized in that, The frequency domain adaptive graph residual block processing process in the frequency domain adaptive graph convolution Transformer includes: Let the features of the input graph nodes be... , The matrix is ​​composed of the feature vectors of all nodes in the graph. The features are transformed from the spatial domain to the frequency domain by a one-dimensional discrete cosine transform (DCT) to obtain the spectral coefficients. ,in The transformation matrix is ​​a DCT; a learnable gated mask is used. Adaptive frequency-selective filtering is applied to the spectral coefficients, and the filtering formula is as follows: ,in This represents element-wise multiplication. The Sigmoid activation function is used to automatically learn weights through training, suppressing high-frequency noise components and preserving low-frequency components that characterize the smoothness of object or human body structures. The filtered spectral coefficients The features are recovered to the spatial domain by inverse discrete cosine transform (IDCT) and smoothed features are obtained. ,in Let A be the transpose of the DCT transformation matrix; let the object context vector be... and Perform feature splicing to obtain Topological neighborhood information is aggregated through graph convolution operations. The graph convolution update formula is: ,in Let be the adjacency matrix of the graph. Let be the degree matrix of the graph. Let be the learnable weight parameters of the graph convolutional layer, and ReLU(·) be the linear rectified activation function.

7. The method according to claim 6, characterized in that, In the graph convolution operation, the construction process of the adjacency matrix A and the degree matrix D is as follows: The adjacency matrix A is dynamically constructed based on the geometric distance between nodes. For any two nodes i and j in the graph, the Euclidean distance between them is calculated. ,in , They are nodes , 3D coordinates; preset distance threshold ,when At that time, determine the node and There are topological connections, and the corresponding elements in the adjacency matrix A are... =1; when When the two are determined to have no topological connection, =0, and =0, the node itself has no connections; Degree matrix It is a diagonal matrix, and its diagonal elements ,in Let be the total number of nodes in the graph. This represents the sum of the elements in the i-th row of the adjacency matrix A, which is the degree of connection of node i.

8. The method according to claim 1, characterized in that, The frequency-domain adaptive graph convolutional Transformer includes a human interaction-aware branch, which employs an interaction-aware adaptive frequency-domain graph convolution to extract features from the human interaction branch as a context vector. And inject it into human characteristics Gated networks; Using graph filtering principles Decoupling is achieved through low-frequency smoothing features and high-frequency sharpening features. The low-frequency path aggregates features using an adjacency matrix A, as shown in the formula: ,in The low-frequency pathway uses learnable weights; the high-frequency pathway extracts high-frequency details by subtracting the smoothing component from the original features, as shown in the formula: ,in These are the learnable weights for high-frequency pathways; Using object context vectors Generate adaptive gating coefficients ,in It is the Sigmoid activation function. Indicates human body characteristics With object context vector splicing It is a multilayer perceptron; through gating coefficients Dynamically fuse low-frequency smoothing features and high-frequency sharpening features to output fused features. When the human body mesh is close to an object, Adaptive reduction, preserving more high-frequency contact details, when moving away from the object It adaptively enlarges to maintain the smoothness of the human body surface.

9. The method according to claim 8, characterized in that, The frequency-domain adaptive graph convolutional Transformer includes an object rigidity constraint branch that employs a hybrid modeling approach combining local smoothing and global rigidity to adapt to the rigid structural characteristics of the object, utilizing a dynamically constructed sparse graph structure. Perform feature smoothing to eliminate reconstruction noise. The dynamic adjacency matrix of the object nodes; local smoothing features are expressed by the formula. Calculation, where The feature matrix of the object nodes, These are the learnable weights for local smoothing pathways; The global mean feature is obtained by averaging the features of all nodes of the object. The features are then input into a multilayer perceptron (MLP) for feature transformation, and then the global features are expanded to the same level using a broadcast operation. With the same dimension, global rigid features are obtained. Features enforce the consistency of the overall structure of an object; the network, through... Automatically learn rigid consistency gating coefficients ,in Use the Sigmoid activation function; outputs object features. The formula is: .

10. The method according to claim 1, characterized in that, The iterative optimization of the frequency domain adaptive graph convolution Transformer is as follows: a staged encoder structure is adopted, the number of feature channels decreases step by step according to a preset ratio, simulating the cognitive process from global semantic understanding to local geometric refinement, and each stage encoder focuses on feature learning at different scales. Input mixed token sequence Global interaction features are extracted through a multi-head self-attention mechanism to capture long-distance dependencies between the human body and objects, different parts of the human body, and different parts of objects. The interacting sequence is then decomposed into human tokens and object tokens, which are fed into the human interaction perception branch and the object rigidity constraint branch for differentiated processing, respectively. The human token focuses on optimizing interaction details, while the object token focuses on preserving rigid structures. The processed human and object features are then concatenated and fused again before being fed into the next level encoder, forming a closed-loop iterative optimization process. Each iteration optimizes the feature representation based on the results of the previous iteration. After multiple layers of alternating processing, the 3D joint coordinates are directly predicted by the regression head. And the coordinates of the vertices of the human body and object mesh, where It is a set of three-dimensional spatial coordinates of key human joints, and outputs the 3D human-object interaction reconstruction results.

Images