A method and system for reconstructing human interaction from monocular images based on graph embedding coding

Abstract

The present invention relates to a method and system for reconstructing monocular image human interaction based on graph embedding coding, comprising: performing data preprocessing on a monocular image of human interaction to be reconstructed; performing 3D initialization reconstruction of a human body and an object based on the image after data preprocessing; optimizing the human body and the object after 3D initialization reconstruction to obtain a final 3D reconstruction result of human interaction. The present invention uses an end-to-end method to solve the disadvantage of slow reasoning speed based on the post-optimization method; in the encoding stage, a multi-layer neural network based on a graph embedding encoder is used to extract feature information for people and objects using different graph residual network modules to improve reconstruction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a method and system for reconstructing human interaction in a monocular image based on graph embedding coding. Background Art

[0002] The task of reconstructing human interaction from monocular images is mainly to reconstruct the 3D models of people and objects from images. Existing methods are mainly based on post-optimization methods, which have the disadvantages of low reconstruction accuracy and slow reasoning speed. At present, there is a lack of a method with high reconstruction accuracy and fast reasoning speed for the monocular human interaction reconstruction task. Summary of the invention

[0003] The purpose of the present invention is to propose a method and system for reconstructing human interaction in a monocular image based on graph embedding coding, so as to solve the problems existing in the above-mentioned prior art, and use an end-to-end method to solve the disadvantage of slow reasoning speed based on the post-optimization method; in the encoding stage, a multi-layer neural network based on a graph embedding encoder is used, and feature information is extracted by using different graph residual network modules for people and objects to improve the reconstruction accuracy.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for reconstructing human interaction in a monocular image based on graph embedding coding, comprising:

[0006] Obtaining a monocular image of human interaction to be reconstructed;

[0007] Perform data preprocessing on the monocular image of human interaction to be reconstructed;

[0008] Based on the images after data preprocessing, 3D initialization reconstruction of human body and objects is performed;

[0009] The human body and objects after 3D initialization reconstruction are optimized to obtain the final 3D reconstruction result of human interaction.

[0010] Optionally, performing data preprocessing on the monocular image of human interaction to be reconstructed includes:

[0011] Using the Detectron2 algorithm, the segmentation mask images of people and objects are extracted from the monocular image of human interaction to be reconstructed;

[0012] Using the image cropping algorithm, the part of the human interaction in the monocular image of human interaction to be reconstructed is cropped to the center of the image;

[0013] Using the image cropping algorithm, the interactive parts of the people in the segmentation mask image of the person and the segmentation mask image of the object are cropped to the center of the image.

[0014] Optionally, performing 3D initialization reconstruction of a human body and an object includes:

[0015] Perform feature extraction on the image after data preprocessing to obtain image feature F;

[0016] Predicting the 3D position and camera projection parameters of human skeleton points from the image features F;

[0017] Project the 3D position of the human skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the body parameter features;

[0018] Based on the body parameter features, predict body parameters body ;

[0019] Predicting the 3D positions of the skeleton points of the human hand from the image features F;

[0020] Project the 3D position of the human hand skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the human hand parameter features;

[0021] Based on the human hand parameter features, predict the human hand parameters hand ;

[0022] The body parameters body and human hand parameters hand Merge and pass through the SMPLH model to obtain the initialized human body 3D model;

[0023] The rotation and translation parameters of the object are predicted from the image feature F, and the object 3D template is rigidly rotated to obtain an initialized object 3D model, wherein the object 3D template refers to the 3D template of the object interacting in the monocular image of the human interaction to be reconstructed.

[0024] Optionally, optimizing the human body and the object after 3D initialization reconstruction includes:

[0025] Based on the camera projection parameters, the initialized human 3D model and the initialized object 3D model are projected onto the two-dimensional plane where the image features are located;

[0026] Based on the projected image feature plane, human skeleton point features, human vertex features and object vertex features are obtained through grid sampling;

[0027] Perform feature splicing on human skeleton point features, human vertex features and object vertex features;

[0028] A multi-layer neural network based on a graph embedding encoder extracts the concatenated features to obtain the 3D coordinates of the human body and the object to be processed;

[0029] Perform an upsampling operation on the 3D coordinates of the human body to be processed to obtain a final reconstructed 3D model of the human body;

[0030] The final rotation and translation are obtained by using a rigid inverse transformation algorithm on the 3D coordinates of the object to be processed; based on the final rotation and translation, the 3D template of the object in the reconstructed monocular image of the human interaction is processed using a rigid transformation algorithm to obtain the final reconstructed 3D model of the object.

[0031] Optionally, the multi-layer neural network based on the graph embedding encoder includes: a plurality of neural network layers of the same encoder;

[0032] For each layer of graph embedding-based encoder, the feature extraction steps are as follows:

[0033] Use a multi-head self-attention encoder to extract the concatenated features and obtain the encoding features;

[0034] Splitting the coding features into human skeleton point coding features, human vertex coding features and object vertex coding features;

[0035] The human body graph convolution residual network is used to extract the human body vertex coding features to obtain the human body vertex graph coding features;

[0036] The object graph convolution residual network is used to extract the feature of the object vertex encoding to obtain the object vertex graph encoding feature;

[0037] Input the human skeleton point encoding features, human vertex map encoding features and object vertex encoding features into the next layer of encoder network;

[0038] The output of the last layer of graph embedding encoder is obtained, namely, the final human skeleton point encoding features, human vertex graph encoding features and object vertex graph encoding features, and the final human vertex graph encoding features and object vertex graph encoding features are extracted to obtain the 3D coordinates of the human body to be processed and the 3D coordinates of the object to be processed.

[0039] A monocular image character interaction reconstruction system based on graph embedding coding, the system comprising: an acquisition module, a preprocessing module, an initialization module and an optimization module;

[0040] The acquisition module is used to acquire the monocular image of the human interaction to be reconstructed;

[0041] The preprocessing module is used to perform data preprocessing on the monocular image of human interaction to be reconstructed;

[0042] The initialization module is used to perform 3D initialization reconstruction of human body and objects based on the images after data preprocessing;

[0043] The optimization module is used to optimize the human body and objects after 3D initialization reconstruction to obtain the final human interaction 3D reconstruction result.

[0044] Optionally, the preprocessing module performs data preprocessing on the monocular image of human interaction to be reconstructed, including:

[0045] Using the Detectron2 algorithm, the segmentation mask images of people and objects are extracted from the monocular image of human interaction to be reconstructed;

[0046] Using the image cropping algorithm, the part of the human interaction in the monocular image of human interaction to be reconstructed is cropped to the center of the image;

[0047] Using the image cropping algorithm, the interactive parts of the people in the segmentation mask image of the person and the segmentation mask image of the object are cropped to the center of the image.

[0048] Optionally, the initialization module performs 3D initialization reconstruction of a human body and an object, including:

[0049] Perform feature extraction on the image after data preprocessing to obtain image feature F;

[0050] Predicting the 3D position and camera projection parameters of human skeleton points from the image features F;

[0051] Project the 3D position of the human skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the body parameter features;

[0052] Based on the body parameter features, predict body parameters body ;

[0053] Predicting the 3D positions of the skeleton points of the human hand from the image features F;

[0054] Project the 3D position of the human hand skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the human hand parameter features;

[0055] Based on the human hand parameter features, predict the human hand parameters hand ;

[0056] The body parameters body and human hand parameters hand Merge and pass through the SMPLH model to obtain the initialized human body 3D model;

[0057] The rotation and translation parameters of the object are predicted from the image feature F, and the object 3D template is rigidly rotated to obtain an initialized object 3D model, wherein the object 3D template refers to the 3D template of the object interacting in the monocular image of the human interaction to be reconstructed.

[0058] Optionally, the optimization module optimizes the human body and the object after 3D initialization reconstruction, including:

[0059] Based on the camera projection parameters, the initialized human 3D model and the initialized object 3D model are projected onto the two-dimensional plane where the image features are located;

[0060] Based on the projected image feature plane, human skeleton point features, human vertex features and object vertex features are obtained through grid sampling;

[0061] Perform feature splicing on human skeleton point features, human vertex features and object vertex features;

[0062] A multi-layer neural network based on a graph embedding encoder extracts the concatenated features to obtain the 3D coordinates of the human body and the object to be processed;

[0063] Perform an upsampling operation on the 3D coordinates of the human body to be processed to obtain a final reconstructed 3D model of the human body;

[0064] The final rotation and translation are obtained by using a rigid inverse transformation algorithm on the 3D coordinates of the object to be processed; based on the final rotation and translation, the 3D template of the object in the reconstructed monocular image of the human interaction is processed using a rigid transformation algorithm to obtain the final reconstructed 3D model of the object.

[0065] Optionally, the multi-layer neural network based on the graph embedding encoder includes: a plurality of neural network layers of the same encoder;

[0066] For each layer of graph embedding-based encoder, the feature extraction steps are as follows:

[0067] Use a multi-head self-attention encoder to extract the concatenated features and obtain the encoding features;

[0068] Splitting the coding features into human skeleton point coding features, human vertex coding features and object vertex coding features;

[0069] The human body graph convolution residual network is used to extract the human body vertex coding features to obtain the human body vertex graph coding features;

[0070] The object graph convolution residual network is used to extract the feature of the object vertex encoding to obtain the object vertex graph encoding feature;

[0071] Input the human skeleton point encoding features, human vertex map encoding features and object vertex encoding features into the next layer of encoder network;

[0072] The output of the last layer of graph embedding encoder is obtained, namely, the final human skeleton point encoding features, human vertex graph encoding features and object vertex graph encoding features, and the final human vertex graph encoding features and object vertex graph encoding features are extracted to obtain the 3D coordinates of the human body to be processed and the 3D coordinates of the object to be processed.

[0073] The beneficial effects of the present invention are:

[0074] The present invention proposes a method and system for reconstructing human interaction from a monocular image based on graph embedding coding. First, data preprocessing is performed on the monocular image of human interaction to be reconstructed. Secondly, 3D initialization reconstruction of the human body and objects is performed based on the image after data preprocessing. Finally, the human body and objects after 3D initialization reconstruction are optimized to obtain the final 3D reconstruction result of human interaction. The end-to-end method is used to solve the shortcoming of slow reasoning speed based on the post-optimization method. In the encoding stage, a multi-layer neural network based on a graph embedding encoder is used to extract feature information for people and objects by using different graph residual network modules to improve the reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0076] Figure 1 A schematic diagram of a flow chart of a method for reconstructing interactive characters in a monocular image based on graph embedding coding according to an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of the workflow of each layer of a graph embedding encoder according to an embodiment of the present invention;

[0078] Figure 3 The present invention is a schematic diagram of a monocular image character interaction reconstruction structure based on graph embedding coding according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] This embodiment proposes a monocular image person interaction reconstruction method based on graph embedding coding, including:

[0082] Obtaining a monocular image of human interaction to be reconstructed;

[0083] Perform data preprocessing on the monocular image of human interaction to be reconstructed;

[0084] Based on the images after data preprocessing, 3D initialization reconstruction of human body and objects is performed;

[0085] The human body and objects after 3D initialization reconstruction are optimized to obtain the final 3D reconstruction result of human interaction.

[0086] Specifically, in this embodiment, Figure 1 As shown, the monocular image person interaction reconstruction method based on graph embedding coding includes:

[0087] S110, using a data preprocessing module to preprocess the input image to obtain an input of an initialization module;

[0088] S111, using an initialization module to realize 3D initialization reconstruction of a human body and an object;

[0089] S112, using an optimization reconstruction module to optimize the 3D initialization reconstruction of the human body and the object, to obtain a final 3D reconstruction of the human body and the object, that is, a human interaction reconstruction.

[0090] Furthermore, data preprocessing of the reconstructed monocular image of human interaction includes:

[0091] Using the Detectron2 algorithm, the segmentation mask images of people and objects are extracted from the monocular image of human interaction to be reconstructed;

[0092] Using the image cropping algorithm, the part of the human interaction in the human interaction monocular image to be reconstructed is cropped to the center of the image;

[0093] Using the image cropping algorithm, the interactive parts of the people in the segmentation mask image of the person and the segmentation mask image of the object are cropped to the center of the image.

[0094] Furthermore, the 3D initialization reconstruction of the human body and objects includes:

[0095] Perform feature extraction on the image after data preprocessing to obtain image feature F;

[0096] From the further image features F, predict the 3D position of the human skeleton points and the camera projection parameters;

[0097] Project the 3D position of the human skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the body parameter features;

[0098] Based on further body parameter features, predict body parameters body ;

[0099] From the further image features F, predict the 3D positions of the human hand skeleton points;

[0100] Project the 3D position of the human hand skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the human hand parameter features;

[0101] Based on further human hand parameter features, predict human hand parameters hand ;

[0102] The body parameters body and human hand parameters hand Merge, input the merged parameters into the SMPLH model to obtain the initialized human body 3D model; SMPLH is a model that can convert parameters into 3D human body vertices;

[0103] From the further image feature F, the rotation and translation parameters of the object are predicted, and the object 3D template is rigidly rotated to obtain an initialized object 3D model, wherein the object 3D template refers to the 3D template of the object interacting in the monocular image of the human interaction to be reconstructed.

[0104] Furthermore, the optimization of the human body and objects after 3D initialization reconstruction includes:

[0105] Based on the camera projection parameters, the initialized human 3D model and the initialized object 3D model are projected onto the two-dimensional plane where the image features are located;

[0106] Based on the projected image feature plane, human skeleton point features, human vertex features and object vertex features are obtained through grid sampling;

[0107] Perform feature splicing on human skeleton point features, human vertex features and object vertex features;

[0108] A multi-layer neural network based on a graph embedding encoder extracts the concatenated features to obtain the 3D coordinates of the human body and the object to be processed;

[0109] Perform an upsampling operation on the 3D coordinates of the human body to be processed to obtain a final reconstructed 3D model of the human body;

[0110] The final rotation and translation are obtained by using the rigid body inverse transformation algorithm on the 3D coordinates of the object to be processed; based on the final rotation and translation, the 3D template of the object in the reconstructed monocular image of human interaction is processed using the rigid body transformation algorithm to obtain the final reconstructed 3D model of the object. The object 3D template refers to the 3D template of the object interacting in the human interaction image.

[0111] Furthermore, a multi-layer neural network based on the graph embedding encoder extracts features from the concatenated features; for each layer of the neural network, the feature extraction step includes:

[0112] Use a multi-head self-attention encoder to extract the concatenated features and obtain the encoding features;

[0113] Splitting the coding features into human skeleton point coding features, human vertex coding features and object vertex coding features;

[0114] The human body graph convolution residual network is used to extract the human body vertex coding features to obtain the human body vertex graph coding features;

[0115] The object graph convolution residual network is used to extract the feature of the object vertex encoding to obtain the object vertex graph encoding feature;

[0116] Input the human skeleton point encoding features, human vertex map encoding features and object vertex encoding features into the next layer of encoder network;

[0117] The output of the last layer of graph embedding encoder is obtained, namely, the final human skeleton point encoding features, human vertex graph encoding features and object vertex graph encoding features, and the final human vertex graph encoding features and object vertex graph encoding features are extracted to obtain the 3D coordinates of the human body to be processed and the 3D coordinates of the object to be processed.

[0118] Figure 2 The arrow in the middle represents the features obtained after the data flows to the encoder and is processed. The circular cross in the middle represents addition. The specific process is as follows:

[0119] 1. The input coding features can be divided into human skeleton point features, human vertex features and object vertex features. They are merged and input into the multi-head self-attention encoder for feature extraction, and residual connections are added to obtain human skeleton point coding features, human vertex coding features and object vertex coding features.

[0120] 2. For the human body vertex encoding features, they are input into the human body graph residual module to obtain the human body vertex graph encoding features.

[0121] 3. For the object vertex encoding features, they are input into the object graph convolution residual module to obtain the object vertex graph encoding features.

[0122] 4. Finally, the human skeleton point encoding features, human vertex map encoding features, and object vertex map encoding features are combined to obtain the encoding features as the input of the next layer of the network.

[0123] like Figure 3 As shown, this embodiment proposes a monocular image character interaction reconstruction system based on graph embedding coding, and the system includes: an acquisition module, a preprocessing module, an initialization module and an optimization module;

[0124] An acquisition module, used for acquiring a monocular image of human interaction to be reconstructed;

[0125] A preprocessing module, used for performing data preprocessing on the monocular image of human interaction to be reconstructed;

[0126] An initialization module is used to perform 3D initialization reconstruction of human bodies and objects based on images after data preprocessing;

[0127] The optimization module is used to optimize the human body and objects after 3D initialization reconstruction to obtain the final 3D reconstruction result of human interaction.

[0128] Furthermore, the preprocessing module performs data preprocessing on the reconstructed monocular image of human interaction, including:

[0129] Using the Detectron2 algorithm, the segmentation mask images of people and objects are extracted from the monocular image of human interaction to be reconstructed;

[0130] Using the image cropping algorithm, the part of the human interaction in the monocular image of human interaction to be reconstructed is cropped to the center of the image;

[0131] Using the image cropping algorithm, the interactive parts of the people in the segmentation mask image of the person and the segmentation mask image of the object are cropped to the center of the image.

[0132] Specifically, in this embodiment, as a feasible implementation method, a monocular camera is used to extract a monocular image of human interaction to be reconstructed;

[0133] Use the Detectron2 algorithm to extract the segmentation mask image of the person and the segmentation mask image of the object from the monocular image of the person interaction to be reconstructed;

[0134] An image cropping algorithm is used on the extracted image to be reconstructed to crop the interactive part of the characters in the image to the center of the image, and the same cropping algorithm is used on the segmentation mask image of the person and the segmentation mask image of the object to obtain the input of the initialization module.

[0135] Furthermore, the initialization module performs 3D initialization reconstruction of the human body and the object including:

[0136] Perform feature extraction on the image after data preprocessing to obtain image feature F;

[0137] From the further image features F, predict the 3D position of the human skeleton points and the camera projection parameters;

[0138] Project the 3D position of the human skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the body parameter features;

[0139] Based on further body parameter features, predict body parameters body ;

[0140] From the further image features F, predict the 3D positions of the human hand skeleton points;

[0141] Project the 3D position of the human hand skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the human hand parameter features;

[0142] Based on further human hand parameter features, predict human hand parameters hand ;

[0143] The body parameters body and human hand parameters hand Merge and pass through the SMPLH model to obtain the initialized human body 3D model;

[0144] From the further image feature F, the rotation and translation parameters of the object are predicted, and the object 3D template is rigidly rotated to obtain the initialized object 3D model.

[0145] Specifically, in this embodiment, as an achievable implementation method, using the initialization module to implement 3D initialization reconstruction of a human body and an object includes:

[0146] The ResNet50 backbone network is used to extract features from the input of the initialization module to obtain the image features F∈RH×W×C, where H and W represent the height and width of the image features, and C represents the number of feature channels.

[0147] Predict the 3D locations J of body skeleton points from image features F using a linear layer network body and camera projection parameters.

[0148] The 3D position J of the body skeleton point body Projected to the plane space of the image feature F, the body parameter features are obtained using grid sampling, i.e., bilinear interpolation algorithm.

[0149] Predict body parameters from body parameter features using a linear layer network body .

[0150] Predict the 3D position J of the human hand skeleton point from the image features F using a linear layer network hand .

[0151] The 3D position J of the human hand skeleton point hand Projected to the plane space of the image feature F, the body parameter features are obtained using grid sampling, i.e., bilinear interpolation algorithm.

[0152] Predicting body parameters from human hand parameter features using a linear layer network hand .

[0153] The body parameters and human hand parameters are combined through the SMPLH model to obtain the initialized human 3D model.

[0154] A linear layer network is used to predict the rotation and translation parameters of the object from the image features, and a rigid body rotation is applied to the object 3D template to obtain the initialized object 3D model.

[0155] Furthermore, the optimization module optimizes the human body and objects after 3D initialization reconstruction, including:

[0156] Based on the camera projection parameters, the initialized human 3D model and the initialized object 3D model are projected onto the two-dimensional plane where the image features are located;

[0157] Based on the projected image feature plane, human skeleton point features, human vertex features and object vertex features are obtained through grid sampling;

[0158] Perform feature splicing on human skeleton point features, human vertex features and object vertex features;

[0159] A multi-layer neural network based on a graph embedding encoder extracts the concatenated features to obtain the 3D coordinates of the human body and the object to be processed;

[0160] Perform an upsampling operation on the 3D coordinates of the human body to be processed to obtain a final reconstructed 3D model of the human body;

[0161] The final rotation and translation are obtained by using a rigid inverse transformation algorithm on the 3D coordinates of the object to be processed; based on the final rotation and translation, the 3D template of the object in the reconstructed monocular image of the human interaction is processed using a rigid transformation algorithm to obtain the final reconstructed 3D model of the object.

[0162] Specifically, in this embodiment, as an achievable implementation method, an optimization reconstruction module is used to optimize the 3D initialization reconstruction of a human body and an object to obtain the final 3D reconstruction of the human body and the object, that is, the interactive reconstruction of the person includes:

[0163] Project the initialized 3D models of human bodies and objects onto the image feature plane using camera parameters;

[0164] The human skeleton point features, human vertex features and object vertex features are obtained through grid sampling;

[0165] The input of the feature extraction unit is obtained by concatenating the human skeleton point features, the human vertex features and the object vertex features;

[0166] A multi-layer neural network based on a graph embedding encoder is used to extract input features, and finally the 3D coordinates of the human body and the object to be processed are obtained.

[0167] The final 3D reconstruction of the human body is obtained by upsampling the 3D coordinates of the human body to be processed;

[0168] The final rotation and translation are obtained by applying a rigid body inverse transformation algorithm to the 3D coordinates of the object to be processed, and the final 3D reconstruction of the object is obtained by applying a rigid body transformation algorithm to the 3D template of the object based on the final rotation and translation.

[0169] Further, the multi-layer neural network based on the graph embedding encoder includes: a plurality of neural network layers of the same encoder;

[0170] For each layer of graph embedding-based encoder, the feature extraction steps are as follows:

[0171] Use a multi-head self-attention encoder to extract the concatenated features and obtain the encoding features;

[0172] Splitting the coding features into human skeleton point coding features, human vertex coding features and object vertex coding features;

[0173] The human body graph convolution residual network is used to extract the human body vertex coding features to obtain the human body vertex graph coding features;

[0174] The object graph convolution residual network is used to extract the feature of the object vertex encoding to obtain the object vertex graph encoding feature;

[0175] Input the human skeleton point encoding features, human vertex map encoding features and object vertex encoding features into the next layer of encoder network;

[0176] The output of the last layer of graph embedding encoder is obtained, namely, the final human skeleton point encoding features, human vertex graph encoding features and object vertex graph encoding features, and the final human vertex graph encoding features and object vertex graph encoding features are extracted to obtain the 3D coordinates of the human body to be processed and the 3D coordinates of the object to be processed.

[0177] This embodiment proposes a monocular image character interaction reconstruction method based on graph embedding coding, which uses an end-to-end approach to solve the shortcoming of slow reasoning speed based on post-optimization methods.

[0178] In the encoding stage, a multi-layer neural network based on a graph embedding encoder is used to extract feature information for people and objects using different graph residual network modules to improve reconstruction accuracy.

[0179] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for reconstructing human interaction from monocular images based on graph embedding coding, characterized in that: include: Obtaining a monocular image of human interaction to be reconstructed; Perform data preprocessing on the reconstructed monocular image of human interaction; Based on the images after data preprocessing, 3D initialization reconstruction of human bodies and objects is performed; Initialized 3D reconstruction of human bodies and objects includes: Perform feature extraction on the image after data preprocessing to obtain image features F; Predicting the 3D positions and camera projection parameters of human skeleton points from the image features F; Project the 3D position of the human skeleton points to the plane space of the image feature F and use the bilinear interpolation algorithm to obtain the body parameter features; Based on the body parameter characteristics, predict body parameters body ; Predicting the 3D positions of the human hand skeleton points from the image features F; Project the 3D position of the human hand skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the human hand parameter features; Based on the human hand parameter features, predict the human hand parameters hand ; The body parameters body and human hand parameters hand Merge, input the merged parameters into the SMPLH model to obtain the initialized human body 3D model; Predict the rotation and translation parameters of the object from the image feature F, perform rigid body rotation on the object 3D template, and obtain an initialized object 3D model; wherein the object 3D template refers to the 3D template of the object interacting in the monocular image of the human interaction; The human body and objects after 3D initialization reconstruction are optimized to obtain the final 3D reconstruction results of human interaction.

2. The method for reconstructing interactive characters from monocular images based on graph embedding coding according to claim 1, characterized in that: Data preprocessing for reconstructed monocular images of human interaction includes: Using the Detectron2 algorithm, the segmentation mask images of people and objects are extracted from the monocular image of human interaction to be reconstructed. Using the image cropping algorithm, the part of the human interaction in the monocular image to be reconstructed is cropped to the center of the image; Using the image cropping algorithm, the interactive parts of the people in the segmentation mask image of the person and the segmentation mask image of the object are cropped to the center of the image.

3. The method for reconstructing interactive characters from monocular images based on graph embedding coding according to claim 1, characterized in that: Optimization of the human body and objects after 3D initialization reconstruction includes: Based on the camera projection parameters, the initialized 3D human body model and the initialized 3D object model are projected onto the two-dimensional plane where the image features are located; Based on the projected image feature plane, human skeleton point features, human vertex features, and object vertex features are obtained through grid sampling; Perform feature splicing on human skeleton point features, human vertex features and object vertex features; A multi-layer neural network based on the graph embedding encoder extracts the spliced ​​features to obtain the 3D coordinates of the human body and the object to be processed; Perform upsampling on the 3D coordinates of the human body to be processed to obtain the final reconstructed 3D model of the human body; The final rotation and translation are obtained by using the rigid inverse transformation algorithm on the 3D coordinates of the object to be processed; based on the final rotation and translation, the 3D template of the object in the reconstructed monocular image of the human interaction is processed using the rigid transformation algorithm to obtain the final reconstructed 3D model of the object.

4. The method for reconstructing interactive characters from monocular images based on graph embedding coding according to claim 3, characterized in that: The multi-layer neural network based on the graph embedding encoder extracts features from the concatenated features, including: The multi-layer neural network based on graph embedding encoder includes: several neural network layers with the same encoder; For each layer of the graph embedding-based encoder, the feature extraction steps are as follows: Use the multi-head self-attention encoder to extract the concatenated features and obtain the encoding features; Splitting the coding features into human skeleton point coding features, human vertex coding features and object vertex coding features; The human body graph convolution residual network is used to extract the human body vertex coding features to obtain the human body vertex graph coding features; The object graph convolution residual network is used to extract the object vertex encoding features to obtain the object vertex graph encoding features; Input the human skeleton point encoding features, human vertex map encoding features and object vertex encoding features into the next layer of encoder network; The output of the last layer of graph embedding encoder is obtained, namely the final human skeleton point encoding features, human vertex graph encoding features and object vertex graph encoding features. The final human vertex graph encoding features and object vertex graph encoding features are extracted to obtain the 3D coordinates of the human body to be processed and the 3D coordinates of the object to be processed.

5. A monocular image character interaction reconstruction system based on graph embedding coding, characterized by: For implementing the method according to any one of claims 1 to 4, the system comprises: an acquisition module, a preprocessing module, an initialization module and an optimization module; The acquisition module is used to acquire the monocular image of the human interaction to be reconstructed; The preprocessing module is used to perform data preprocessing on the monocular image of human interaction to be reconstructed; The initialization module is used to perform 3D initialization reconstruction of human bodies and objects based on the images after data preprocessing; The initialization module performs 3D initialization reconstruction of the human body and objects, including: Perform feature extraction on the image after data preprocessing to obtain image features F; Predicting the 3D positions and camera projection parameters of human skeleton points from the image features F; Project the 3D position of the human skeleton points to the plane space of the image feature F and use the bilinear interpolation algorithm to obtain the body parameter features; Based on the body parameter characteristics, predict body parameters body ; Predicting the 3D positions of the human hand skeleton points from the image features F; Project the 3D position of the human hand skeleton points to the plane space of the image feature F, and use the bilinear interpolation algorithm to obtain the human hand parameter features; Based on the human hand parameter features, predict the human hand parameters hand ; The body parameters body and human hand parameters hand Merge and pass the SMPLH model to obtain the initialized human body 3D model; Predict the rotation and translation parameters of the object from the image feature F, perform rigid body rotation on the object 3D template, and obtain an initialized object 3D model; wherein the object 3D template refers to the 3D template of the object interacting in the monocular image of the human interaction; The optimization module is used to optimize the human body and objects after 3D initialization reconstruction to obtain the final human interaction 3D reconstruction result.

6. The monocular image character interaction reconstruction system based on graph embedding coding according to claim 5 is characterized in that: The preprocessing module performs data preprocessing on the reconstructed monocular image of human interaction, including: Using the Detectron2 algorithm, the segmentation mask images of people and objects are extracted from the monocular image of human interaction to be reconstructed. Using the image cropping algorithm, the part of the human interaction in the monocular image to be reconstructed is cropped to the center of the image; Using the image cropping algorithm, the interactive parts of the people in the segmentation mask image of the person and the segmentation mask image of the object are cropped to the center of the image.

7. The monocular image character interaction reconstruction system based on graph embedding coding according to claim 5 is characterized in that: The optimization module optimizes the human body and objects after 3D initialization and reconstruction, including: Based on the camera projection parameters, the initialized 3D human body model and the initialized 3D object model are projected onto the two-dimensional plane where the image features are located; Based on the projected image feature plane, human skeleton point features, human vertex features, and object vertex features are obtained through grid sampling; Perform feature splicing on human skeleton point features, human vertex features and object vertex features; A multi-layer neural network based on the graph embedding encoder extracts the spliced ​​features to obtain the 3D coordinates of the human body and the object to be processed; Perform upsampling on the 3D coordinates of the human body to be processed to obtain the final reconstructed 3D model of the human body; The final rotation and translation are obtained by using the rigid inverse transformation algorithm on the 3D coordinates of the object to be processed; based on the final rotation and translation, the 3D template of the object in the reconstructed monocular image of the human interaction is processed using the rigid transformation algorithm to obtain the final reconstructed 3D model of the object.

8. The monocular image character interaction reconstruction system based on graph embedding coding according to claim 7 is characterized in that: The multi-layer neural network based on graph embedding encoder includes: several neural network layers with the same encoder; For each layer of the graph embedding-based encoder, the feature extraction steps are as follows: Use the multi-head self-attention encoder to extract the concatenated features and obtain the encoding features; Splitting the coding features into human skeleton point coding features, human vertex coding features and object vertex coding features; The human body graph convolution residual network is used to extract the human body vertex coding features to obtain the human body vertex graph coding features; The object graph convolution residual network is used to extract the object vertex encoding features to obtain the object vertex graph encoding features; Input the human skeleton point encoding features, human vertex map encoding features and object vertex encoding features into the next layer of encoder network; The output of the last layer of graph embedding encoder is obtained, namely the final human skeleton point encoding features, human vertex graph encoding features and object vertex graph encoding features. The final human vertex graph encoding features and object vertex graph encoding features are extracted to obtain the 3D coordinates of the human body to be processed and the 3D coordinates of the object to be processed.

Images