Monocular character reconstruction method based on stacked normalized flow model

Abstract

This invention discloses a monocular character reconstruction method based on a stacked normalized flow model. The main innovations of this invention are: (1) It proposes a way to characterize the three-dimensional spatial position of people and objects, which is more complete, explicit and efficient than previous characterization methods. (2) It designs a neural network to extract the spatial relationship between people and objects. This network takes into account the ambiguity problem of monocular reconstruction itself and extracts the probability distribution of the spatial relationship between people and objects from the image, rather than point estimation, thus making it more suitable for three-dimensional reconstruction in monocular scenes. (3) This invention proposes a character offset loss in the post-optimization step, which can more efficiently constrain the spatial constraints between people and objects compared with previous algorithms.

Description

Technical Field

[0001] This invention relates to a monocular reconstruction method for three-dimensional mesh models of people and objects, belonging to the field of computer vision in computer science. This field has a wide range of applications, such as in human behavior perception and understanding, human-computer interaction, augmented reality, and embodied intelligence. This field is a relatively new field in recent years and is constantly developing, with various new technologies, models, and algorithms emerging one after another. Background Technology

[0002] In recent years, monocular reconstruction of humans and objects has been studied separately without considering their interaction. However, increasing research suggests that combining human and object reconstruction can yield physically plausible results and even improve the accuracy of both. Monocular reconstruction of humans and objects requires reconstructing mesh models of both humans and objects from a single RGB image. The interaction between humans and objects plays a crucial role in this process. How to utilize this interaction to achieve better reconstruction results remains an unsolved problem.

[0003] The most crucial technique in monocular 3D reconstruction of people and objects is depicting the 3D spatial positions between them and perceiving their spatial relationship from images. Existing techniques for depicting the 3D spatial relationships of people can be divided into two categories: methods based on contact surfaces and methods based on implicit surface equations. The contact surface is defined as the area where the mesh models of a person and an object come into contact during interaction. (PHOSA) [1] Based on common sense, possible contact areas are manually defined when people interact with certain objects. This method reconstructs people and objects separately from images, and uses the contact areas in post-optimization to fine-tune the relative positions between people and objects. This method is simple and effective, but it requires manually defining contact areas (i.e., constructing contact surfaces for each object and each interaction type using prior human knowledge) and is not applicable to objects with a high diversity of interaction methods. It cannot adaptively learn prior knowledge of human-object interaction relationships from data. CHORE [2] Using implicit equations [3] To characterize the positional relationship between people and objects, this method constructs a CHORE domain between people and objects based on the input image. During the optimization process, the SMPL model and the object's template model are fitted into the CHORE domain. This method achieves better reconstruction accuracy than the previous PHOSA method, but its optimization process is relatively cumbersome. [2] The method is not explicit enough in terms of spatial relationships between characters, which consumes a lot of time in the later optimization steps. In addition, the method cannot robustly handle cases of severe occlusion. Summary of the Invention

[0004] The technical problem to be solved by this invention is that there are still some problems and shortcomings in the methods in the field of computer vision: (1) how to characterize the spatial relationship between people and objects, that is, to find a more suitable vectorized encoding of the three-dimensional spatial relationship between people and objects; (2) how to extract this spatial relationship from the image and apply it to the monocular three-dimensional reconstruction of people and objects.

[0005] To solve the above technical problems, the present invention: (1) uses the anchor point offset between people and objects to represent the three-dimensional spatial relationship between people and objects; (2) in order to extract this spatial relationship from the image, the present invention uses a hierarchical Normalizing Flow, first obtaining the probability distribution of the human pose from the image, and then obtaining the probability distribution of the relationship between people and objects based on the probability distribution of the human pose and the content of the image. This method can stabilize the training process and speed up the training process; (3) in the post-optimization process, the anchor point offset loss between people and objects is used to constrain the spatial position between people and objects. Compared with the method of reconstructing people and objects separately without offset loss, better accuracy of joint reconstruction of people and objects is achieved.

[0006] This invention discloses a monocular character reconstruction method based on a stacked normalized flow model, characterized by comprising the following steps:

[0007] Step 1: Obtain the latent vector of the spatial relationship between people and objects in the image corresponding to each human interaction instance in the training dataset. Implicit vectors of spatial relationships The neural network model was trained using auxiliary labels, where a pair of people and objects appearing in the same image were defined as human interaction instances.

[0008] Step 2: Construct and train a neural network model to extract spatial relationships between people and objects. This neural network model predicts the posterior probability distribution of spatial relationships between people and objects from the input image. Its specific implementation includes the following steps:

[0009] Step 2-1: The neural network model encodes the input image into a visual feature vector c;

[0010] Step 2-2: Use a multilayer perceptron model to extract the extrinsic parameters of the camera and the shape parameter β of the SMPL model from the visual feature vector c. init ;

[0011] Steps 2-3: Using the normalized flow model—conditional on the visual feature vector c—sample z from the normal distribution. θ ~N(0, I) is transformed into human pose θ, thereby constructing the probability distribution p of human pose θ. Θ|I (θ|c);

[0012] Steps 2-4: Using Normalized Flow Model II, with visual feature vector c and human pose θ as conditions, the samples z taken from the normal distribution are... γ ~n(0, I) is transformed into the spatial relationship vector γ between people and objects, thereby constructing the probability distribution p of the spatial relationship between people and objects. Γ|I；Θ (γ|c,θ);

[0013] Steps 2-5: Sample the data with the highest probability density γ from the spatial relationship distribution of people and objects. * Further based on x * =Vγ * +μ, from the coordinate offset vector x * Extract the coordinate offset vector D = {d} between the human anchor point and the object anchor point according to the corresponding dimension. i,j}, i=1,...,m,j=1,...,n, where V is the projection matrix and μ is the x * The mean vector;

[0014] Steps 2-6: Select the pose with the highest probability density from the probability distribution of human pose θ. Using θ init and the shape parameter β obtained in step 2-2 init As the initialization of the human body mesh model SMPL, the following formula is then optimized to obtain the relative rotation matrix R of the object. init and relative translation coordinates t init :

[0015]

[0016] Among them, L rel The offset loss is shown in the following formula:

[0017]

[0018] In the formula, This represents the three-dimensional coordinates of the i-th anchor point on the surface of the SMPL mesh model, which is determined by the pose parameter θ and the shape parameter β. Let represent the three-dimensional coordinates of the j-th anchor point on the surface of the object mesh model, which takes the rotation matrix R and the coordinate translation vector t as parameters;

[0019] Steps 2-7: Use the SMPL model to calculate θ init and β init Convert to a human body mesh model, and use the rotation matrix R init Translation vector t init The 3D position of the object's mesh model is obtained by applying a template model to the object;

[0020] Step 3: Use the character offset loss and reprojection loss to perform post-optimization on the results of the neural network model established in Step 2. During model training, calculate the gradient of the calculated loss and backpropagate it until the model converges, and finally train the stacked normalized flow model.

[0021] Preferably, step 1 further includes the following steps:

[0022] Step 1-1: Randomly sample m points from the surface of the human body mesh model SMPL to form the human body anchor point set A. SMPL An object anchor point set A is formed by randomly sampling n points from the surface of different object mesh models. object ;

[0023] Step 1-2: For each human interaction instance in the training dataset, calculate the relative anchor point coordinate offset between all m human anchor points and n object anchor points to obtain m×n coordinate offsets. Connect these coordinate offsets to form a coordinate offset vector x.

[0024] Steps 1-3: Combine the coordinate offset vectors x of all character interaction instances into a coordinate offset matrix X, extract the first k principal component vectors of the coordinate offset matrix X, and use these k principal component vectors to construct the projection matrix V;

[0025] Steps 1-4: For any instance of a person interacting with an object, project it into the latent space using the projection matrix V:

[0026] γ=V T (x-μ)

[0027] In the formula, γ is the compressed representation of the coordinate offset of the person and the object, and μ is the mean vector of x.

[0028] Subsequently, through the inverse projection process, γ is restored to an approximate form of coordinate offset, thus obtaining the implicit vector of spatial relationship.

[0029] Preferably, in steps 1-2, the i-th human body anchor point Anchor point of the j-th object The relative anchor point coordinate offset d between i,j Represented as:

[0030] Preferably, step 3 further includes the following steps:

[0031] Step 3-1: Extract the coordinate correspondence map of key human body nodes and objects from the input image;

[0032] Step 3-2: Optimize the following loss function to obtain the optimal solution (θ)* ,β * R * , t * ):

[0033]

[0034] Where: λ J , λ coor , λ rel and λ posteriori These are weight parameters;

[0035] This refers to the loss of key points in the human body, including:

[0036]

[0037] in, These are the three-dimensional coordinates of human joints. These are the two-dimensional coordinates of the corresponding key human body nodes on the image;

[0038] The loss is the 2D-3D coordinate projection of the object, which includes:

[0039]

[0040] in, These are the three-dimensional coordinates of the object. These are the predicted two-dimensional coordinates of the object. These are the predicted two-dimensional coordinate weights;

[0041] For offset loss;

[0042] It is considered a posteriori loss.

[0043] CHORE [2] While this method achieves better reconstruction accuracy than the previous PHOSA method, its optimization process is quite cumbersome. Therefore, this invention proposes a novel approach to characterizing the 3D spatial relationship between people and objects. This approach uses the anchor point offsets between the surfaces of the person and object mesh models to encode the 3D spatial positional relationship. This characterization method is global, whereas contact-surface-based methods rely on the initial poses of the person and object. Furthermore, this method is efficient, easily integrated with post-optimization processes, and is more explicit and efficient than the implicit surface equation characterization method. Other existing methods exist for characterizing the positional relationship between people and objects, such as using the relative distances between some key nodes of the person and object. [4] 3D coordinate offset of the center point of a person and an object [5]Or the 3D coordinate offset and pose rotation matrix between key nodes of a person and an object. [6] However, this method differs from these others by using a fine-grained, dense form of relative coordinate offsets to characterize the three-dimensional spatial relationship between people and objects. Compared to representations of contact areas, the relative offset-based representation provides more global and complete information; and compared to representations based on implicit equations, this method can be efficiently combined with post-optimization steps.

[0044] One challenge in monocular reconstruction is the ambiguity caused by incomplete viewpoints. Some methods incorporate probabilistic models into the reconstruction algorithm, such as Pro-HMR. [7] Using Normalizing Flow [8] This invention extracts the probability distribution of human poses from input images. Following this technical approach, it is the first to introduce Normalizing Flow to characterize the probability distribution of spatial relationships between people and objects. Compared to previous single-point estimation, the probability distribution provides a potential solution space for subsequent optimization. Furthermore, in the post-optimization step, this technique uses a regression-then-optimization framework, first extracting the spatial positional relationships between people and objects from the image, and then fine-tuning the results based on the image content. The introduction of spatial relationships significantly improves the performance of object reconstruction algorithms alone. [9] Human reconstruction algorithm [7] The accuracy of joint reconstruction of people and objects.

[0045] In summary, the main innovations of this invention are: (1) It proposes a method for characterizing the three-dimensional spatial position of people and objects, which is more complete, explicit, and efficient than previous methods. (2) It designs a neural network for extracting the spatial relationship between people and objects. This network takes into account the ambiguity of monocular reconstruction itself and extracts the probability distribution of the spatial relationship between people and objects from the image, rather than point estimation, thus making it more suitable for three-dimensional reconstruction in monocular scenes. (3) In the post-optimization step, this invention proposes a human offset loss, which can more efficiently constrain the spatial constraints between people and objects compared to previous algorithms.

[0046] Compared with existing technical solutions, the present invention has the following advantages:

[0047] (1) This invention proposes a three-dimensional spatial relationship between a person and an object based on the anchor point offset between the person and the object. Compared with the representation of local contact surfaces, it depicts the global relationship between the person and the object. Compared with the representation of implicit surface equations, it explicitly represents the relative positional relationship between the person and the object. It can also be efficiently combined with post-optimization.

[0048] (2) Compared with previous methods, the method proposed in this invention not only achieves better accuracy in the joint reconstruction of people and objects, but also greatly reduces the time consumed by post-optimization. Attached Figure Description

[0049] Figure 1 This diagram illustrates the relative coordinate offsets between anchor points on a person and an object. It shows how the relative coordinate offsets between the human body and an object are calculated. Given an example of interaction between a person and an object, the coordinate offsets are obtained by subtracting each pair of anchor points on the person and the object. These offsets capture a highly detailed correlation between the body parts and the object and are a quantitative representation that encodes the three-dimensional spatial relationship between them.

[0050] Figure 2 This is a flowchart of the method proposed in this invention. The input image is processed by StackFLOW to obtain the initial prediction result β. init ,θ init ,R init ,t init In the upper and lower branches, OpenPose and Epro-pnp are used respectively to extract the coordinate correspondence between human key nodes and objects. In the post-optimization step, the initial prediction is fine-tuned to obtain the final prediction β. final ,θ final ,R final ,t final .

[0051] Figure 3 This is the data flow diagram of the neural network used in this invention. The input image is processed by a residual network to extract visual features, and these visual features are then processed by an MLP to obtain the initialization result β of the shape parameters in the SMPL. init And the global root node rotation matrix R of SMPL SMPL The human pose normalization flow model, conditioned on visual features, transforms the normal distribution into a distribution P of human poses in an image. Θ|I (θ|c) The point of maximum probability of this distribution corresponds to the initialization parameter θ of the attitude in the SMPL model. init The normalization flow of character relationships, conditioned on visual features and human posture, transforms the normal distribution into a spatial relationship distribution P of characters in an image. Γ|Θ,I (γ|θ,c), the point with the highest probability in this distribution corresponds to the initialization result γ of the spatial relationship. init . Detailed Implementation

[0052] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0053] This invention discloses a method based on stacked normalized flow. This method uses human offset to characterize the spatial relationship between people and objects, and uses a normalized flow model to extract the posterior probability distribution of the spatial relationship between people and objects from the image. Finally, this method also designs an efficient optimization strategy so that the reconstructed structure can fit well with the input image.

[0054] The input of this invention is a single image containing human-object interaction content. This method outputs the pose parameters θ, shape parameters β, rotation matrix R, and offset vector t of the 3D human mesh model SMPL. The method is divided into three modules: vectorized construction of the spatial relationship between humans and objects; prediction of the posterior probability distribution of the spatial relationship between humans and objects from the input image; and optimization of the final result using human offset loss and reprojection loss.

[0055] The specific technical solution used for the vectorized construction of the relationship between people and objects is as follows:

[0056] Step 1-1: Randomly sample m points from the surface of the human body mesh model SMPL to form the human body anchor point set β. SMPL An object anchor point set A is formed by randomly sampling n points from the surface of different object mesh models. object .

[0057] Step 1-2: For each human-object interaction instance in the training dataset (this example uses the BEHAVE dataset, which provides labels for the human 3D mesh and object 3D mesh corresponding to each image; we call a pair of people and objects appearing in the same image a human-object interaction instance), calculate the relative anchor point coordinate offset between the SMPL surface anchor point and the object anchor point:

[0058]

[0059] In the formula, d i,j Represents the i-th human anchor point Anchor point of the j-th object The relative anchor point coordinate offset between them.

[0060] After calculating the relative anchor point coordinate offsets between all m human anchor points and n object anchor points in the example, we will obtain m×n coordinate offsets. These coordinate offsets are then concatenated to form a coordinate offset vector. The spatial relationship of this character interaction instance is represented as x after the coordinate offset is calculated.

[0061] Steps 1-3: Calculate the coordinate offset vector for each character interaction instance in the training dataset. If there are t instances in the training dataset, then a coordinate offset matrix will be obtained. Principal component analysis (PCA) is used to extract the first k principal component vectors of the coordinate offset matrix X. These k principal component vectors constitute the basis vectors of the spatial relationship between the characters and are used to construct the projection matrix V.

[0062] Steps 1-4: For any instance of human-object interaction, we calculate the coordinate offset in Step 1-2 to obtain the coordinate offset vector x, and then project it into the latent space using the projection matrix V:

[0063] γ=V T (x-μ)

[0064] In the formula, γ is the compressed representation of the coordinate offset of the person and the object, and μ is the mean vector of x, i.e.

[0065] Through the inverse projection process, we can restore γ to an approximate form of coordinate offset, thus obtaining the implicit vector of spatial relationship.

[0066]

[0067] Since each instance of human interaction in the training dataset corresponds to one image, the spatial relationship latent vector between people and objects in each image can be obtained through the above steps. These spatial relationship implicit vectors It was used as an auxiliary label to train the network.

[0068] The specific technical solution used to predict the posterior probability distribution between people and objects from input images is as follows:

[0069] Step 2-1: Use a deep convolutional network to encode the input image into a visual feature vector c, which contains high-level visual semantic information of the input image.

[0070] Step 2-2: Use a multilayer perceptron model to extract the camera's extrinsic parameters (rotation angle R) from the visual feature vector c. global Translation vector T global) and the shape parameter β of the human body mesh model SMPL init .

[0071] Steps 2-3: Using the normalized flow with the visual feature vector c as a condition, sample z from the normal distribution θ ~N(0, I) is transformed into human pose θ, thereby constructing the probability distribution p of human pose θ. Θ|I (θ|c), that is:

[0072] θ=f SMPL (z θ |c)

[0073] Steps 2-4: Using another normalized flow model conditioned on the visual feature vector c and the human pose θ, the samples z taken from the normal distribution are... γ ~N(0,I) is transformed into the spatial relationship vector γ between people and objects, thereby constructing the probability distribution p of the spatial relationship between people and objects. Γ|I；Θ (γ|c, θ), that is:

[0074] γ=f distance (z γ |c,θ)

[0075] Steps 2-5: Sample the data with the highest probability density γ from the spatial relationship distribution of people and objects. * :

[0076]

[0077] Based on steps 1-4, the coordinate offset vector x of the person and the object can be obtained. * =Vγ * +μ, from the coordinate offset vector x * Extract the coordinate offset vector D = {d} between the human anchor point and the object anchor point according to the corresponding dimension. i,j}, i=1,...,m,n=1,...,j.

[0078] Steps 2-6: Select the pose with the highest probability density from the human pose probability distribution. Using θ init and β obtained in step 2-2 init As the initialization of the human body mesh model SMPL, the following formula is then optimized to obtain the relative rotation matrix R of the object. init and relative translation coordinates t init :

[0079]

[0080] Among them, the offset loss L rel Defined as:

[0081]

[0082] In the formula, This represents the three-dimensional coordinates of the i-th anchor point on the surface of the SMPL mesh model, which is determined by the pose parameter θ and the shape parameter β. Let represent the three-dimensional coordinates of the j-th anchor point on the surface of the object mesh model, which takes the rotation matrix R and the coordinate translation vector t as parameters.

[0083] Steps 2-7: Use the SMPL model to calculate θ init and β init Convert to a human body mesh model, and use the rotation matrix R init Translation vector t init The template model of the object is applied to obtain the 3D position of the object's mesh model. This completes the initial prediction from a single image to a human and object mesh model.

[0084] The specific technical solution used to optimize the final result after applying character offset loss and reprojection loss is as follows:

[0085] Step 3-1: Using the currently mature methods OpenPose for extracting human skeletons and Epro-pnp for monocular 6D object pose estimation, extract the coordinate correspondence map of human key nodes and objects from the input image.

[0086] Step 3-2: Optimize the following loss function to obtain the optimal solution (θ) * ,β * R * , t * ):

[0087]

[0088] The definitions of each loss in the above formula are as follows: It is the loss of key points in the human body, that is

[0089]

[0090] in, These are the three-dimensional coordinates of human joints. The key is the corresponding two-dimensional coordinates on the image. It is the projection equation of the camera.

[0091] It is the 2D-3D coordinate projection loss of the object, and its specific form is as follows:

[0092]

[0093] in, These are the three-dimensional coordinates of the object. These are the two-dimensional coordinates of the object predicted by the network. These are the two-dimensional coordinate weights predicted by the network.

[0094] The offset loss is consistent with that in steps 2-6.

[0095] Defined as posterior loss, its specific form is:

[0096]

[0097] λ J , λ coor , λ rel and λ posteriori These are the weight parameters.

[0098] This invention proposes a person reconstruction method based on a layered normalized flow model. This method uses person offsets to explicitly characterize the 3D spatial relationship between people and objects, and extracts this spatial relationship from images to aid in subsequent optimization steps of 3D person reconstruction. The technology disclosed in this invention can learn the spatial relationship between people and objects from data and can handle cases of severe occlusion between people and objects. The proposed method is based on the publicly available BEHAVE...

[10] InterCap

[11] The method was compared with previous methods on the dataset, and it surpasses previous algorithms in both reconstruction accuracy and runtime efficiency. The method disclosed in this invention has important research reference value for subsequent human-object reconstruction algorithms, multi-person reconstruction algorithms, and scene reconstruction algorithms, and also has wide application value in human-computer interaction, game entertainment, and behavior understanding.

[0099] StackFLOW neural network model is a monocular reconstruction algorithm based on human coordinate offset. The algorithm uses a normalized flow model to extract the three-dimensional spatial relationship between people and objects from the input image, and uses coordinate offset to constrain the spatial relationship between people and objects in the post-optimization step, outputting the pose and shape parameters of the human SMPL model and the rotation matrix and offset vector of the object.

[0100] This embodiment builds, trains, and validates a model on the BEHAVE dataset based on this idea, and further explains the above technical solution, specifically including the following steps:

[0101] Step 1: Selection of anchor points for the human body and objects; Randomly select 32*22=704 anchor points from the surface of the SMPL human body mesh model to form the human body anchor point set A. SMPL 32 anchor points are selected from the template surface of each object to form object anchor point set A. objectThese anchor points remain consistent for any person-object interaction instance, and they will be used in subsequent steps to calculate the coordinate offset between the person and the object.

[0102] Step 2: Construct the projection matrix; The BEHAVE dataset provides labels for the human and object 3D meshes corresponding to each image. We call a pair of people and objects appearing in the same image a person-object interaction instance. Traverse the entire training dataset, calculate the coordinate offsets between the anchor points of the 3D mesh model in each person-object interaction instance, and connect these offsets in an ordered manner according to the specified dimensions to form an offset vector with dimensions 704*32*3=67584. Calculate all offset vectors in the dataset and form an offset matrix. Use PCA to extract the first 32 principal components of this offset matrix, and use these 32 principal vectors to construct the projection matrix along the column direction.

[0103] Step 3: Obtain the 3D person-object relationship vector; For each person-object interaction instance in the dataset, calculate its offset vector and use the projection matrix to reduce its dimension from 67584 to 32 dimensions, thus obtaining a 32-dimensional person-object 3D relationship vector. This vector describes the 3D spatial positional relationship between the human body and the object. Since each person-object interaction instance in the dataset corresponds to an image, the spatial relationship latent vector between the person and the object in the image can be obtained through the above steps for each image. These latent vectors are used as auxiliary labels to train the network.

[0104] Step 4: Construct the network model structure; construct a 50-layer residual network, maintaining the same parameters as the standard ResNet-50. Construct a perceptron model with two layers of neurons, whose internal dimensions are 2048-1024-13. Construct two normalized flow models: one for human pose with an input-output dimension of 144 and a conditional vector dimension of 2048; the other for modeling the probability distribution of spatial relationships between people, with an input-output dimension of 32 and a conditional vector input dimension of 2048. And follow... Figure 3 The specified data stream connects these network models.

[0105] Step 5: Preprocess the training data; randomly select a sample from the training dataset. This sample contains an RGB image I, the camera parameter matrix K of the image, and the pose and shape parameters θ and β of the corresponding human SMPL model, as well as the rotation matrix R and translation vector t of the object. Crop and normalize the input image to form a 224x224x3 tensor. Calculate the 22 key points in the SMPL model, and use the camera parameter matrix to calculate the 2D key points projected onto the image from these 3D key points. Finally, calculate the latent vector of the human-object spatial relationship as in Step 3. Each sample contains information including an image tensor. SMPL model parameters Two-dimensional key nodes of human SMPL Implicit Vector of Human-Object Spatial Relationship

[0106] Step 6: Build training data; repeat step 5 to obtain 32 samples to form a batch of data.

[0107] Step 7: Input the batch data into the neural network model; input the batch data obtained in Step 6 into the neural network model constructed in Step 4. The specific data flow is as follows: The image tensor I is first fed into ResNet-50 and encoded into a visual vector c. This visual vector is processed through two layers of MLP to obtain the initial shape parameter β of SMPL and the camera coordinate offset vector t. Two samples z1 and z2 are randomly sampled from the normal distribution of human pose, and the sample z0 = 0 with the highest probability density is also randomly sampled. These three samples are input into the normalized flow model of human pose, using visual features as conditional vectors, to obtain the human pose samples θ0, θ1, and θ2. Two samples z1' and z2' are randomly sampled from the normal prior distribution of spatial relationships of people, and the sample z0' = 0 with the highest probability density is also randomly sampled. These three samples are input into the normalized flow model of spatial relationships of people, using visual features and human pose θ0 as conditional vectors, to obtain the spatial relationship samples γ0, γ1, and γ2.

[0108] Step 8: Calculate the loss. The training loss is as follows:

[0109] L train =λ θ L θ +λ γ L γ +λ NLL L NLL +λ J L J

[0110] Calculate the loss for human pose samples:

[0111] L θ =|θ0-θ gt |1+λexp (|θ1-θ gt |1+|θ²-θ gt |1)

[0112] Where, θ gt It is the annotation label of human pose in the dataset, λ exp Take 0.1.

[0113] Calculate the loss for the relationship vector samples:

[0114] L γ =|γ0-γ gt |1+λ exp (|γ1-γ gt |1+|γ2-γ gt |1)

[0115] Where, γ gt It is a spatial relationship vector of people calculated based on the labels in the dataset.

[0116] Calculate the negative log-probability loss in a normal flow:

[0117] L NLL =-lnp Γ|I,Θ (γ gt |c,θ gt )-lnp Θ|I (θ gt |c)

[0118] The predicted human pose parameters θ and shape parameters β are input into the SMPL model to obtain the human joint keypoints in SMPL. The 3D joint keypoints of the human body are projected onto the 2D image plane using the camera translation coordinates and camera intrinsic parameters, and the keypoint loss is calculated.

[0119]

[0120] Step 9: Gradient backpropagation; sum all losses according to certain weights to obtain the training loss, backpropagate the loss to calculate the gradient, and update the network parameters according to the gradient.

[0121] Step 10: Network training main loop; calculate the gradient based on the loss calculated in step 13 and backpropagate it, repeat steps 5-13 until the network converges.

[0122] Step 11: Initialize prediction; Given an image, after normalization, input it into the neural network trained in Step 10 to obtain the initialization parameter β. init ,θ initThe latent vector γ0 corresponding to the spatial relationship of the character and sample z0' is obtained. Using the projection matrix obtained in step 2, γ0 is back-projected to obtain the character anchor point offset. Finally, the following formula is optimized to obtain the initialization results of the object rotation matrix and coordinate offset vector.

[0123]

[0124] in, It is the i-th anchor point on the surface of the SMPL model. It is the j-th anchor point on the object's surface. It is the coordinate offset between the corresponding anchor points of the characters, which is obtained by decoding from the character relationship vector γ0.

[0125] Step 12: Extract 2D information; use Openpose to extract 2D key nodes of the human body from the input image. 2D Epro-PnP is used to extract the 2D-3D coordinates and weights x of the object from the input image. 3D ,x 2D ,w 2D .

[0126] Step 13: Post-optimization and fine-tuning of results; combining the initial prediction results β obtained in Step 11. init ,θ init ,R init ,t init The two-dimensional information J obtained in step 12 2D ,x 3D ,x 2D ,w 2D Based on the stacked normalized model trained in step 10, optimize the following loss to fine-tune the results.

[0127]

[0128] in It is the two-dimensional projection loss of the SMPL joints, which is defined as

[0129]

[0130] The 2D-3D reprojection loss of an object is defined as follows:

[0131]

[0132] It is the posterior distribution loss of human posture.

[0133]

[0134] It is the posterior distribution loss of spatial relationships between characters.

[0135]

[0136] It is the character anchor point offset loss, which is defined as

[0137]

[0138] Use the Adam optimizer to obtain the fine-tuned result.

[0139]

[0140] Step 14: Validate the model; run steps 11-13 for each image in the BEHAVE validation set and calculate the Chamfer index between people and objects. The indexes we obtained in the experiment were 4.61 (SMPL) and 9.86 (object).

[0141] The above examples demonstrate improved reconstruction accuracy and post-optimization efficiency on the BEHAVE dataset compared to previous methods.

[0142] The references cited in this specification are as follows:

[0143] [1] Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from asingle image in the wild. In European Conference on Computer Vision (ECCV), page 34–51, 2020.

[0144] [2]Xianghui Xie,Bharat Lal Bhatnagar,and Gerard Pons-Moll. Chore:Contact,human and object reconstruction from a single rgb image.In European Conference on Computer Vision(ECCV),page 125–145,October 2022.

[0145] [3]Korrawe Karunratanakul,Jinlong Yang,Yan Zhang,Michael J.Black,Krikamol Muandet,and Siyu Tang.Grasping field:Learning implicitrepresentations for human grasps.In 2020International Conference on 3D Vision(3DV),pages333–344,2020.

[0146] [4]Bo Wan,Desen Zhou,Yongfei Liu,Rongjie Li,and Xuming He.Pose-awaremulti-level feature network for human object interaction detection.InProceedings of the IEEE / CVF International Conference on Computer Vision(ICCV),pages9469–9478,October2019.

[0147] [5]Yixin Chen,Siyuan Huang,Tao Yuan,Siyuan Qi,Yixin Zhu,and Song-ChunZhu.Holistic++scene understanding:Single-view 3d holistic scene parsing andhuman pose estimation with human-object interaction and physicalcommonsense.In Proceedings of the IEEE / CVF International Conference onComputer Vision(ICCV),pages 8648–8657,October 2019.

[0148] [6]Manolis Savva,Angel X.Chang,Pat Hanrahan,Matthew Fisher,andMatthias Nieβner.PiGraphs:Learning Interaction Snapshots fromObservations.ACM Transactions on Graphics(TOG),35(4),2016.

[0149] [7]Nikos Kolotouros,Georgios Pavlakos,Dinesh Jayaraman,and KostasDaniilidis.Probabilistic modeling for human mesh recovery.In Proceedings ofthe IEEE / CVF International Conference on Computer Vision(ICCV),pages11605–11614,October 2021.

[0150] [8]Danilo Rezende and Shakir Mohamed.Variational inference withnormalizing flows.In Francis Bach and David Blei,editors,Proceedings ofthe32nd International Conference on Machine Learning,volume 37 of Proceedings ofMachine Learning Research,pages 1530–1538,Lille,France,07–09 Jul 2015.PMLR.

[0151] [9]Hansheng Chen,Pichao Wang,Fan Wang,Wei Tian,Lu Xiong,and HaoLi.Epro-pnp:Generalized end-to-end probabilistic perspective-n-points formonocular object pose estimation.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR),pages 2781–2790,June 2022.

[0152]

[10] Bharat Lal Bhatnagar,Xianghui Xie,Ilya A.Petrov,CristianSminchisescu,Christian Theobalt,and Gerard Pons-Moll.Behave:Dataset andmethod for tracking human object interactions.In Proceedings ofthe IEEE / CVFConference on Computer Vision and Pattern Recognition(CVPR),pages 15935–15946,June 2022.

[0153]

[11] Yinghao Huang,Omid Taheri,Michael J.Black,and DimitriosTzionas.Intercap:Joint markerless 3d tracking ofhumans and objects ininteraction.In Pattern Recognition,pages 281–299.Springer InternationalPublishing,2022.

Claims

1. A monocular character reconstruction method based on a stacked normalized flow model, characterized in that, Includes the following steps: Step 1: Obtain the latent vector of the spatial relationship between people and objects in the image corresponding to each human interaction instance in the training dataset. Implicit vectors of spatial relationships As auxiliary labels, the neural network model is trained, where a pair of people and objects appearing in the same image are defined as human interaction instances. Step 2: Construct and train a neural network model to extract spatial relationships between people and objects. This neural network model predicts the posterior probability distribution of spatial relationships between people and objects from the input image. Its specific implementation includes the following steps: Step 2-1: The neural network model encodes the input image into a visual feature vector. ; Step 2-2: Use a multilayer perceptron model to extract visual feature vectors Extracting the extrinsic parameters of the camera and the shape parameters of the SMPL model ; Steps 2-3: Use the normalized flow model with visual feature vectors Given the condition, samples will be drawn from a normal distribution. Transform into human posture Thus constructing human posture probability distribution ; Steps 2-4: Use Normalized Flow Model 2 with visual feature vectors and human posture As a condition, samples will be taken from a normal distribution. Transform into a spatial relationship vector between people and objects This allows us to construct the probability distribution of spatial relationships between people and objects. ; Steps 2-5: Sample the data with the highest probability density from the spatial relationship distribution of people and objects. Further basis From the coordinate offset vector Extract the coordinate offset vector between the human body anchor point and the object anchor point according to the corresponding dimension. ,in, For the projection matrix, for The mean vector; Steps 2-6: From the perspective of human posture Choose the pose with the highest probability density from the probability distribution. ,use and the shape parameters obtained in step 2-2 As the initialization of the human body mesh model SMPL, the following formula is then optimized to obtain the relative rotation matrix of the object. and relative translation coordinates : in, The offset loss is shown in the following formula: In the formula, The surface of the SMPL mesh model is represented by the first... The SMPL mesh model consists of the three-dimensional coordinates of each anchor point and the pose parameters. and shape parameters Decide, Represents the surface of the object mesh model. The three-dimensional coordinates of each anchor point, the object mesh model, are represented by a rotation matrix. And coordinate translation vector For parameters; Steps 2-7: Use the SMPL model to... and Convert to a human body mesh model, and then use the rotation matrix. Translation vector The 3D position of the object's mesh model is obtained by applying a template model to the object; Step 3: Use the character offset loss and reprojection loss to perform post-optimization on the results of the neural network model established in Step 2. During model training, calculate the gradient of the calculated loss and backpropagate it until the model converges, and finally train the stacked normalized flow model.

2. The monocular character reconstruction method based on a stacked normalized flow model as described in claim 1, characterized in that, Step 1 further includes the following steps: Step 1-1: Randomly sample from the surface of the human body mesh model SMPL These points constitute the human body anchor point set. Random sampling from the surface of different object mesh models These points constitute the object's anchor point set. ; Step 1-2: For each person and object interaction instance in the training dataset, calculate all... Personal body anchor points and The relative anchor point coordinate offset between the anchor points of each object is obtained. These coordinate offsets are connected to form a coordinate offset vector. ; Steps 1-3: Set the coordinate offset vectors of all character interaction instances Forming a coordinate offset matrix Extract the coordinate offset matrix The former Each principal component vector, using this The projection matrix is ​​constructed from the principal component vectors. ; Steps 1-4: For any instance of character interaction where a person interacts with an object, use the projection matrix. Project it into the hidden space: In the formula, It is a compressed representation of the coordinate offsets of people and objects. yes The mean vector; Then, through the inverse projection process, Reducing it to an approximate form of coordinate offsets yields the implicit vector of spatial relationships. , .

3. The monocular character reconstruction method based on a stacked normalized flow model as described in claim 2, characterized in that, In steps 1-2, the first Personal body anchor point With the Anchor points of objects Relative anchor point coordinate offset between Represented as: , , .

4. The monocular character reconstruction method based on a stacked normalized flow model as described in claim 1, characterized in that, Step 3 further includes the following steps: Step 3-1: Extract the coordinate correspondence map of key human body nodes and objects from the input image; Step 3-2: Optimize the following loss function to obtain the optimal solution. : in: , , as well as These are weight parameters; This refers to the loss of key points in the human body, including: in, These are the three-dimensional coordinates of human joints. These are the two-dimensional coordinates of the corresponding key human body nodes on the image; The loss is the 2D-3D coordinate projection of the object, which includes: in, These are the three-dimensional coordinates of the object. These are the predicted two-dimensional coordinates of the object. These are the predicted two-dimensional coordinate weights; For offset loss; For a posteriori loss, .

Images