Human and scene interaction generation method and system based on geometric and physical constraints

Abstract

The invention provides a human and scene interaction generation method and system based on geometric and physical constraints. Relates to the technical field of human and scene interaction, and aims to solve the problems that an existing method is low in model placement precision, scene interaction is not natural enough, and penetration cannot be solved. A three-dimensional human body model and a three-dimensional scene model are used; defining and generating a semantic tag of the initial human body model, and performing posture recognition on the given human body model based on a posture classifier of a convolutional neural network to optimize the semantic tag; based on the three-dimensional scene model, candidate position anchor points are generated, and the optimal position in the scene is determined by calculating geometric and semantic distances of human body model vertexes with semantic tags on each anchor point position; and based on inverse dynamics, adjusting the posture of the human body model placed at the optimal position in the scene and outputting an interaction result. The problems that the model placement precision is not high, and scene interaction is not natural enough are solved, and the naturalness and accuracy of the interaction scene are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-scene interaction, and in particular to a human-scene interaction generation method and system based on geometric and physical constraints. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The booming development of augmented reality (AR), virtual reality (VR), computer graphics and machine learning has made human-scene interaction have application prospects in game design, human-computer interaction and other aspects. Especially with the introduction of the concepts of metaverse and virtual human in recent years, the generation of interaction between three-dimensional human body and three-dimensional scene has gained widespread attention as a new communication medium between people. In this context, workers engaged in three-dimensional development, such as scene designers, need to generate real interactive scenes. However, traditional methods of capturing key points of human body or manually adjusting human body posture to adapt to the scene often require a lot of cost and time. Therefore, developing a method that can automatically generate interactive scenes can not only improve efficiency and reduce costs, but also ensure the natural interaction and high-accuracy placement of human models in complex scenes, which is an effective way to solve current problems.

[0004] Most existing methods are based on simulated 3D human models, and generate interactive scenes by using the semantic information of the scene and formulating physical rules between models to encourage contact and avoid penetration. Some methods focus on placing existing human models in the scene. The core of these methods is to place pre-created human models into the 3D scene and dynamically adjust the posture to adapt to the diversity of the scene. Some methods focus on generating human models with reasonable postures in a given 3D scene. They use artificial intelligence technology to predict and generate human postures that are suitable for the scene by analyzing the scene content and context information. A recent method, POSA, uses conditional variational autoencoders (cVAE) to encode latent semantic labels on the vertices of human models and then places them in the scene, which can adapt to different scenes. Although these methods have made some progress, they do not consider the vertical relationship between the contact surfaces between models and lack a clear mechanism to avoid penetration between models. This is crucial for generating interactions between human models of different body shapes and postures in complex scenes.

[0005] In daily life, humans interact with the surrounding scenes all the time. For example, when we walk, our feet land on the ground; when we lie down, our bodies touch the bed. These interactions not only follow geometric rationality, but also reflect physical rationality. Specifically, the geometric characteristics emphasize the spatial relationship and position matching between the human body and the scene objects. This means that the human interaction behavior is consistent with the design of daily scenes and objects in terms of spatial configuration and position arrangement. In addition, under the influence of the earth's gravity, the contact surface between humans and scenes presents a vertical position relationship in most cases, which is the result of the earth's gravity pulling the human body toward the surface. This vertical position relationship is a ubiquitous feature in the interaction between humans and scenes and forms a common observation framework in perception and interaction design. However, existing methods often only consider the spatial distance between the human body and the scene, without considering the vertical relationship between the contact surfaces between models, thus limiting the geometric rationality of the interaction. In terms of physics, physical contact between the human body and scene objects is indispensable, which allows us to establish a connection with the surrounding environment through actual touch. A key feature of physical interaction is that there is no penetration between the human body and scene objects, which ensures that the interaction is intuitive and realistic. However, existing methods often rely on the interaction features learned by the model to implicitly reduce the occurrence of penetration, lacking a clear mechanism to solve the penetration phenomenon, thus limiting the physical rationality of the interaction. Therefore, how to improve the naturalness and accuracy of the interaction scene through more sophisticated geometric matching and physical interaction simulation remains a challenging problem. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and system for generating human-scene interaction based on geometric and physical constraints. A three-dimensional human-scene interaction framework is proposed, which uses geometric and physical constraints to place a three-dimensional human model with a given posture at a suitable position in a three-dimensional scene, and at the same time ensures the interactivity between the human body and the scene, solving the problems of low model placement accuracy and unnatural scene interaction.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of the present invention discloses a method for generating human-scene interaction based on geometric and physical constraints, comprising:

[0009] A 3D human body model is used to represent a 3D human body in a given posture, and a 3D scene model is used to represent a scene;

[0010] Define and generate semantic labels for the initial human body model, and perform posture recognition on the given human body model through a posture classifier based on a convolutional neural network to optimize the semantic labels;

[0011] Based on the 3D scene model, candidate position anchor points are generated, and the best position in the scene is determined by calculating the geometric and semantic distances of the human body model vertices with semantic labels at each anchor point position;

[0012] Based on inverse kinematics, the pose of the human model placed at the best position in the scene is adjusted;

[0013] Output the results of human-scene interaction.

[0014] As a further technical solution, a three-dimensional human body model is used to represent a three-dimensional human body in a given posture, wherein the three-dimensional human body model adopts the SMPL-X model and is defined as:

[0015] M h =(P h , V h , L h )

[0016] Among them, P h is the model parameter information represented by the function M(β, θ, ψ, t), V h is the vertex set of the model, L h It is the set of semantic labels of model vertices.

[0017] As a further technical solution, the semantic labels of the initial human body model are defined and generated, specifically:

[0018] Input the model parameter information into the encoder of the conditional variational autoencoder network to extract key features;

[0019] Based on the key features, semantic labels are generated by the decoder.

[0020] As a further technical solution, a posture classifier based on a convolutional neural network is used to perform posture recognition on a given human body model to optimize the semantic label, specifically:

[0021] Input the model parameter information into the convolutional neural network for posture classification;

[0022] By constructing a loss function, the posture classification is optimized;

[0023] The contact range is adjusted according to the posture classification results, and the semantic labels within the contact range are retained.

[0024] As a further technical solution, candidate position anchor points are generated based on the 3D scene model, specifically:

[0025] Divide the objects in the 3D scene model into multiple regions, and the scene model vertices in the same region have the same semantic label;

[0026] Calibrate the diagonal boundary position of the scene to indicate the scope of the scene;

[0027] In the scene range, obtain candidate anchor points at a given distance;

[0028] Use the distance field to detect whether the candidate anchor point collides with the scene. If not, the candidate anchor point is inside the nearest scene object and is removed.

[0029] If there is a collision, the candidate anchor point is outside the scene object closest to it and serves as the candidate position anchor point.

[0030] As a further technical solution, the best position in the scene is determined by calculating the geometric and semantic distances of the human model vertices with semantic labels at each anchor position, specifically:

[0031] Project each vertex in the human body model vertex set along the Y-axis direction of the world coordinate system to obtain the scene model vertex closest to it;

[0032] The human body model vertices and their corresponding scene model vertices form human body-scene vertex pairs;

[0033] Calculate the geometric distance between human-scene vertex pairs;

[0034] Calculate the semantic distance between human-scene vertex pairs;

[0035] Calculate the total distance between all vertex pairs through geometric distance and semantic distance;

[0036] Based on the total distance, the optimal position of the human model in the scene is calculated.

[0037] As a further technical solution, based on inverse dynamics, the posture of the human model placed at the best position in the scene is adjusted, specifically:

[0038] Based on the human body model at the best position in the scene, the human body parts and interaction points to be adjusted are determined by constructing a heterogeneous kinematic tree and performing collision detection;

[0039] Based on the determined interaction points, inverse dynamics is used to calculate the rotation information of the human body's joints and update and adjust them.

[0040] The second aspect discloses a human-scene interaction generation system based on geometric and physical constraints, comprising:

[0041] A human body model and scene model setting module, used to use a three-dimensional human body model to represent a three-dimensional human body of a given posture, and use a three-dimensional scene model to represent a scene;

[0042] The posture-constrained human contact semantic generation module is used to define and generate the semantic labels of the initial human body model, and perform posture recognition on the given human body model through a posture classifier based on a convolutional neural network to optimize the semantic labels;

[0043] A 3D human model placement module is used to generate candidate position anchor points based on the 3D scene model, and determine the best position in the scene by calculating the geometric and semantic distances of the human model vertices with semantic labels at each anchor point position;

[0044] A posture adjustment module, used to adjust the posture of the human model placed at the best position in the scene based on inverse dynamics;

[0045] The human-scene interaction output module is used to output the human-scene interaction results.

[0046] A third aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0048] One or more of the above technical solutions have the following beneficial effects:

[0049] In this embodiment, based on the three-dimensional human body and scene interaction framework, geometric and physical constraints are used to achieve the placement of the human body model and scene interaction, which solves the problems of low model placement accuracy and unnatural scene interaction. The human body model can be placed at a suitable position in the scene, and physically and semantically credible interactions between people and scenes are generated, thereby improving the naturalness and accuracy of the interactive scene.

[0050] In this embodiment, a method for generating human body contact semantics taking into account posture constraints is proposed to ensure the semantic accuracy of the physical contact and interaction area between the human body and the scene; a method for placing a three-dimensional human body model combining geometry and semantics is proposed to realize the placement of the three-dimensional human body model in the scene; an inverse dynamics posture adjustment method is proposed to realize the interaction of the three-dimensional human body model in the scene; and through the inverse dynamics posture adjustment and collision detection technology, it is ensured that there is no penetration between human bodies.

[0051] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0053] Figure 1 A flow chart of generating a three-dimensional human body and scene interaction framework according to the first embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a method for generating human body contact semantics considering posture constraints according to a first embodiment of the present invention;

[0055] in, Figure 2 (a) is the contact vertex range labeling result of different postures in the first embodiment of the present invention, Figure 2 (b) is the human body contact semantic generation process of the first embodiment of the present invention, and the same color represents the same semantics;

[0056] Figure 3 A schematic diagram of a human joint point representation and a corresponding heterogeneous kinematic tree according to a first embodiment of the present invention;

[0057] in, Figure 3 (a) is a demonstration of human joint points in the first embodiment of the present invention, with arrows pointing from a parent node to its child node. Figure 3 (b) is a heterogeneous kinematic tree structure according to the first embodiment of the present invention, and the nodes in the dashed box are operable nodes;

[0058] Figure 4 This is a solution example based on inverse dynamics according to the first embodiment of the present invention;

[0059] in, Figure 4 (a) is an example of the decoupled components of the torsional rotation and the swinging rotation of the forearm in the first embodiment of the present invention, Figure 4 (b) is a solution example of the first embodiment of the present invention, the human body area where the collision occurs and its corresponding collision subtree marked with a dotted box, and the posture after the leaf node of each collision subtree is moved to the collision point position through inverse dynamics solution. DETAILED DESCRIPTION

[0060] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0061] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0062] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0063] The overall idea proposed in the present invention is as follows: first, a three-dimensional human body model and a three-dimensional scene model are defined; then, a method for generating human body contact semantics taking posture into consideration is proposed, which generates initial human body semantic labels and classifies the given human body postures to optimize the results; then, a method for placing human body models by combining geometric and semantic constraints is proposed, which determines the appropriate position of the human body model in the scene by constraining the geometric proximity and semantic consistency between models; finally, a posture adjustment method based on inverse dynamics is proposed, which finds the target interaction point between the person and the scene and solves the rotation matrix of the joint points of the human body model after the scene interaction, thereby determining the human body posture after the scene interaction, and then generating a three-dimensional interactive scene.

[0064] Embodiment 1

[0065] This embodiment discloses a method for generating human-scene interaction based on geometric and physical constraints.

[0066] In order to more clearly illustrate this embodiment, the process of generating task construction schedule planning based on human-scene interaction with geometric and physical constraints can be specifically described as follows:

[0067] A method for generating human-scene interactions based on geometric and physical constraints is provided, including:

[0068] S1. Use a 3D human body model to represent a 3D human body in a given posture, and use a 3D scene model to represent a scene;

[0069] S2, define and generate semantic labels of the initial human body model, and perform posture recognition on the given human body model through a posture classifier based on a convolutional neural network to optimize the semantic labels;

[0070] S3, based on the three-dimensional scene model, generate candidate position anchor points, and determine the best position in the scene by calculating the geometric and semantic distances of the human body model vertices with semantic labels at each anchor point position;

[0071] S4. Based on inverse dynamics, the posture of the human model placed at the best position in the scene is adjusted;

[0072] S5. Output the results of the interaction between people and scenes.

[0073] like Figure 1 As shown, in step S1, a three-dimensional human body model is used to represent a three-dimensional human body in a given posture, and a three-dimensional scene model is used to represent a scene.

[0074] S11. Use a three-dimensional human body model to represent a three-dimensional human body in a given posture.

[0075] In this embodiment, the SMPL-X model is used to represent a three-dimensional human body. SMPL-X is a human body generation model that combines face, hands and body, and realizes the simulation of human body shape and posture changes through vertex linear blending skinning and blend deformation.

[0076] The SMPL-X model has 10475 vertices and 55 joints, which can be represented by the function M(β, θ, ψ, t). is the shape parameter, which is used to represent the shape of the model; θ=(θ b ,θ f ,θ h ) is the posture parameter, which consists of three parts. is the body posture parameter, are the facial (chin, left eye, right eye) posture parameters, are the hand posture parameters, all of which are expressed in axis-angle form; is the expression parameter; t=(t p , t r ) is the displacement parameter, which consists of two parts, is the location parameter of the model, are the rotation parameters of the model, which are all represented by Euler angles.

[0077] In this embodiment, the SMPL-X human body model is defined as M h =(P h , V h , L h ), where P h is the model parameter information represented by the function M(β, θ, ψ, t), V h is the vertex set of the model, L h It is the set of semantic labels of model vertices.

[0078] S12. Use a three-dimensional scene model to represent the scene.

[0079] Given a 3D scene model, define it as M s =(V s , L s ). Among them, V s is the vertex set of the scene model, L s is the semantic label of each vertex of the scene model.

[0080] The objects in the 3D scene model are divided into multiple regions. The scene model vertices in the same region have the same semantic label, and the diagonal boundary position C of the scene is calibrated at the same time. min and C max To represent the scope of the scene, they are all represented by three-dimensional coordinates (x, y, z).

[0081] like Figure 1 As shown, in step S2, the semantic labels of the initial human body model are defined and generated, and the posture recognition of the given human body model is performed through a posture classifier based on a convolutional neural network to optimize the semantic labels.

[0082] In order to more accurately represent the geometric relationship between models, we use human contact semantic information to describe it. Figure 2 (b) As shown. First, we use the POSA model based on the conditional variational autoencoder (cVAE) network to generate a human body model M for a given posture h The human semantic label L h Then, we use a posture classifier based on convolutional neural network (CNN) to perform more accurate semantic label range division according to human posture. Finally, based on the semantic label range constrained by human posture, optimized semantic information can be obtained.

[0083] S21. Define and generate semantic labels for the initial human body model.

[0084] The model parameter information is input into the encoder of the conditional variational autoencoder network to extract key features, and semantic labels are generated through the decoder based on the key features.

[0085] In this embodiment, for a model vertex Human semantic label Defined as a one-hot encoding, the formula is:

[0086]

[0087] Among them, N o is the number of object categories.

[0088] For a human model M h , its P h The β,θ parameters under θ are input to the encoder of the cVAE network to extract key features. The encoder maps these parameters to the latent space z, and then the decoder generates a detailed feature map f based on z and conditional information. map By minimizing the objective function combining reconstruction loss and KL divergence, the network learns the mapping of human pose and shape to interaction features, and generates semantic labels L through the decoder. h Since the network uses variational inference techniques to estimate the distribution in the latent space and obtains random variables through sampling, it can sample many possible feature maps for a given pose.

[0089] S22. Perform posture recognition on a given human body model through a posture classifier based on a convolutional neural network to optimize semantic labels.

[0090] S221. Input the model parameter information into the convolutional neural network for posture classification, and optimize the posture classification by constructing a loss function.

[0091] In this embodiment, different human postures often lead to changes in the body parts that may contact the scene. For example, when we are standing, our feet will contact the ground, but our back and thighs generally will not contact the object. Based on this interaction feature, we generate the semantic label L h Specifically, we input the β,θ parameters into the CNN network for posture classification, and the loss function can be expressed as cross entropy;

[0092]

[0093] Among them, N p is the number of posture categories. According to the real interaction scenarios, postures are divided into four categories: sitting, standing, lying down and others. i is the true label, a i is the Softmax probability of the i-th category.

[0094] After the above steps, the semantic labels are optimized through the loss function to achieve improvement of the semantic labels.

[0095] S222: Adjust the contact range according to the posture classification result, and retain the semantic labels within the contact range.

[0096] Following Hassan et al., we annotate a set of vertices that frequently touch the world As the contact vertex range of the human body interacting with the scene. Specifically, the contact vertex range is defined as: 786 hand vertices, 66 elbow vertices, 204 back vertices, 94 hip vertices, 62 thigh vertices and 194 foot vertices. Based on this, the contact range V c Adjust the range according to the posture classification results and keep only V c The semantic tag L h The contact vertex range annotation results of different postures are as follows: Figure 2 (a) shown.

[0097] After the above steps, the generated semantic labels provide detailed semantic information for the vertices of the human body model to interact with specific objects in the scene, and are optimized according to the posture of the human body, preparing for the subsequent placement of the three-dimensional human body model.

[0098] like Figure 1As shown, in step S3, candidate position anchor points are generated based on the three-dimensional scene model, and the best position in the scene is determined by calculating the geometric and semantic distances of the human body model vertices with semantic labels at each anchor point position.

[0099] The semantic labels generated from step S2 provide detailed semantic information about the interactions between the vertices of the human model and specific objects in the scene, which are optimized according to the human body's posture. We propose a joint geometric and semantic 3D human model placement method. First, we discretize the scene into a series of uniformly distributed candidate anchor points, which represent potential locations where the human model may interact with the scene. Then, we calculate the geometric and semantic distances of the human model vertices under each anchor point, and select the best anchor point position by constraining the geometric proximity and semantic consistency between models, so as to achieve a suitable position of the human model in the scene.

[0100] S31. Generate candidate position anchor points based on the three-dimensional scene model.

[0101] S311 . Calibrate the diagonal boundary position of the scene to indicate the range of the scene.

[0102] For a given scene, we discretize it into a series of uniformly distributed candidate anchor points. First, according to the diagonal boundary position C of the scene, min and C max Determine the range of the entire scene, within which we get the given spacing l - A series of uniformly distributed candidate anchor points X0=(x i , i = 1, ..., n), we uniformly sample 12 different directions parallel to the ground plane to construct candidate directions.

[0103] S312: Obtain candidate anchor points at a given distance within the scene range.

[0104] We then use the Signed Distance Field (SDF) to detect x for all generated anchor candidates. i With scene M s Specifically, the distance field is represented by a uniform voxel grid of size 256×256×256, and each voxel unit v i Store its center c i The nearest normal vector is The scene vertex Therefore, we define the signed distance field SDF as:

[0105]

[0106] S313, using the conforming distance field to detect whether the candidate anchor point collides with the scene; if not, the candidate anchor point is inside the scene object closest to it, and is removed;

[0107] If there is a collision, the candidate anchor point is outside the scene object closest to it and serves as the candidate position anchor point.

[0108] Formula (3) calculates a positive value representing the candidate anchor point x i Outside the nearest scene object, if a negative value is calculated, it means that the candidate anchor point x i The nearest scene object is removed. Finally, the remaining candidate anchor points x i The set is denoted by X.

[0109] After the above steps, the geometric and semantic distances of the vertices of the human body model under each anchor point are calculated to determine the optimal position of the model for a given posture in the global scene.

[0110] S32, determining the best position in the scene by calculating the geometric and semantic distances of the human body model vertices with semantic labels at each anchor point position.

[0111] In this embodiment, the geometric and semantic distances of the vertices of the human body model under each anchor point are calculated to determine the optimal position of the model for a given posture in the global scene.

[0112] S321. Project each vertex in the human body model vertex set along the Y-axis direction of the world coordinate system to obtain the scene model vertex closest to it.

[0113] The root point of the skeleton hierarchy of the 3D human model is taken as the center of the human model, that is, the coordinate origin. Considering the vertical relationship between the human body and the scene contact surface, when the coordinate origin of the human body model is any candidate anchor point x i When we model the contact vertex set V c Each vertex in Project along the Y axis of the world coordinate system to obtain the scene vertex closest to it

[0114] S322, the human body model vertices and their corresponding scene model vertices form human body-scene vertex pairs, calculate the geometric distance between the human body-scene vertex pairs, and calculate the semantic distance between the human body-scene vertex pairs.

[0115] For any There is only one Correspondingly, it is recorded as a human-scene vertex pair Then calculate the geometric distance d between the vertex pairs v , the formula is:

[0116]

[0117] Here, |P| represents the number of vertex pairs.

[0118] For each vertex pair, the human semantic label L h and scene semantic label L s , to determine whether they have the same semantics, using the semantic distance d s It is expressed as:

[0119]

[0120] Where I(·) represents the indicator function. If the function condition is met, the value of the indicator function is 1; otherwise, the value is 0.

[0121] S323, calculating the total distance between all vertex pairs through geometric distance and semantic distance;

[0122] In this embodiment, d between all vertex pairs is calculated. v and d s The sum of , that is, the total distance d, is:

[0123]

[0124] Where λ1 and λ2 are the geometric distances d v and semantic distance d s , and λ1+λ2=1.

[0125] S324: Based on the total distance, calculate the optimal position of the human body model in the scene.

[0126] The total distance d measures the distance between the current candidate anchor point x i Lower human body model M h For the scene model M s The smaller the d, the better the suitability. i Next M h With M s The closer they are physically and semantically, the smaller the current position x is at the minimum distance. i Then the best coordinate point position x is obtained. * , the formula is:

[0127]

[0128] Among them, function f represents the function of determining the optimal position of the human body model in the scene. Specifically, f represents the optimal position of the human body model in the scene at the current candidate anchor point x. i Next, human model M h For the scene model M sThe total distance d is calculated as shown in Algorithm 1.

[0129]

[0130]

[0131] Algorithm 1 summarizes the steps of combining vertex geometry and semantic distance to find the best position. In the algorithm, the function FindNearestVertex(·) is used to represent the given Get the scene vertex that is closest to it when projected along the Y axis of the world coordinate system

[0132] Through the above steps, the placement of the human body model is optimized, the human body model can be placed at a suitable position in the scene, and the three-dimensional human body model can be accurately placed in the scene.

[0133] like Figure 1 As shown, in step S4, based on inverse kinematics, the posture of the human body model placed at the best position in the scene is adjusted.

[0134] For the human model placed at the best position in the scene, the posture of the current human model is finely adjusted to achieve a more natural interaction between the human body and the scene. To this end, we adopt a step-by-step strategy. First, by combining heterogeneous kinematic trees and collision detection techniques, we identify the human body parts that need to be adjusted and their potential interaction points. Second, we use an inverse kinematics-based method to accurately adjust the identified human body parts to facilitate the interaction between the human model and the scene.

[0135] S41. Based on the human body model at the best position in the scene, the human body parts and interaction points to be adjusted are determined by constructing a heterogeneous kinematic tree and performing collision detection.

[0136] According to the attitude parameter θ b The human body model M h Semantic segmentation is performed to obtain 21 different part ranges K = {K1, ..., K 21}. Any vertex on the human body model According to the location of its vertices, it is classified into the corresponding human body parts K i , and each K i Corresponding to a joint point B i ={t p , t r}. is the joint point position parameter, are the rotation parameters of the joints, all of which are expressed by Euler angles.

[0137] Human Model Mh The part range K and its corresponding joint point B can form a heterogeneous dynamic tree according to the joint hierarchy relationship. Figure 3 shown.

[0138] Calculate M by formula (3) h With M s Is there a collision? If the calculated value is close to 0, it means that the human body vertex is in contact with the surface of the scene. Therefore, for the joint point B of the part that needs to be adjusted i , we need to find the interaction points that need to be adjusted according to the human posture tree. Specifically, find the interaction points in the human posture tree with B i Leaf node B of the collision subtree of the parent node leaf , as the interaction point for posture adjustment. For example: if the left knee, left ankle, and left foot collide with the scene, the left foot needs to be placed at the collision point to ensure that the "penetration" phenomenon does not occur. In addition, if there are multiple collision points, multiple collision subtrees may be generated, so there will eventually be multiple pairs of human-scene interaction points.

[0139] S42. Based on the determined interaction points, inverse dynamics is used to calculate the rotation information of the human body's degree of freedom joints and update and adjust them.

[0140] After determining the target interaction points of the human body, in order to achieve a more accurate interaction posture, we use inverse kinematics to calculate the rotation information of the 3 degrees of freedom (DoFs) joints of the human body.

[0141] Inverse kinematics is the reverse process of forward kinematics (FK). In the forward kinematics system, the transformation of the child joint is controlled by its parent joint. The transformation of the bottom-level child joint can be determined from top to bottom. The inverse kinematics system first determines the transformation of the bottom-level child joint, and reverses the transformation of each parent joint from bottom to top. It can be expressed as:

[0142] R=IK(B,T)∈SO(3) (8)

[0143] Among them, R represents the rotation matrix of each node, which is represented by the three-dimensional rotation Lie group SO(3); B and T represent the initial posture and target posture respectively, both represented by the three-dimensional position of each node.

[0144] By calculating the rotation matrix R, the position of the end effector of the model can be made close to the target position T. R is considered to be a relative rotation applied on the basis of the initial posture B. In this way, by adjusting the joint angles, the posture at the target position T can be achieved.

[0145] We follow the method of Li et al. and decompose the rotation matrix R into the twist rotation R twistand swing rotation R swing , the swing range determines the end position of the joint, and the twist range determines the rotation along the bone axis. The rotation matrix R decouples the twist rotation and swing rotation components, and the formula is:

[0146]

[0147] in, and They represent the initial body part vector and the target vector respectively, and φ represents the twisting angle.

[0148] In order to find the closed solution efficiently, the Rodrigues rotation formula is used to calculate the rotation matrix R.

[0149] Swing Rotation R swing The formula is

[0150]

[0151]

[0152] in, express and The normal vector of , I represents the 3×3 identity matrix, Representation vector The antisymmetric matrix of Indicates the swing angle.

[0153] Twist and rotate R twist The formula is:

[0154]

[0155] in, Represents the target vector The antisymmetric matrix of .

[0156] In this example, we will collide with the leaf node B in the subtree leaf The position placed at the collision point is taken as the target posture T, and B is calculated recursively leaf The twisting rotation R of each parent node twist and swing rotation R swing , get the rotation matrix R to update the posture parameter θ b . Solve the example Figure 4 shown.

[0157] When the rotation matrix R of the human body posture is updated, there is a potential problem that penetration may occur inside the human body. Therefore, after the posture is updated, collision detection technology needs to be used to detect the collision point, and inverse dynamics is continued to be applied to adjust the posture to ensure that there is no penetration between the human bodies.

[0158] After the above steps, the interaction of the 3D human model in the scene is realized, generating physically and semantically credible interactions between people and scenes.

[0159] like Figure 1 As shown, in step S5, the result of the interaction between the person and the scene is output.

[0160] In this embodiment, given a seated human body model and a room scene, semantics are generated by classifying the posture of the human body model, as well as scene position planning under the geometric constraints of the human body and posture adjustment under physical constraints. The three-dimensional interaction result of the human body sitting on the sofa in the scene is output, ensuring the rationality and naturalness of the posture while achieving interaction.

[0161] Based on the three-dimensional human-scene interaction framework, the human body model can be placed at the appropriate position in the scene, and physically and semantically credible interactions between people and scenes can be generated.

[0162] Embodiment 2

[0163] The purpose of this embodiment is to provide a human-scene interaction generation system based on geometric and physical constraints, including:

[0164] A human body model and scene model setting module, used to use a three-dimensional human body model to represent a three-dimensional human body of a given posture, and use a three-dimensional scene model to represent a scene;

[0165] The posture-constrained human contact semantic generation module is used to define and generate the semantic labels of the initial human body model, and perform posture recognition on the given human body model through a posture classifier based on a convolutional neural network to optimize the semantic labels;

[0166] A 3D human model placement module is used to generate candidate position anchor points based on the 3D scene model, and determine the best position in the scene by calculating the geometric and semantic distances of the human model vertices with semantic labels at each anchor point position;

[0167] A posture adjustment module, used to adjust the posture of the human model placed at the best position in the scene based on inverse dynamics;

[0168] The human-scene interaction output module is used to output the human-scene interaction results.

[0169] The human-scene interaction generation system based on geometric and physical constraints implements a human-scene interaction generation method based on geometric and physical constraints in the first embodiment.

[0170] Embodiment 3

[0171] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0172] Embodiment 4

[0173] The purpose of this embodiment is to provide a computer-readable storage medium.

[0174] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0175] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0176] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0177] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for generating human-scene interaction based on geometric and physical constraints, characterized in that: include: A 3D human body model is used to represent a 3D human body in a given posture, and a 3D scene model is used to represent a scene; Define and generate semantic labels for the initial human body model, and perform posture recognition on the given human body model through a posture classifier based on a convolutional neural network to optimize the semantic labels; Based on the 3D scene model, candidate position anchor points are generated, and the best position in the scene is determined by calculating the geometric and semantic distances of the human body model vertices with semantic labels at each anchor point position; Based on inverse kinematics, the pose of the human model placed at the best position in the scene is adjusted; Output the results of human-scene interaction.

2. A method for generating human-scene interaction based on geometric and physical constraints as claimed in claim 1, characterized in that: A three-dimensional human body model is used to represent a three-dimensional human body in a given posture, wherein the three-dimensional human body model adopts the SMPL-X model and is defined as: M h =(P h ,V h ,L h ) Among them, P h is the model parameter information represented by the function M(β, θ, ψ, t), V h is the vertex set of the model, L h It is the set of semantic labels of model vertices.

3. A method for generating human-scene interaction based on geometric and physical constraints as claimed in claim 1, characterized in that: Define and generate semantic labels for the initial human body model, specifically: Input the model parameter information into the encoder of the conditional variational autoencoder network to extract key features; Based on the key features, semantic labels are generated by the decoder.

4. A method for generating human-scene interaction based on geometric and physical constraints as claimed in claim 1, characterized in that: The posture classifier based on convolutional neural network is used to perform posture recognition on the given human body model to optimize the semantic label, specifically: Input the model parameter information into the convolutional neural network for posture classification; By constructing a loss function, the posture classification is optimized; The contact range is adjusted according to the posture classification results, and the semantic labels within the contact range are retained.

5. The method for generating human-scene interaction based on geometric and physical constraints as claimed in claim 1, characterized in that: Based on the 3D scene model, candidate position anchor points are generated, specifically: Divide the objects in the 3D scene model into multiple regions, and the scene model vertices in the same region have the same semantic label; Calibrate the diagonal boundary position of the scene to indicate the scope of the scene; In the scene range, obtain candidate anchor points at a given distance; Use the distance field to detect whether the candidate anchor point collides with the scene. If not, the candidate anchor point is inside the nearest scene object and is removed. If there is a collision, the candidate anchor point is outside the scene object closest to it and serves as the candidate position anchor point.

6. A method for generating human-scene interaction based on geometric and physical constraints as claimed in claim 1, characterized in that: The best position in the scene is determined by calculating the geometric and semantic distances of the human model vertices with semantic labels at each anchor position, specifically: Project each vertex in the human body model vertex set along the Y-axis direction of the world coordinate system to obtain the scene model vertex closest to it; The human body model vertices and their corresponding scene model vertices form human body-scene vertex pairs; Calculate the geometric distance between human-scene vertex pairs; Calculate the semantic distance between human-scene vertex pairs; Calculate the total distance between all vertex pairs through geometric distance and semantic distance; Based on the total distance, the optimal position of the human model in the scene is calculated.

7. A method for generating human-scene interaction based on geometric and physical constraints as claimed in claim 1, characterized in that: Based on inverse dynamics, the posture of the human model placed at the best position in the scene is adjusted as follows: Based on the human body model at the best position in the scene, the human body parts and interaction points to be adjusted are determined by constructing a heterogeneous kinematic tree and performing collision detection; Based on the determined interaction points, inverse dynamics is used to calculate the rotation information of the human body's joints and update and adjust them.

8. A human-scene interaction generation system based on geometric and physical constraints, characterized in that: include: A human body model and scene model setting module, used to use a three-dimensional human body model to represent a three-dimensional human body of a given posture, and use a three-dimensional scene model to represent a scene; The posture-constrained human contact semantic generation module is used to define and generate the semantic labels of the initial human body model, and perform posture recognition on the given human body model through a posture classifier based on a convolutional neural network to optimize the semantic labels; A 3D human model placement module is used to generate candidate position anchor points based on the 3D scene model, and determine the best position in the scene by calculating the geometric and semantic distances of the human model vertices with semantic labels at each anchor point position; A posture adjustment module, used to adjust the posture of the human model placed at the best position in the scene based on inverse dynamics; The human-scene interaction output module is used to output the human-scene interaction results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are performed.

Images