Human body three-dimensional constraint body shape deformation method and system based on semantic parameter mapping

Abstract

The invention discloses a human body three-dimensional constraint body shape deformation method and system based on semantic parameter mapping, and the method comprises the steps: training a YOLO-Pose model to detect human body joint points, and reconstructing a three-dimensional human body grid through SMPL-X and Hybrid IK-X; constructing a model database of semantic parameters and SMPL-X shape parameters; three-dimensional-two-dimensional mapping is established, and image triangular meshes are divided; constructing a sparse linear system containing shape deformation constraint, neighborhood regularization and deviation penalty, and performing pre-decomposition; by adjusting semantic parameters to retrieve target shape parameters, updating and solving system boundary conditions, a deformed image is generated, the method is combined with human body three-dimensional structure constraints, parameterization accurate control over the body shape is achieved, the deformation result is natural and real, original image features are reserved, calculation is efficient, and the method is suitable for scenes such as virtual fitting and digital human modeling.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D graphics processing technology, specifically involving a method and system for human body 3D constraint deformation based on semantic parameter mapping, which is mainly applied to scenarios such as customized human body image editing, digital human modeling, and virtual try-on. Background Technology

[0002] With the development of virtual try-on and digital human technologies, the demand for controllable human body shape editing is increasing. Existing technologies suffer from the following drawbacks: traditional image editing software relies on manual local stretching, which is cumbersome and prone to proportional distortion; generative models (such as GANs and diffusion models) produce highly random editing results, making it difficult to precisely control local body shapes and easily resulting in the loss of the original person's identity and texture details; traditional two-dimensional mesh deformation methods lack three-dimensional structural constraints, easily leading to unreasonable human body structures and unnatural deformations. Therefore, there is an urgent need for a human body shape editing technology that combines three-dimensional human structural information, is easy to operate, and computationally efficient, enabling parametric, precise, and controllable human body shape deformation while ensuring the consistency of two-dimensional image quality and person's identity. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for three-dimensional constrained human body shape deformation based on semantic parameter mapping, which solves the problems of difficult precise control, insufficient result stability, and inability to balance the rationality of three-dimensional structure and the quality of two-dimensional image in the prior art. It achieves convenient operation, high computational efficiency, and natural and realistic results in parametric editing of human body shape.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solution: This invention relates to a method for deforming a three-dimensional human body shape under semantic parameter mapping, which includes the following steps: Step S1: Prepare human joint annotation data and train human joint detection model; acquire human image to be processed, and use the human joint detection model to process the human image to obtain the position information of human joints. Step S2: Based on the position information of the human body joints, a parametric three-dimensional human body reconstruction method is used to reconstruct the human body image in three dimensions to obtain a three-dimensional human body mesh model aligned with the human body image. Step S3: Based on the three-dimensional human body reconstruction method, construct a model body shape database, store the shape parameters of the three-dimensional mesh reconstruction of the model, and make the semantic parameters correspond to the semantically unrelated three-dimensional reconstruction shape parameters; Step S4: Perform three-dimensional human reconstruction on the human image to be edited, establish a correspondence between the three-dimensional human body space and the two-dimensional image space, and obtain the corresponding human body joint points and body contour points in the three-dimensional human body and the two-dimensional image. Step S5: Using the human body joints and body contour points in the planar space as a set of constraint points, perform triangular meshing on the human body image to generate a triangular mesh for image deformation basis. Step S6: Using the vertex positions of the deformed triangular mesh as unknowns, construct a least-squares linear sparse system that includes a volume deformation energy term, a neighborhood regularization term, and a deviation penalty term, and pre-decompose the sparse system. Step S7: Adjust semantic parameters, generate a target body shape 3D mesh with the same pose as the input human body image by retrieving shape parameters from the model body shape library, calculate the boundary conditions of the sparse system by the difference between the target body shape 3D mesh and the original mesh and solve it in real time, and output the human body shape deformation result.

[0005] Preferably, in step S1, the annotation content of the human joint annotation data includes the two-dimensional joint coordinates and semantic labels of the human body's external bounding box, head, neck, shoulder, elbow, wrist, hip, knee, and ankle; the human joint detection model is a YOLO-Pose network model, which is trained based on the YOLO object detection framework, and the training loss function is the mean squared error loss of the joint coordinates. ;in For training loss values, Let i be the coordinates of the i-th joint predicted by the model. Here are the coordinates of the i-th joint, and N is the number of joints.

[0006] Preferably, in step S2, the parametric 3D human body reconstruction method is based on the SMPL-X model and the HybrIK-X regression algorithm, and the vertex generation formula for the 3D human body mesh model is: ;in Here, S represents the original 3D human body mesh model, and S is the vertex generation function of the SMPL-X model. For shape parameters, These are attitude parameters. For facial expression parameters; the HybrIK-X regression algorithm takes the joint positions from step S1 as input and regresses to obtain... and .

[0007] Preferably, in step S3, the semantic parameters include height H, weight W, bust B, waist Wa, and hip circumference Hip; the model body shape database is constructed by collecting images of models in standard poses, and performing step S2 on each image to obtain shape parameters. Connect H, W, B, Wa, and Hip with Associative storage is used to form a mapping relationship between semantic parameters and shape parameters.

[0008] Preferably, in step S4, the correspondence between the three-dimensional human body space and the two-dimensional image space is realized through the camera projection matrix, as shown in the formula: ;in The mapped two-dimensional coordinates The projection matrix is ​​composed of the camera's intrinsic and extrinsic parameters. The coordinates of the three-dimensional human body joints or contour points are obtained through three-dimensional model partitioning, mask projection, and contour filtering. Specifically, the three-dimensional mesh is divided into sub-regions such as left arm, right arm, left leg, right leg, and torso. After projecting to generate a mask image, the outermost contour points of the human body are filtered out.

[0009] Preferably, in step S6, the formula for the body shape deformation energy term is: The formula for the neighborhood regularization term is: The formula for the deviation penalty term is: ;in For body shape deformation energy term, This is the set of vertex positions of the deformed triangular mesh. The original two-dimensional mesh edge vectors, These are the edge vectors of the original 3D mesh. The edge vectors of the deformed 2D mesh. These are the edge vectors of the deformed 3D mesh. For neighborhood regularization, Let i be the set of adjacent triangles of the i-th triangle. Let be the affine transformation matrix of the i-th triangle. Let be the affine transformation matrix of the j-th triangle. Let I be the deviation penalty term, and let I be the identity matrix.

[0010] Preferably, in step S6, the optimization objective formula for the least squares linear sparse system is: ;in To optimize the overall energy value, The weight of the energy term along the skeletal direction. This is an energy term along the skeletal direction. The weight of the energy term in the vertical bone direction. This is the energy term in the vertical bone direction. For the contour energy term weights, For the contour energy term, The weight of the neighborhood regularization term. For neighborhood regularization, The weight of the deviation penalty term, This is a bias penalty term; the sparse system is pre-decomposed into a system of linear equations. Perform QR decomposition on matrix A and store the decomposition results, where A is the coefficient matrix and b is the right-hand vector.

[0011] Preferably, in step S7, the formula for generating the 3D mesh of the target body is: ;in Let S be the 3D mesh of the target body, and S be the vertex generation function of the SMPL-X model. For the target shape parameters, The original attitude parameters obtained in step S2, For facial expression parameters; the boundary conditions of the sparse system are updated by calculating the ratio of the length changes of corresponding edges between the original human body 3D mesh model and the target body 3D mesh. And substitute it into the body deformation energy term to update the vector b on the right side of the equation system.

[0012] This invention relates to a three-dimensional constrained human body deformation system based on semantic parameter mapping, comprising a human body deformation device, wherein the human body deformation device includes: The joint detection module is used to execute step S1 and output the two-dimensional position information of human joints. The 3D human body reconstruction module is used to execute step S2 and output a 3D human body mesh model. ; The body shape database creation module is used to execute step S3 and store the correspondence between semantic parameters and SMPL-X shape parameters; The image mesh generation module is used to execute steps S4-S5 and output a two-dimensional triangular mesh. ; The sparse system construction and solution module is used to execute steps S6-S7 and output a human body shape deformation image; The output of the joint detection module is connected to the input of the 3D human body reconstruction module; the output of the 3D human body reconstruction module is connected to the input of the body shape database creation module and the image mesh generation module respectively; the output of the image mesh generation module is connected to the input of the sparse system construction and solution module; and the output of the body shape database creation module is connected to the input of the sparse system construction and solution module.

[0013] Preferably, it also includes an electronic device, which includes an electrically connected memory, a processor, and a communication unit; the memory is capable of storing program code for the human body deformation device; the processor executes the program code to realize the functions of each module; and the communication unit is used to realize data communication between the electronic device and external devices.

[0014] Beneficial effects: This invention can realize parametric editing of human body shape at the two-dimensional image level by making full use of the three-dimensional structural information of the human body. It takes into account the realism, continuity and controllability of body shape deformation, while avoiding the randomness problem brought about by generative methods. It has the advantages of high computational efficiency, good stability and strong engineering implementation. It is suitable for a variety of application scenarios such as virtual fitting, digital human modeling and human body image content editing. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the invention system structure.

[0016] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0017] Figure 3 This is a schematic diagram of step S26 of the present invention.

[0018] Figure 4 This is a flowchart illustrating the sub-step of step S23 of the present invention.

[0019] Figure 5 This is a flowchart illustrating the sub-step of step S24 of the present invention.

[0020] Figure 6 This is a flowchart illustrating the sub-step of step S25 of the present invention.

[0021] Figure 7 This is a schematic diagram of the human body shape deformation device of the present invention.

[0022] Figure 8 This is a schematic diagram of human body shape deformation provided in Embodiment 1 of the present invention.

[0023] Icons: 100 - Electronic device; 110 - Human body shape deformation device; 111 - Memory; 112 - Processor; 113 - Communication unit; 1101 - Joint point detection module; 1102 - 3D human body reconstruction module; 1103 - Body shape database creation module; 1104 - Image mesh generation module; 1105 - Sparse system construction and solution module. Detailed Implementation

[0024] The following will refer to the accompanying drawings in the embodiments of the present invention. Figures 1 to 8 The technical solutions in the embodiments of the present invention will be clearly and completely described.

[0025] Parametric human model SMPL-X (Skinned Multi-Person Linear-eXpressive); Regression algorithm / model HybrIK-X (Hybrid Inverse Kinematics-eXpressive).

[0026] Principle of this invention: The technical solution of this invention includes steps such as human joint detection, three-dimensional human reconstruction, model body shape database construction, three-dimensional to two-dimensional mapping, image meshing processing, sparse system modeling and solving, and semantic parameterized body shape editing. The steps cooperate with each other to form a complete human body shape deformation processing flow.

[0027] First, before body shape editing, the existing human image data is annotated to obtain the human bounding box and the joint positions and corresponding semantic labels of key parts such as the torso, arms, and legs. Based on the above annotated data, a human joint detection model is trained using a YOLO-Pose convolutional neural network. This model can automatically and stably detect the positions of human joints in input human images, providing reliable structural prior information for subsequent human pose restoration and body shape analysis.

[0028] Based on the acquisition of human joint information, this invention employs a 3D human reconstruction method based on a parametric human model to restore the 3D human structure from input human images. Specifically, the 3D human reconstruction method is based on the SMPL-X parametric human model and the HybrIK-X regression algorithm / model. By establishing the correspondence between human joints and the parametric human mesh, the HybrIK-X regression algorithm / model is used to quickly regress the shape parameters, pose parameters, and expression parameters of the SMPL-X model, thereby obtaining a 3D human mesh model that is consistent with the input human image in terms of body shape and pose. The 3D mesh model has a unified topological structure, which can provide a stable and continuous geometric representation for subsequent body shape editing.

[0029] Based on the aforementioned 3D human body reconstruction method, this invention constructs a model body shape database. The database pre-stores images of human models with rich body shapes and accurate semantic parameters (such as height, weight, bust, waist, and hip circumference). The images in the model body shape database are reconstructed into 3D meshes, and their body shape description parameters are saved. These parameters are then used to quickly maintain the pose of the target body shape 3D mesh by combining it with the pose parameters of the image to be edited. By uniformly storing the body shape information of different models in a parametric form, standardized and reusable target body shape references can be provided to users, offering data support for subsequent rapid editing based on semantic body shape parameters.

[0030] When processing a user's editable human body image, the same 3D human body reconstruction process as described above is first employed to restore the corresponding 3D human body mesh model. Subsequently, the 3D human body mesh model is partitioned according to the human body structure, and boundary regions that characterize the human body's shape features within each partition are sampled to obtain human body contour points; simultaneously, the corresponding human body joint points in the 3D human body mesh model are obtained. By mapping the human body joint points and the sampled human body contour points between the 3D human body space and the 2D image space, a mapping relationship between the 3D human body mesh space and the 2D image space is established, enabling changes in the body shape of the 3D human body mesh to be reflected in the 2D image space.

[0031] Based on establishing a 3D-to-2D correspondence, this invention uses the 2D human body joints and contour points projected onto the image space as a constraint point set to perform triangular mesh division on the input human body image. By connecting adjacent human body contour points and combining human body joint information, the Triangle algorithm library is used to further subdivide the human body image, generating a 2D triangular mesh model composed of multiple triangular units. This 2D triangular mesh model can effectively reduce the complexity of subsequent image deformation calculations while ensuring the continuity of the image structure.

[0032] After image meshing, this invention constructs a least-squares linear sparse system using the vertex positions of the deformed two-dimensional triangular mesh as unknowns. The sparse system includes a body shape deformation energy term, a neighborhood regularization term, and a deviation penalty term. The body shape deformation energy term constrains the changes in bone length, limb or torso width perpendicular to the bones, and body contour length of key parts of the human body before and after deformation, ensuring that the body shape deformation result conforms to the actual structural proportions of the human body. The neighborhood regularization term maintains the continuity between adjacent triangular mesh units, avoiding discontinuities or folding phenomena. The deviation penalty term limits the degree of deviation between the deformation result and the original body shape, thereby improving the stability and rationality of the overall deformation result.

[0033] To improve the solution efficiency during body shape editing, this invention performs matrix pre-decomposition on the least-squares linear sparse system and temporarily stores the decomposition results. In subsequent settings of different semantic body shape parameters or multiple interactive editing processes, only rapid solution is needed based on new boundary conditions, thereby significantly reducing computational overhead and improving system response speed.

[0034] Within the aforementioned sparse system modeling and solution framework, users can modify semantic body shape parameters such as height, weight, chest circumference, waist circumference, and hip circumference to obtain the corresponding 3D human body mesh modeling parameters from a model body shape database through matching or mapping. Combined with the pose parameters of the human image to be edited, a 3D human body mesh model with the target body shape is generated while maintaining the original human image's pose. Subsequently, based on the differences between the original and target body shape 3D mesh models in corresponding deformation units, these differences are converted into boundary conditions for 2D image mesh deformation, thereby achieving rapid and controllable deformation of the overall human body shape.

[0035] Through the above technical solution, the present invention can realize parametric editing of human body shape at the two-dimensional image level by making full use of the three-dimensional structural information of the human body. It takes into account the realism, continuity and controllability of body shape deformation, while avoiding the randomness problem brought about by generative methods. It has the advantages of high computational efficiency, good stability and strong engineering implementation, and is suitable for a variety of application scenarios such as virtual fitting, digital human modeling and human body image content editing.

[0036] Content of this invention: The human body 3D constrained deformation method based on semantic parameter mapping includes the following: Step S1: Prepare human joint annotation data and train a human joint detection model; acquire the human image to be processed, and use the human joint detection model to process the human image to obtain the position information of the human joints; wherein, before the human body shape is deformed, the data annotation of the human position and human joints is completed, and the model is trained based on a convolutional neural network, specifically including: annotating the existing human image data, the annotation content includes the human body position box and the joints of key parts such as shoulders, hips, elbows, and knees and their corresponding joint labels; using the annotated human image data, the human joint detection model is trained using a YOLO-Pose convolutional neural network to obtain the pose information of the human image to be edited.

[0037] Step S2: Based on the position information of the human joints, a parametric 3D human reconstruction method is used to reconstruct the human body image in 3D to obtain a 3D human body mesh model aligned with the human body image. The 3D human body reconstruction method is based on the SMPL-X parametric human body model and the HybrIK-X regression algorithm / model, and specifically includes: obtaining the joint positions of the human body image through a human joint detection model; and regressing the pose parameters, shape parameters, and expression parameters of SMPL-X by the HybrIK-X model based on the human body image joints and the corresponding SMPL-X 3D mesh human body joints to obtain a 3D human body mesh model aligned with the human body image posture and body shape.

[0038] Step S3: Based on the aforementioned 3D human body reconstruction method, a model body shape database is constructed to store the shape parameters of the 3D mesh reconstruction of the user's own models, establishing a correspondence between semantic parameters such as height, weight, chest circumference, waist circumference, and hip circumference and the shape parameters of the non-semantic 3D reconstruction. The model body shape database stores a large number of model images with the same pose and diverse body shapes, and includes corresponding body shape information such as height, weight, chest circumference, waist circumference, and hip circumference. The aforementioned 3D human body reconstruction method is used to perform 3D reconstruction on the user's own model images, establishing a correspondence between semantic body shape parameters and the shape parameters of the non-semantic SMPL-X model. By modifying the semantic body shape parameters, including height, weight, chest circumference, waist circumference, and hip circumference, the user maps the body shape parameters to the target model in the model body shape database, obtains the shape parameters of the corresponding SMPL-X model, and combines them with the pose parameters of the human image to be edited to generate a 3D human body mesh model with the target body shape while maintaining the original human image pose. Based on the change range of the original body shape 3D mesh model and the target body shape 3D mesh model in the corresponding deformation units, boundary conditions for 2D image mesh deformation are generated to achieve global body shape deformation.

[0039] Step S4: Perform 3D human reconstruction on the human image to be edited, establish a correspondence between the 3D human body space and the 2D image space, and obtain corresponding human joint points and body contour points in the 3D human body and the 2D image; establish a mapping relationship between the 3D human body mesh space and the 2D image space so that the 2D image responds to changes in the 3D mesh, specifically including: on the 3D human body mesh model reconstructed from the human image to be edited, divide the 3D human body mesh model into partitions according to the human body structure; in each partition, sample the boundary areas that can characterize the human body shape features to obtain human contour points; map the human joint points and the sampled body contour points between the 3D human body space and the 2D image space through projection and inverse projection.

[0040] Step S5: Using the human body joints and body contour points in the planar space as a set of constraint points, the human body image is divided into triangular meshes to generate a triangular mesh for image deformation basis. Specifically, to reduce the computational load of image processing, the human body image is divided into a set of triangular meshes, which includes: using the two-dimensional joints of the human body projected onto the image space and the two-dimensional contour points corresponding to the three-dimensional points of the human body contour as a set of mesh points to be subdivided; connecting adjacent two-dimensional points of the human body contour as a set of mesh edges; and using the Triangle algorithm library to subdivide the human body image into triangular meshes to generate a set of two-dimensional triangular meshes for image deformation.

[0041] Step S6: Using the vertex positions of the deformed triangular mesh as unknowns, construct a least-squares linear sparse system containing a body shape deformation energy term, a neighborhood regularization term, and a deviation penalty term, and pre-decompose the sparse system; specifically: using the vertices of the deformed image triangular mesh as unknowns, construct a least-squares energy function and perform matrix decomposition, temporarily storing the decomposition results for subsequent reuse. The energy function includes: a body shape deformation energy term, used to constrain the bone length between adjacent joints, the limb or torso width perpendicular to the bone direction, and the length of the human body contour; a neighborhood regularization term, used to maintain the continuity between adjacent mesh cells; and a deviation penalty term, used to limit the deviation between the deformation result and the original body shape.

[0042] Step S7: Adjust semantic parameters such as height, weight, chest circumference, waist circumference, and hip circumference. By retrieving shape parameters from the model body shape database, generate a target body shape 3D mesh with the same pose as the input human body image. Calculate the boundary conditions of the sparse system based on the difference between the target body shape 3D mesh and the original mesh, solve the problem in real time, and output the human body shape deformation result.

[0043] This invention relates to a three-dimensional constrained human body deformation system based on semantic parameter mapping, comprising: The human joint detection module is used to obtain the position information of human joints; The 3D human body reconstruction module is used to reconstruct the human body from images based on human body joint points. The body shape database creation module is used to store the shape parameters of predefined SMPL-X models; The image mesh generation module is used to divide human body images into triangular meshes; The sparse system construction and solution module is used to construct and solve least-squares linear sparse systems and output the human body shape deformation results.

[0044] Example 1:

[0045] like Figure 1 The diagram shown is a structural schematic of the system of the present invention, which includes an electronic device 100, a human body shape deformation device 110, a memory 111, a processor 112, and a communication unit 113.

[0046] The components of the memory 111, processor 112, and communication unit 113 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0047] The memory 111 may be, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 112 executes these programs upon receiving execution instructions. The communication unit 113 establishes communication connections between the electronic device 100 and other devices (such as user terminals) via a network, and receives and sends data through the network. For example, in this embodiment, the electronic device 100 communicates with external devices via the communication unit 113.

[0048] like Figure 7 The diagram shows the structure of the human body deformation device of the present invention, wherein the joint point detection module, the input end of the three-dimensional human body reconstruction module, the body shape database creation module, the image mesh generation module, and the sparse system construction and solution module are connected in communication.

[0049] like Figure 2 The diagram shown is a schematic representation of the method flow of this invention. This application provides a method for three-dimensional constrained human body deformation based on semantic parameter mapping, the method comprising: Step 21: Obtain the human image to be edited.

[0050] Step 22: Perform human joint detection and SMPL-X model 3D human reconstruction on the input image to obtain human joint and 3D structural information.

[0051] Step 23: Establish the mapping relationship between the three-dimensional human body mesh and the planar image space, and divide the planar image space into triangular meshes.

[0052] Step 24, using the new vertex positions of the deformed triangular mesh. For the unknowns, construct an optimization energy function that includes three-dimensional structural constraints and two-dimensional visual regularization. If the deformable element is designed as a linear operation of the vertices of a triangular mesh, then... It can be represented as a linear least squares sparse system, in which the sparse matrix can be pre-decomposed in one step.

[0053] Step 25: Adjust body shape semantic parameters such as height, weight, chest circumference, waist circumference, and hip circumference to quickly find the 3D mesh of the target body shape model. The variation range between the target body shape mesh and the original mesh is converted into boundary conditions for a linear least-squares sparse system, which are then quickly solved. The final deformation result is obtained through texture mapping technology. If the body shape deformation service does not meet expectations, the body shape semantic parameters can be repeatedly adjusted, and the deformation results can be fed back in real time.

[0054] like Figure 3 The diagram shown illustrates step S26 of this invention. Before body shape editing, a human joint detection model is required to provide reliable input for subsequent 3D human reconstruction. Simultaneously, parametric human reconstruction technology is applied to establish a proprietary model body shape library, facilitating the rapid provision of customized body shape editing targets. This specifically includes the following steps: Step 26: Construct a human joint point training dataset. Collect human images containing multiple poses, body types, and scenes, and manually annotate them. The annotations should include at least: The bounding box of the overall position of the human body.

[0055] Two-dimensional joint coordinates and semantic labels of key parts of the human body. The joints should cover the head, neck, shoulders, elbows, wrists, hips, knees, ankles, etc., to fully describe the human body's posture structure.

[0056] Next, based on the labeled data, the YOLO-Pose network structure is used for training. YOLO-Pose is an end-to-end pose estimation model that, based on the YOLO object detection framework, can complete multi-person detection and corresponding keypoint regression in a single forward propagation.

[0057] Step 27: For any input human body image to be edited First, the trained YOLO-Pose model is used for detection to obtain a set of two-dimensional joint coordinates. Subsequently, 3D human reconstruction was performed based on the SMPL-X parametric human body model and the HybrIK-X regression algorithm. SMPL-X is a high-fidelity human body mesh model containing 10K+ vertices, whose shape is determined by shape parameters. Attitude parameters and expression parameters Control, where shape parameters These belong to latent space feature vectors and lack semantic meaning. The HybrIK-X algorithm uses the aforementioned two-dimensional key points... Using the input as input, the corresponding SMPL-X model parameters are accurately regressed through hybrid inverse kinematics principles. (Emoji parameters) (Set to neutral). Inputting the parameters obtained from the regression into the SMPL-X model will generate a 3D human body mesh model that is aligned with the input image I in terms of body shape and pose. The model has a unified topological structure, and its vertex set is denoted as . This provides a stable three-dimensional geometric reference for subsequent processing.

[0058] Step 28: To support parametric editing, a model body shape database needs to be pre-built. This database consists of images of models with different body shapes in standard poses. The aforementioned 3D reconstruction process is performed on each image to obtain the corresponding SMPL-X model shape parameters. . Use predefined semantic body parameters and These parameters are linked and stored together in the database. The semantic body shape parameters include: height, weight, bust, waist, hips, bra size, waist shape, belly fat, hip shape, hip width, upper and lower body width ratio, upper and lower body length ratio, shoulder width, hip width, thigh circumference, and calf circumference. This database constitutes a system based on semantic body shape parameters. to low-dimensional shape hidden space The mapping supports target parameters input by the user. Quickly retrieve the target shape parameters .

[0059] like Figure 4 As shown Figure 2 A flowchart illustrating the sub-steps of step S23. In this embodiment, step S23 may include sub-steps S231, S232, and S233.

[0060] Step 231, Mapping of Human Joints. Map the 3D human body mesh model... The 3D vertices corresponding to the key joints (such as shoulder, elbow, wrist, hip, knee, ankle, etc.) predefined in the SMPL-X model. The camera projection matrix obtained during the reconstruction process using the HybrIK-X algorithm The data is directly projected onto the image plane to obtain a precise set of two-dimensional joints. These point pairs form the basic human skeleton, providing a calculation basis for subsequent deformable elements.

[0061] Step 232, Sampling and mapping of human body external contour points. To better capture the human body's external contour, this invention employs a precise contour point sampling method based on part segmentation.

[0062] 3D model partitioning. This involves partitioning the complete SMPL-X 3D model. The area is pre-divided into five sub-grid regions, corresponding to: left arm, right arm, left leg, right leg, and torso (including head).

[0063] Project and draw the mask image. This will create a complete 3D mesh. And the five subgrids mentioned above, each using the same camera projection matrix. The image is drawn into image space to generate a corresponding binary mask image. This results in a complete human body mask. and masks for five parts Extract the set of outer contour points from the mask image. Since individual part masks may be discontinuous in image space (e.g., where an arm connects to the torso), directly extracted contour points may contain internal boundaries. To filter out points that truly lie on the outermost contour of the entire human body, utilize a global mask. Perform verification: Only keep those in Points on the outer contour of the mask are denoted as the filtered 2D contour point set. This step ensures that the sampled contour points correspond to the actual visible outer contour of the human body from the current viewpoint.

[0064] 3D corresponding point search, obtaining 3D-2D point pairs. Then, for each filtered 2D outer contour point... By using the inverse projection relationship to map back to three-dimensional space, a three-dimensional point located near the surface of the three-dimensional mesh is obtained. Then, in In the vertex set, find the vertex with the kd-tree. Euclidean distance to the nearest grid vertex Thus, a pair of high-confidence 3D-2D mapping points is obtained. .

[0065] Step 233, triangular mesh generation. All the two-dimensional joints obtained in step 231... and all two-dimensional outer contour points obtained in step 232 The points are merged and used as a set of constraint points. The input image region is divided into a two-dimensional triangular mesh using the Delaunay triangulation algorithm (provided by the Triangle library). Its vertex set is denoted as This set contains all and This mesh will serve as the geometric basis for subsequent image deformation.

[0066] like Figure 5 As shown Figure 2 A flowchart illustrating the sub-steps of step S24. In this embodiment, step S24 may include sub-steps S241, S242, S243, S244, and S245.

[0067] Step 241, construct the body deformation energy term. This is to represent the three-dimensional body deformation... (from To the target model To accurately transmit the data to a two-dimensional image, a uniform length variation constraint is defined. Let... Let e ​​be a pair of 2D-3D edges, where e is an edge vector in the 2D mesh and w is the 3D model. The corresponding edge vectors in the diagram. After transformation, they become... and Constrain its proportional variation to be consistent across two-dimensional and three-dimensional spatial scales: Based on this, three specific constraints are constructed: 1) Deformation energy term along the bone direction. Connect adjacent joints as the current part of the skeleton, sample 3D point pairs along a direction roughly parallel to the skeleton, and construct an edge pair set. This energy item controls changes in the length of limbs or trunk (such as altering height or arm length). 2) Deformation energy item perpendicular to the skeletal direction. Within each segment of the human body, samples pairs of three-dimensional points perpendicular to the skeletal axis to construct a set of edge pairs. This energy item controls changes in the width / circumference of limbs or torso (such as changes in arm circumference or waist circumference).

[0068] Human body contour energy item Sample adjacent points along the outer contour of the 3D model to construct a set of edge pairs. This energy level controls the overall deformation of the human body's external contours, ensuring a natural and smooth contour after deformation.

[0069] Step 242, construct neighborhood regularization terms To force adjacent triangular elements to deform as uniformly as possible, maintain local mesh continuity, and prevent visual defects such as folding and tearing, the neighborhood regularization term is constructed as follows: ;in The affine transformation matrix before and after the transformation of the i-th triangle can be represented as: ;in These are the centroid coordinates of the i-th triangle, which can be calculated by inversely taking the homogeneous coordinate matrix of the original triangle vertices (column pivoted). It is the coordinate matrix of the vertices after the i-th triangle is deformed. Therefore, the neighborhood regularization term is also the energy term for the vertices of the deformed triangular mesh.

[0070] Step 243, construct the deviation penalty term To limit the variation in important areas such as faces and hairstyles, and to avoid disrupting visual consistency, a deviation penalty term is constructed as follows: ;in It is an identity matrix.

[0071] Step 244: Construct a weighted linear least squares sparse system. Combining the above objective functions, the constructed optimization objective is: The weighting parameter ω is used to balance the strength of the various constraints. In a preferred embodiment, the following empirical configuration is employed to achieve a balance between the realism of body shape changes and the visual quality of the image: ; in To optimize the overall energy value, The weight of the energy term along the skeletal direction. This is an energy term along the skeletal direction. The weight of the energy term in the vertical bone direction. This is the energy term in the vertical bone direction. For the contour energy term weights, For the contour energy term, The weight of the neighborhood regularization term. For neighborhood regularization, The weight of the deviation penalty term, This is a penalty item for deviation.

[0072] Step 245, the above energy function It is about The quadratic form can be transformed into a large sparse linear system of equations. The sparse matrix A depends only on the original grid. and 3D model The geometric relationships are independent of the deformation target. Therefore, before the interaction begins, matrix A can be decomposed once using QR decomposition, and the decomposition results can be temporarily stored.

[0073] like Figure 6 As shown Figure 2 A flowchart illustrating the sub-steps of step S25. In this embodiment, step S25 may include sub-steps S251, S252, and S253.

[0074] Step 251, semantic body shape parameter adjustment. The user inputs the target body shape parameters. The system searches its own database of model models for the nearest neighbor with the best body type. The three-dimensional reconstructed shape parameters are obtained. Combined with the original posture Generate target 3D model .calculate and The length changes of the corresponding sides are used to update the vector b on the right-hand side of the sparse linear equation system.

[0075] Step 252: Texture mapping before and after mesh deformation to obtain the result image. Obtain the vertices of the deformed mesh. Then, the affine transformation from the original triangle to the new position is calculated. Using texture mapping technology, the new mesh position is inversely transformed back to the original mesh, and the pixel value of the new position is calculated by bilinear interpolation to generate the final high-quality deformed image, Ideformed.

[0076] like Figure 8 The diagram shows the body shape editing flowchart of this invention. In this embodiment, the user inputs a human image to be edited, performs joint point detection and 3D human reconstruction on the image, and obtains the original body shape parameters and shape parameters. Based on the semantic body shape parameters such as height, weight, and bust, waist, and hip measurements input by the user, the shape parameters corresponding to the target body shape are retrieved from the model body shape database, and a 3D human mesh model of the target body shape is generated while maintaining the original human posture. Based on the difference between the original body shape 3D human mesh model and the target body shape 3D human mesh model, a sparse linear system is constructed and solved, and the corresponding human body shape deformation result image is output. If the deformation effect does not meet expectations, the semantic parameters are changed, and the body shape is edited in real time.

[0077] Example 2: The following is an explanation based on specific parameter data.

[0078] Human Joint Detection and 3D Reconstruction: 10,000 multi-pose human images were acquired, and 17 joints were labeled; a YOLO-Pose model was trained (input resolution 512×512, 50 iterations); for the image to be edited, a 2D joint set was output. ;Will Inputting HybrIK-X, the shape parameters of SMPL-X are obtained through regression. Attitude parameters Generate a 3D human body mesh model containing 10,475 vertices. Model body shape database construction: 5000 T-pose model images were collected, labeled with height (150-190cm, stride length 1cm), weight (40-100kg, stride length 0.5kg), and bust, waist, and hip measurements (60-120cm, stride length 1cm); 3D reconstruction was performed on each model, and semantic parameters were combined with shape parameters. Store in the database.

[0079] 3D-2D mapping and mesh generation: The region was divided into 5 sub-regions, and a mask image was generated by projection. 200 two-dimensional contour points were obtained by filtering. The two-dimensional joint points and contour points were merged, and approximately 2000 two-dimensional triangular meshes were created using the Triangle library. .

[0080] Sparse system construction: setting weight parameters Constructing the energy function It is then converted into a 1034×1034 sparse matrix A, QR decomposition is performed and the matrix is ​​stored.

[0081] Semantic parameter editing and deformation solving: The user inputs target parameters (height 175cm, weight 65kg, chest circumference 90cm, waist circumference 75cm, hip circumference 95cm); the target shape parameters are retrieved. Generate a 3D human body mesh model ; Calculate the percentage change in side length and update b; Solve get A 512×512 deformed image is generated by bilinear interpolation.

[0082] In summary, this invention obtains a stable and consistent 3D human body structure representation by performing human body key point detection and parametric 3D human body reconstruction on input human body images. Based on this, a model body shape database containing rich semantic body shape parameters is constructed to achieve a mapping between semantic body shape parameters and parametric 3D body shape modeling parameters. Furthermore, a correspondence between the 3D human body mesh space and the 2D image space is established, and the human body image is triangularly meshed based on human body key joints and outer contour constraints. By constructing and pre-decomposing a least-squares linear sparse system containing body shape deformation constraints, neighborhood continuity constraints, and deviation penalty constraints, 3D body shape changes are stably and continuously transmitted to the 2D image space, thereby achieving parametric, controllable, and real-time deformation of the human body shape without regenerating image content.

[0083] This invention fully integrates three-dimensional human body structural information with two-dimensional image deformation processing, and can achieve continuity, realism and stability of body shape editing results while maintaining the consistency of the original human body posture, identity features and texture details. Compared with human body shape editing methods based on generative models, this invention has the advantages of strong result controllability, high computational efficiency, fast response speed and good engineering implementation, and is suitable for application scenarios such as virtual try-on, digital human modeling and customized human body image editing.

[0084] Finally, it should be noted that the present invention is not limited to the above embodiments, and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for three-dimensional constrained human body deformation based on semantic parameter mapping, characterized in that... The method includes the following steps: Step S1: Prepare human joint annotation data and train human joint detection model; acquire human image to be processed, and use the human joint detection model to process the human image to obtain the position information of human joints. Step S2: Based on the position information of the human body joints, a parametric three-dimensional human body reconstruction method is used to reconstruct the human body image in three dimensions to obtain a three-dimensional human body mesh model aligned with the human body image. Step S3: Based on the three-dimensional human body reconstruction method, construct a model body shape database, store the shape parameters of the three-dimensional mesh reconstruction of the model, and make the semantic parameters correspond to the semantically unrelated three-dimensional reconstruction shape parameters; Step S4: Perform three-dimensional human reconstruction on the human image to be edited, establish a correspondence between the three-dimensional human body space and the two-dimensional image space, and obtain the corresponding human body joint points and body contour points in the three-dimensional human body and the two-dimensional image. Step S5: Using the human body joints and body contour points in the planar space as a set of constraint points, perform triangular meshing on the human body image to generate a triangular mesh for image deformation basis. Step S6: Using the vertex positions of the deformed triangular mesh as unknowns, construct a least-squares linear sparse system that includes a volume deformation energy term, a neighborhood regularization term, and a deviation penalty term, and pre-decompose the sparse system. Step S7: Adjust semantic parameters, generate a target body shape 3D mesh with the same pose as the input human body image by retrieving shape parameters from the model body shape library, calculate the boundary conditions of the sparse system by the difference between the target body shape 3D mesh and the original mesh and solve it in real time, and output the human body shape deformation result.

2. The method for human body three-dimensional constrained deformation based on semantic parameter mapping according to claim 1, characterized in that, In step S1, the annotation content of the human joint annotation data includes the two-dimensional joint coordinates and semantic labels of the human body's external bounding box, head, neck, shoulder, elbow, wrist, hip, knee, and ankle. The human joint detection model is a YOLO-Pose network model, which is trained based on the YOLO object detection framework, and the training loss function is the mean squared error loss of the joint coordinates. ;in For training loss values, Let i be the coordinates of the i-th joint predicted by the model. Here are the coordinates of the i-th joint, and N is the number of joints.

3. The method for human body three-dimensional constrained deformation based on semantic parameter mapping according to claim 1, characterized in that, In step S2, the parametric 3D human body reconstruction method is based on the SMPL-X model and the HybrIK-X regression algorithm. The vertex generation formula for the 3D human body mesh model is: ;in Here, S represents the original 3D human body mesh model, and S is the vertex generation function of the SMPL-X model. For shape parameters, These are attitude parameters. For facial expression parameters; the HybrIK-X regression algorithm takes the joint positions from step S1 as input and regresses to obtain... and .

4. The method for three-dimensional constrained human body deformation based on semantic parameter mapping according to claim 1, characterized in that, In step S3, the semantic parameters include height H, weight W, bust B, waist Wa, and hip circumference Hip; the model body shape database is constructed by collecting images of models in standard poses, and performing step S2 on each image to obtain shape parameters. Connect H, W, B, Wa, and Hip with Associative storage forms a mapping relationship between semantic parameters and shape parameters.

5. The method for three-dimensional constrained human body deformation based on semantic parameter mapping according to claim 1, characterized in that, In step S4, the correspondence between the three-dimensional human body space and the two-dimensional image space is realized through the camera projection matrix, and the formula is: ;in The mapped two-dimensional coordinates The projection matrix is ​​composed of the camera's intrinsic and extrinsic parameters. The coordinates of the three-dimensional human body joints or contour points are obtained through three-dimensional model partitioning, mask projection, and contour filtering. Specifically, the three-dimensional mesh is divided into sub-regions such as left arm, right arm, left leg, right leg, and torso. After projecting to generate a mask image, the outermost contour points of the human body are filtered out.

6. The method for three-dimensional constrained human body deformation based on semantic parameter mapping according to claim 1, characterized in that, In step S6, the formula for the body shape deformation energy term is: The formula for the neighborhood regularization term is: ; The formula for the deviation penalty term is: ;in For the energy term of body shape deformation, This is the set of vertex positions of the deformed triangular mesh. The original two-dimensional mesh edge vectors, These are the edge vectors of the original 3D mesh. The edge vectors of the deformed 2D mesh. These are the edge vectors of the deformed 3D mesh. For neighborhood regularization, Let i be the set of adjacent triangles of the i-th triangle. Let be the affine transformation matrix of the i-th triangle. Let be the affine transformation matrix of the j-th triangle. Let I be the deviation penalty term, and let I be the identity matrix.

7. The method for three-dimensional constrained human body deformation based on semantic parameter mapping according to claim 1 or 6, characterized in that, In step S6, the optimization objective formula for the least squares linear sparse system is: ;in To optimize the overall energy value, The weight of the energy term along the skeletal direction. This is an energy term along the skeletal direction. The weight of the energy term in the vertical bone direction. This is the energy term in the vertical bone direction. For the contour energy term weights, For the contour energy term, The weight of the neighborhood regularization term. For neighborhood regularization, The weight of the deviation penalty term, This is a penalty item for deviations; The sparse system is pre-decomposed into a system of linear equations Perform QR decomposition on matrix A and store the decomposition results, where A is the coefficient matrix and b is the right-hand vector.

8. The method for human body three-dimensional constrained deformation based on semantic parameter mapping according to claim 3, characterized in that, In step S7, the formula for generating the 3D mesh of the target body is: ;in Let S be the 3D mesh of the target body, and S be the vertex generation function of the SMPL-X model. For the target shape parameters, The original attitude parameters obtained in step S2, For facial expression parameters; The boundary conditions of a sparse system are updated by calculating the ratio of the length changes of corresponding edges between the original 3D human body mesh model and the target body mesh. And substitute it into the body deformation energy term to update the vector b on the right side of the equation system.

9. A system for realizing the three-dimensional constrained body shape deformation of the human body as described in any one of claims 1-8, characterized in that, Includes a human body shape deformation device, the human body shape deformation device comprising: The joint detection module is used to execute step S1 and output the two-dimensional position information of human joints. The 3D human body reconstruction module is used to execute step S2 and output a 3D human body mesh model. ; The body shape database creation module is used to execute step S3 and store the correspondence between semantic parameters and SMPL-X shape parameters; The image mesh generation module is used to execute steps S4-S5 and output a two-dimensional triangular mesh. ; The sparse system construction and solution module is used to execute steps S6-S7 and output a human body shape deformation image; The output of the joint detection module is connected to the input of the 3D human body reconstruction module; the output of the 3D human body reconstruction module is connected to the input of the body shape database creation module and the image mesh generation module respectively; the output of the image mesh generation module is connected to the input of the sparse system construction and solution module; and the output of the body shape database creation module is connected to the input of the sparse system construction and solution module.

10. The system according to claim 9, characterized in that, It also includes an electronic device, which includes an electrically connected memory, a processor, and a communication unit; the memory is capable of storing the program code of the human body deformation device; the processor executes the program code to realize the functions of each module; and the communication unit is used to realize data communication between the electronic device and external devices.

Images