Human body parametric 3D model deformation method based on neural network joint point estimation

Through the neural network-based node estimation method, the problem of low accuracy of the human parametric three-dimensional model is solved, and higher model accuracy and action simulation effects are achieved.

CN112233222BActive Publication Date: 2024-11-22SHENZHEN ESUN DISPLAY
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202011046379.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-29
Publication Date
2024-11-22
Estimated Expiration
2040-09-29

AI Technical Summary

Technical Problem

When obtaining a parameterized three-dimensional model of the human body, the prior art is susceptible to factors such as the measured human body movement speed, resulting in low model accuracy.

Method used

Using the neural network-based estimation method, the three-dimensional joint nodes are obtained by obtaining the two-dimensional image of the target three-dimensional object model, performing joint node detection and spatial mapping, and obtaining the three-dimensional joint nodes, and then fitting the three-dimensional object model, and controlling the model by fitting the three-dimensional joint nodes to improve the accuracy and similarity of the model.

Benefits of technology

The accuracy of the human parametric three-dimensional model and the similarity between the fitting model and the target object are improved, so that the fitting model can more accurately simulate the movements of the real human body.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112233222B_ABST
    Figure CN112233222B_ABST
Patent Text Reader

Abstract

The present application relates to a method, device, computer equipment and storage medium for deformation of a parametric three-dimensional model of a human body based on neural network joint point estimation. The method comprises: extracting a two-dimensional image of a target three-dimensional object model to obtain a set of target two-dimensional object images; performing two-dimensional joint point detection on target two-dimensional object images in the set of target two-dimensional object images using a target two-dimensional joint point detection model to obtain a set of target two-dimensional joint points corresponding to each target two-dimensional object image; obtaining target three-dimensional joint points corresponding to the target two-dimensional joint points according to the spatial mapping relationship corresponding to each target two-dimensional joint point in the set of target two-dimensional joint points corresponding to the target two-dimensional object images; performing object fitting using the target three-dimensional joint points corresponding to each target two-dimensional joint point to obtain a fitted three-dimensional object model corresponding to the target three-dimensional object model. The use of this method can improve the accuracy of the parametric model of the human body.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, computer equipment and storage medium for deforming a parametric three-dimensional model of a human body based on neural network joint point estimation. Background Art

[0002] With the development of computer technology, 3D (three dimensional) technology has emerged, and the application of human body 3D models in 3D printing, clothing customization, virtual fitting, and animation production has been continuously explored. For example, a human body parametric model similar to a real person can be used to realize a virtual catwalk. Therefore, in 3D applications, how to obtain a human body parametric model similar to a real person is particularly important.

[0003] At present, depth cameras and other equipment are usually used to measure the human body to obtain a human body parametric model. However, this method is easily affected by factors such as the movement speed of the measured human body, resulting in the failure to obtain a human body parametric model that is very similar to the real human body, that is, the obtained human body parametric model has low accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a human body parametric three-dimensional model deformation method, device, computer equipment and storage medium based on neural network joint point estimation, which can improve the accuracy of the human body parametric model in order to address the technical problem of low accuracy of the human body parametric model obtained above.

[0005] A method for deforming a parameterized three-dimensional model of a human body based on neural network joint point estimation, the method comprising: obtaining a target three-dimensional object model, performing two-dimensional image extraction on the target three-dimensional object model to obtain a target two-dimensional object image set, wherein the target three-dimensional object model is obtained by scanning the target object; obtaining a target two-dimensional joint point detection model, performing two-dimensional joint point detection on target two-dimensional object images in the target two-dimensional object image set using the target two-dimensional joint point detection model to obtain a target two-dimensional joint point set corresponding to each of the target two-dimensional object images; obtaining target three-dimensional joint points corresponding to the target two-dimensional joint points according to the spatial mapping relationship corresponding to each of the target two-dimensional joint points in the target two-dimensional joint point set corresponding to the target two-dimensional object images; performing object fitting using the target three-dimensional joint points corresponding to each of the target two-dimensional joint points to obtain a fitting three-dimensional object model corresponding to the target three-dimensional object model, so as to control the fitting three-dimensional object model to move.

[0006] In some embodiments, obtaining the target two-dimensional joint point detection model includes: obtaining an initial two-dimensional joint point detection model and a training two-dimensional object image; determining an image training label corresponding to the training two-dimensional object image, wherein the image training label is determined based on the labeled two-dimensional joint points in a set of labeled two-dimensional joint points corresponding to the training two-dimensional object image; training the initial two-dimensional joint point detection model based on the training two-dimensional object image and the image training label corresponding to the training two-dimensional object image to obtain the target two-dimensional joint point detection model.

[0007] In some embodiments, the image training label includes confidence maps corresponding to each labeled two-dimensional joint point in the labeled two-dimensional joint point set; the method also includes: determining the target feature position of the target feature point corresponding to the labeled two-dimensional joint point based on the feature information corresponding to the labeled two-dimensional joint point; obtaining the associated feature position of the associated feature point corresponding to the target feature point, and determining the confidence map corresponding to the labeled two-dimensional joint point based on the difference between the target feature position and the associated feature position.

[0008] In some embodiments, the image training label includes a limb vector map corresponding to the target limb, and the joint points corresponding to the target limb are the labeled two-dimensional joint points in the labeled two-dimensional joint point set; the method also includes: obtaining a target limb vector corresponding to the target limb, and the target limb vector is determined according to the position of the joint points corresponding to the target limb; determining a target point vector corresponding to the target limb point on the target limb according to the target limb vector; and obtaining a limb vector map corresponding to the target limb according to the target point vectors corresponding to each of the target limb points.

[0009] In some embodiments, the object fitting is performed using the target three-dimensional joint points corresponding to each of the target two-dimensional joint points to obtain the fitting three-dimensional object model corresponding to the target three-dimensional object model, including: obtaining a standard three-dimensional object model and a set of standard three-dimensional joint points corresponding to the standard three-dimensional object model; the set of standard three-dimensional joint points includes multiple standard three-dimensional joint points; obtaining target joint point constraints, the target joint point constraints include the difference between a limb length adjustment item and a target limb length, the limb length adjustment item is the difference result between a first joint point adjustment item and a second joint point adjustment item; the first joint point adjustment item is the result of an operation between the position of the first standard three-dimensional joint point and a morphological adjustment parameter, and the second joint point adjustment item is the result of an operation between the position of the second standard three-dimensional joint point and the morphological adjustment parameter; the first standard three-dimensional joint point and the second standard three-dimensional joint point correspond to a target limb type, and the target limb length is the position difference between the first target three-dimensional joint point corresponding to the target limb type and the corresponding second target three-dimensional joint point; when determining that the target joint point constraint item satisfies the minimization constraint condition, the value corresponding to the morphological adjustment parameter is obtained to obtain a target position adjustment factor; and the joint points of the standard three-dimensional object model are adjusted in position using the target position adjustment factor to obtain the fitting three-dimensional object model.

[0010] In some embodiments, the target joint point constraint item also includes a third joint point adjustment item corresponding to the first standard three-dimensional joint point, and the difference between the position of the first target three-dimensional joint point, and the third joint point adjustment item is the result of calculation of the posture adjustment parameter and the first joint point adjustment item; the first target three-dimensional joint point is consistent with the joint point type of the first standard three-dimensional joint point.

[0011] In some embodiments, when determining that the target joint point constraint item satisfies the minimization constraint condition, the numerical value corresponding to the morphology adjustment parameter to obtain the target position adjustment factor includes: when determining that the target joint point constraint item satisfies the minimization constraint condition, the numerical value corresponding to the morphology adjustment parameter and the numerical value corresponding to the posture adjustment parameter to obtain the target position adjustment factor.

[0012] A human body parameterized three-dimensional model deformation device based on neural network joint point estimation, the device comprises: a target two-dimensional object image set acquisition module, used to acquire a target three-dimensional object model, perform two-dimensional image extraction on the target three-dimensional object model, and obtain a target two-dimensional object image set, wherein the target three-dimensional object model is obtained by scanning the target object; a target two-dimensional joint point set acquisition module, used to acquire a target two-dimensional joint point detection model, perform two-dimensional joint point detection on the target two-dimensional object images in the target two-dimensional object image set using the target two-dimensional joint point detection model, and obtain a target two-dimensional joint point set corresponding to each of the target two-dimensional object images; a target three-dimensional joint point acquisition module, used to obtain target three-dimensional joint points corresponding to the target two-dimensional joint points according to the spatial mapping relationship corresponding to each of the target two-dimensional joint points in the target two-dimensional joint point set corresponding to the target two-dimensional object images; and a fitting three-dimensional object model acquisition module, used to perform object fitting using the target three-dimensional joint points corresponding to each of the target two-dimensional joint points, and obtain a fitting three-dimensional object model corresponding to the target three-dimensional object model, so as to control the fitting three-dimensional object model to move.

[0013] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned human body parameterized three-dimensional model deformation method based on neural network joint point estimation are implemented.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned human body parametric three-dimensional model deformation method based on neural network joint point estimation.

[0015] The above-mentioned human body parameterized three-dimensional model deformation method, device, computer equipment and storage medium based on neural network joint point estimation obtain a target three-dimensional object model, perform two-dimensional image extraction on the target three-dimensional object model to obtain a target two-dimensional object image set, obtain a target two-dimensional joint point detection model, use the target two-dimensional joint point detection model to perform two-dimensional joint point detection on the target two-dimensional object images in the target two-dimensional object image set, obtain a target two-dimensional joint point set corresponding to each target two-dimensional object image, obtain the target three-dimensional joint point corresponding to the target two-dimensional joint point according to the spatial mapping relationship corresponding to each target two-dimensional joint point in the target two-dimensional joint point set corresponding to the target two-dimensional object image, and use the target three-dimensional joint point corresponding to each target two-dimensional joint point to perform object fitting to obtain a fitting three-dimensional object model corresponding to the target three-dimensional object model. Thus, the three-dimensional joint points of the three-dimensional object model are accurately obtained through the two-dimensional joint points of the two-dimensional object image, and the accuracy of the three-dimensional joint points is improved, so that the similarity between the fitting three-dimensional object model and the target three-dimensional object model can be improved, thereby improving the accuracy of the fitting three-dimensional object model. In addition, by controlling the fitted three-dimensional object model according to the target three-dimensional joint points, the fitted three-dimensional object model can be made to move in the form of the target three-dimensional object model, thereby simulating the movements of a real human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A diagram of an application environment of a human body parameterized three-dimensional model deformation method based on neural network joint point estimation in some embodiments;

[0017] Figure 2 A schematic diagram of a process of a human body parameterized three-dimensional model deformation method based on neural network joint point estimation in some embodiments;

[0018] Figure 3 is a schematic diagram of a target three-dimensional object model in some embodiments;

[0019] Figure 4 is a schematic diagram of a two-dimensional object image in some embodiments;

[0020] Figure 5 A schematic diagram of obtaining three-dimensional joint points according to two-dimensional joint points in some embodiments;

[0021] Figure 6 is a schematic diagram of a target three-dimensional object model and a standard three-dimensional object model in some embodiments;

[0022] Figure 7 A schematic diagram of obtaining a fitted three-dimensional object model through initial matching and non-rigid registration in some embodiments;

[0023] Figure 8A schematic diagram of a three-dimensional object model after initial registration and a non-rigid registration three-dimensional object model in some embodiments;

[0024] Fig. 9 A schematic diagram of a flow chart of steps for obtaining a target two-dimensional joint point detection model in some embodiments;

[0025] Fig.10 is a schematic diagram of a two-dimensional joint point detection model in some embodiments;

[0026] Fig.11 is a schematic diagram of a parameterized model in some embodiments;

[0027] Fig.12 is a schematic diagram of three-dimensional joint points marked in a three-dimensional human body model in some embodiments;

[0028] Fig.13 A schematic flow chart of steps for obtaining a fitted three-dimensional object model in some embodiments;

[0029] Fig.14 These are effect diagrams of shape adjustment and posture adjustment in some embodiments;

[0030] Fig.15 A schematic diagram of initially registering a three-dimensional object model and non-rigidly registering a three-dimensional object model in some embodiments;

[0031] Fig.16 is a schematic diagram of a process of non-rigid registration in some embodiments;

[0032] Fig.17 It is a structural block diagram of a human body parameterized three-dimensional model deformation device based on neural network joint point estimation in some embodiments;

[0033] Fig.18 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] The human body parameterized 3D model deformation method based on neural network joint point estimation provided in this application can be applied to Figure 1 In the application environment shown, the application environment includes a scanning device 102 and a terminal device 104. The scanning device 102 can communicate with the terminal device 104 via wired or wireless communication.

[0036] Specifically, the scanning device 102 can scan the object to obtain a scanned three-dimensional (3D, three-dimensional) object model. The terminal device 104 can obtain the scanned three-dimensional object model from the scanning device 102, for example, the target three-dimensional object model obtained by scanning the target object can be obtained from the scanning device 102. The terminal device 104 can perform two-dimensional (2D, two-dimensional) image extraction on the target three-dimensional object model to obtain a target two-dimensional object image set; the terminal device 104 can obtain a target two-dimensional joint point detection model, and use the target two-dimensional joint point detection model to perform two-dimensional joint point detection on the target two-dimensional object images in the target two-dimensional object image set to obtain a target two-dimensional joint point set corresponding to each target two-dimensional object image; the terminal device 104 can obtain the target three-dimensional joint point corresponding to the target two-dimensional joint point set according to the spatial mapping relationship corresponding to each target two-dimensional joint point in the target two-dimensional joint point set corresponding to the target two-dimensional object image; the terminal device 104 can use the target three-dimensional joint points corresponding to each target two-dimensional joint point to perform object fitting, obtain a fitting three-dimensional object model corresponding to the target three-dimensional object model, and can control the fitting three-dimensional object model to move according to the target three-dimensional joint point.

[0037] The scanning device 102 may be, but is not limited to, various devices capable of scanning a three-dimensional human body model, such as a laser scanner. The terminal device 104 may be, but is not limited to, various personal computers and notebook computers.

[0038] It is understood that the above application scenario is only an example and does not constitute a limitation on the method provided in the embodiment of the present application. The method provided in the embodiment of the present application can also be applied in other scenarios. For example, the human body parametric three-dimensional model deformation method based on neural network joint point estimation provided in the present application can be executed by a server device. The server device can be implemented by an independent server or a server cluster composed of multiple servers.

[0039] In some embodiments, Figure 2 As shown in the figure, a method for human body parametric 3D model deformation based on neural network joint point estimation is provided. Figure 1 The terminal device 104 in the example is used as an example to illustrate, and the following steps are included:

[0040] S202, acquiring a target three-dimensional object model, performing two-dimensional image extraction on the target three-dimensional object model, and obtaining a target two-dimensional object image set. The target three-dimensional object model is obtained by scanning the target object.

[0041] Specifically, the object may include at least one of a person or an animal. The target object is an object corresponding to the target three-dimensional object model. When the target object is a person, the target three-dimensional object model may be referred to as a target three-dimensional human body model, and when the target object is an animal, the target three-dimensional object model may be referred to as a target three-dimensional animal body model. The target three-dimensional object model may be a three-dimensional object point cloud model obtained by scanning the target object with a point cloud scanning device, for example, a three-dimensional human body point cloud model. Figure 3 As shown, a target three-dimensional object model is shown.

[0042] The target two-dimensional object image set may include multiple target two-dimensional object images. The target two-dimensional object image is a two-dimensional object image corresponding to the target three-dimensional object model. Extracting the two-dimensional image of the target three-dimensional object model may include projecting the target three-dimensional object model, that is, the target two-dimensional object image may be obtained by projecting the target three-dimensional object model, for example, each target two-dimensional object image in the target two-dimensional object image set may be each two-dimensional image obtained by projecting the target three-dimensional object model in different directions.

[0043] In some embodiments, the target two-dimensional object image may also be obtained by performing two-dimensional image rendering on the target three-dimensional object model, for example, it may be an RGB image obtained by rendering. The terminal device may perform two-dimensional image rendering on the target three-dimensional object model through OpenGL to obtain target two-dimensional object images at multiple angles, for example, a target two-dimensional object image at 0 degrees and a target two-dimensional object image at 180 degrees. When the target two-dimensional object image at 0 degrees is a front human body image, the target two-dimensional object image at 180 degrees is a back human body image. Figure 4 As shown, the rendered two-dimensional object image is shown.

[0044] S204, obtaining a target two-dimensional joint point detection model, and using the target two-dimensional joint point detection model to perform two-dimensional joint point detection on target two-dimensional object images in a target two-dimensional object image set to obtain target two-dimensional joint point sets corresponding to each target two-dimensional object image.

[0045] Specifically, joints may also be referred to as key points, and the types of joints may include head joints, neck joints, left hand joints, left elbow joints, left shoulder joints, right hand joints, right elbow joints, right shoulder joints, pelvic joints, left foot joints, left knee joints, left hip joints, right foot joints, right knee joints, and right hip joints, etc.

[0046] A two-dimensional joint point detection model refers to a model for performing two-dimensional joint point detection on a two-dimensional object image. The input of the two-dimensional joint point detection model is a two-dimensional object image, and the output of the two-dimensional joint point detection model may include at least one of a two-dimensional joint point and a confidence map corresponding to the two-dimensional joint point. Each type of joint point may correspond to a confidence map, and the confidence map may be a Gaussian distribution map centered on the joint point. The target two-dimensional joint point detection model is a trained two-dimensional joint point detection model obtained after training with a two-dimensional object image. The greater the confidence of a joint point at a certain position coordinate, the more likely the joint point is to be located at the position coordinate, so the terminal device can select the position with the highest confidence in the confidence map to obtain the position of the joint point.

[0047] The target two-dimensional joint point set may include multiple target two-dimensional joint points. The target two-dimensional joint points may correspond to the types of relevant nodes and the position information of the joint points in the target two-dimensional object image. The target two-dimensional joint points are obtained according to the results output by the target two-dimensional joint point detection model. For example, the target two-dimensional joint points may be two-dimensional joint points output by the target two-dimensional joint point detection model, or may be joint points obtained by processing at least one of the two-dimensional joint points output by the target two-dimensional joint point detection model or the joint points obtained by the output confidence map. For example, each two-dimensional joint point output by the target two-dimensional joint point detection model may be processed by at least one of missing or symmetric judgment.

[0048] In some embodiments, when the output result of the target two-dimensional joint point detection model is a two-dimensional joint point, the terminal device can input the target two-dimensional object image into the target two-dimensional joint point detection model to obtain at least one of the two-dimensional joint point output by the model, the type of the two-dimensional joint point, or the position of the two-dimensional joint point in the target two-dimensional object image. When the output result of the target two-dimensional joint point detection model is a confidence map corresponding to the two-dimensional joint point, the terminal device can determine the type of the corresponding joint point and the position in the target two-dimensional object image according to the confidence map output by the target two-dimensional joint point detection model.

[0049] S206, obtaining target three-dimensional joint points corresponding to the target two-dimensional joint points according to the spatial mapping relationships corresponding to the target two-dimensional joint points in the target two-dimensional joint point sets corresponding to the target two-dimensional object images.

[0050] Specifically, the spatial mapping relationship can be customized or pre-set, and refers to the correspondence between the position of a two-dimensional joint point and the position of a corresponding three-dimensional joint point. The position of the three-dimensional joint point can be obtained from the position of the two-dimensional joint point through the spatial mapping relationship, and the position of the two-dimensional joint point can also be obtained from the position of the three-dimensional joint point according to the spatial mapping relationship.

[0051] In some embodiments, the terminal device may determine the spatial mapping relationship between the three-dimensional joint points and the two-dimensional joint points based on the parameters of the camera. The parameters of the camera may include the intrinsic parameters and extrinsic parameters of the camera. For example, the spatial mapping relationship may be determined using OpenGL and the parameters of the camera. Figure 5 , shows a schematic diagram of obtaining three-dimensional joint points based on two-dimensional joint points. First, based on the three-dimensional human body model, the two-dimensional renderings of the front and back are obtained. Then, the target two-dimensional joint point detection model is used to detect the two-dimensional joint points of the two-dimensional renderings of the front and back to obtain the two-dimensional joint points of the two-dimensional renderings of the front and back. Then, based on the two-dimensional joint points of the two-dimensional renderings of the front and back, the three-dimensional joint points of the three-dimensional human body model are obtained.

[0052] S208, performing object fitting using the target three-dimensional joint points corresponding to the target two-dimensional joint points, to obtain a fitted three-dimensional object model corresponding to the target three-dimensional object model, so as to control the fitted three-dimensional object model to move.

[0053] Specifically, the fitted three-dimensional object model can be obtained by fitting the target three-dimensional joint points and the standard three-dimensional object model. The fitting may include adjusting the morphology or posture in the standard three-dimensional object model through the target three-dimensional joint points. For example, the positions of the vertices or the positions of the joints of the standard three-dimensional object model may be adjusted according to the target three-dimensional joint points to achieve fitting. The posture and morphology may also be adjusted by existing methods, for example, by the LBS (Linear Blend Skinning) method. The fitting process may also include adjusting the vertex positions of the standard three-dimensional object model by the vertex positions corresponding to the target three-dimensional object model. Among them, the stage of adjusting the morphology and posture may be called the initial matching stage, and the stage of adjusting the vertices may be called the non-rigid registration stage.

[0054] The standard 3D object model can also be called a standard parametric model. When the object is a human, it can also be called a standard 3D human body model. The standard 3D object model refers to a parametric model with clear vertices and joints, and the positions of the vertices and joints can be adjusted. It can be a custom 3D object model or an existing 3D object model, such as a makehuman parametric model. Figure 6 As shown, Figure 6 The two human models in (a) are schematic diagrams of the target three-dimensional object models. Figure 6 The two human body models in (b) are schematic diagrams of standard three-dimensional object models.

[0055] In some embodiments, the differences in the three-dimensional object models are mainly reflected in two aspects: morphology and posture. The terminal device can use the target three-dimensional joint points to adjust the morphology of the standard three-dimensional object model, optimize the length of the limbs and trunk of the standard three-dimensional object model, and obtain a fitting three-dimensional object model, so that the fitting three-dimensional object model and the target three-dimensional object model (i.e., the actual object model) have the same length of limbs and trunk. For example, the terminal device can estimate the length of human limbs according to each target three-dimensional joint point, and adjust the position of the joint point of the standard three-dimensional object model according to the estimated length of human limbs. The terminal device can also use the target three-dimensional joint point to adjust the posture of the standard three-dimensional object model or the standard three-dimensional object model after morphology adjustment to obtain a fitting three-dimensional object model, so that the fitting three-dimensional object model and the target three-dimensional object model have the same posture, for example, the terminal device can optimize the bone rotation angle of the standard three-dimensional object model according to the rotation angle of the target three-dimensional object model.

[0056] In some embodiments, the terminal device can also adjust the positions of the vertices of the three-dimensional object model obtained in the initial matching stage to complete the non-rigid registration and obtain a fitted three-dimensional object model, so that the fitted three-dimensional object model and the target three-dimensional object model have the same body shape, such as the same arm thickness and length, and the same waist size. Figure 7 As shown in FIG, a schematic diagram of obtaining a fitted 3D object model through initial matching and non-rigid registration is shown. Figure 8 As shown, Figure 8 The left image in (a) corresponds to the target 3D object model. Figure 8 The left image in (a) corresponds to the standard three-dimensional object model. Figure 8 (b) corresponds to the 3D object model after initial matching, Figure 8 (c) Corresponding non-rigid registration 3D object model.

[0057] In some embodiments, the terminal device can calculate the size of the fitted three-dimensional object model, and estimate the size of the target object based on the size of the fitted three-dimensional object model, so that the object size can be measured automatically, and the human body size can be measured automatically, thereby improving the efficiency and accuracy of human body size measurement. The terminal device can also control the position of the joint points of the fitted three-dimensional object model so that the fitted three-dimensional object model moves according to the movement of the real human body.

[0058] In some embodiments, the terminal device can adjust the posture and form of the standard three-dimensional object model through formula (1).

[0059] M(β,θ)=W(T(β),J(β),θ,w) (1)

[0060] in,

[0061]

[0062]

[0063] Among them, β is a morphological parameter (also called a morphological adjustment parameter), θ is a posture parameter (also called a posture adjustment parameter), T is a standard three-dimensional object model, T(β) is a morphological model in a standard posture, and T(β) is obtained by applying a number of offsets B(β) on the basis of T. Different morphological parameters β constitute different offsets, reflecting different morphological models. M(β, θ) is obtained by a posture of a model T(β) in a given standard posture, which is controlled by the posture parameter θ. M(β, θ) can be understood as the human body model rotating around each joint point J(β) by an angle θ. J(β) is the joint position in the standard posture, which is affected by the morphological parameter β. The length of different parts of the human body can be controlled by the morphological adjustment parameters, such as controlling the length of arms and legs.

[0064] Usually, after a parameterized model is determined, the corresponding T, morphological base S n , the bone weights w (such as LBS bone weights) are known. Therefore, to determine a human body model, the fitted parameterized model can be obtained by estimating the parameters β and θ.

[0065] In some embodiments, the fitting process can be divided into two steps: constructing an error equation and minimizing the error equation. The terminal device can construct the error equation according to the fitting target, and the fitting target can be, for example, a point cloud of a human body model. For example, the error equation can be expressed as formula (5).

[0066] E(β,θ)=∑||M(β,θ) i -V corr(i) || 2 +E reg (5)

[0067] Among them, M(β,θ) i represents a point on the model M(β,θ), V corr(i) Represents the human body point cloud model and M(β,θ) i Corresponding points, coor(i) represents the corresponding relationship of dynamic changes, E reg represents the regularization term. The minimization error equation can be expressed as formula (6)

[0068]

[0069] In the above-mentioned human body parameterized three-dimensional model deformation method based on neural network joint point estimation, a target three-dimensional object model is obtained, a two-dimensional image is extracted from the target three-dimensional object model to obtain a target two-dimensional object image set, a target two-dimensional joint point detection model is obtained, and a target two-dimensional joint point detection model is used to detect two-dimensional joint points of the target two-dimensional object images in the target two-dimensional object image set to obtain a target two-dimensional joint point set corresponding to each target two-dimensional object image, and a target three-dimensional joint point corresponding to the target two-dimensional joint point set is obtained according to the spatial mapping relationship corresponding to each target two-dimensional joint point in the target two-dimensional joint point set corresponding to the target two-dimensional object image, and an object is fitted using the target three-dimensional joint points corresponding to each target two-dimensional joint point to obtain a fitting three-dimensional object model corresponding to the target three-dimensional object model. Thus, the three-dimensional joint points of the three-dimensional object model are accurately obtained through the two-dimensional joint points of the two-dimensional object image, and the accuracy of the three-dimensional joint points is improved, so that the similarity between the fitting three-dimensional object model and the target three-dimensional object model can be improved, thereby improving the accuracy of the fitting three-dimensional object model. In addition, the fitting three-dimensional object model is controlled according to the target three-dimensional joint points, so that the fitting three-dimensional object model can move in the form of the target three-dimensional object model, thereby simulating the action of a real human body.

[0070] In some embodiments, Fig. 9 As shown, obtaining the target two-dimensional joint point detection model in step S204 includes:

[0071] S902, obtaining an initial two-dimensional joint point detection model and a training two-dimensional object image.

[0072] Specifically, the training two-dimensional object image can be obtained by extracting a two-dimensional image from the training three-dimensional object model. The training three-dimensional object model can be obtained by scanning the training object, and the training object can be an actual human body, can be obtained by scanning a human body with a specific posture, or can be obtained by scanning a human body with an arbitrary posture. The specific posture can be, for example, the A Pose posture. The training three-dimensional object model can be a fixed posture or an arbitrary posture. When scanning the human body, in order to ensure the accuracy of the scanned model, a human body wearing a tights can be selected for scanning, and a variety of human bodies can be selected for scanning, for example, human bodies of different heights or different fatness can be selected for scanning. When scanning the same training object, multiple models can be scanned by changing the posture of the training object, so that more data can be obtained and data duplication can be avoided. For example, when scanning the human body, the human body can be kept in the A Pose posture, and the degree of opening of the human body's arms can be changed, so as to avoid data duplication.

[0073] The initial 2D joint point detection model refers to an untrained 2D joint point detection model, which can be a custom neural network model, an existing neural network model, or a deformation of an existing neural network model. For example, it can be a convolutional neural network model, specifically Fig.10 The neural network model shown in (denoted as neural network model A). Neural network model A includes a baseline network and multiple task modules. Among them, the task module refers to Stage1 (stage 1) to Staget (stage t) in the figure, t≥2. The baseline network can be a deformation of the existing network model, for example, it can be the first 10 layers of VGG19 (Visual Geometry Group 19). Each Stage includes two branches, namely Branch1 and Branch2. Branch1 is used to predict the confidence map corresponding to each joint point, and Branch2 is used to preset the affinity vector. The input of the baseline network includes a training two-dimensional object image, and the output of the baseline network includes image features, namely F in the figure. The input of Stage1 is F, and the input of Stage2 to Staget includes F and the output of the previous task module. For example, the input of Stage2 includes F and the output of Stage1. The output of each Stage includes the corresponding outputs of Branch1 and Branch2. For example, the output of Stage1 includes the corresponding output S of Branch1. 1 and the corresponding Branch2 output L 1 . S 1 Represents the confidence map, which can be expressed as S 1 =ρ 1 (F), L 1 Represents the affinity vector, which can be expressed as L 1 =φ 1 (F), where ρ 1 Represents the CNN (Convolutional Neural Networks) structure corresponding to Branch1 in Stage1, φ 1 Represents the CNN structure corresponding to Branch2 in Stage1. t represents the CNN corresponding to Branch1 in Staget, φ t Represents the CNN structure corresponding to Branch2 in Staget. t The output can be expressed as S t =ρ t (F,S t-1 ,L t-1 ), φ t The output can be expressed as L t =φ t (F,S t-1,L t-1 ). Each Stage corresponds to two loss functions, and the loss function corresponding to Staget is, for example, Formula (7) and Formula (8). Among them, represents the true confidence map (also called heat map), represents the real body joint connection vector (also called affinity vector), J represents the number of joint points, j represents the jth joint point, C represents the number of limbs, and c represents the cth limb. P represents the point in the image. One limb corresponds to two joint points. The overall loss function of the neural network model A can be expressed as formula (9).

[0074]

[0075]

[0076]

[0077] S904, determining an image training label corresponding to the training two-dimensional object image, wherein the image training label is determined according to the labeled two-dimensional joint points in the labeled two-dimensional joint point set corresponding to the training two-dimensional object image.

[0078] Specifically, the image training label is a label corresponding to the training two-dimensional object image. The image training label may include confidence maps corresponding to each labeled two-dimensional joint point in the labeled two-dimensional joint point set, and may also include a limb vector map corresponding to the target limb, and the joint point corresponding to the target limb is the labeled two-dimensional joint point in the labeled two-dimensional joint point set. A limb is composed of two joints of the human body and the parts between the joints, and one limb corresponds to two joints. Limb types may include calves, thighs, forearms, and upper arms, etc. The target limb can be selected as needed, there may be multiple, and may include different types of limbs.

[0079] The set of annotated two-dimensional joint points may include multiple annotated two-dimensional joint points. The annotated two-dimensional joint points may be obtained by annotating the joint points of a training two-dimensional object image, or by annotating the three-dimensional joint points corresponding to a training three-dimensional object model. The annotated two-dimensional joint points are obtained by annotating the joint points of a training three-dimensional object model, and the training three-dimensional object model may be a three-dimensional human point cloud model obtained by scanning a real human body. Each annotated two-dimensional joint point and the type of each annotated three-dimensional joint point may be customized, or may be determined based on each joint point of an existing parametric model, for example, it may be each joint point corresponding to a makehuman parametric model. Fig.11 As shown in Figure 1, the corresponding joint points of the makehuman parameterized model are shown. Fig.12 As shown, the three-dimensional joint points annotated in the three-dimensional human body model are displayed.

[0080] In some embodiments, the terminal device can determine the target feature position of the target feature point corresponding to the marked two-dimensional joint point according to the feature information corresponding to the marked two-dimensional joint point, obtain the associated feature position of the associated feature point corresponding to the target feature point, and determine the confidence map corresponding to the marked two-dimensional joint point according to the difference between the target feature position and the associated feature position. Among them, the associated feature point can be a feature point adjacent to the target feature point in position, or a feature point whose distance from the target feature point is less than the feature distance threshold. The feature distance threshold can be set as needed. The marked two-dimensional object image can correspond to multiple feature points. The feature position can include a horizontal coordinate position and a vertical coordinate position. The confidence map corresponding to the marked two-dimensional joint point can be the difference between the target feature position and the associated feature position, or the square of the difference between the target feature position and the associated feature position, or it can be calculated based on the difference between the target feature position and the associated feature position and a preset formula, for example, it can be calculated based on the difference between the target feature position and the associated feature position and a two-dimensional Gaussian function.

[0081] In some embodiments, a target limb vector corresponding to a target limb is obtained, and the target limb vector is determined according to the position of the joint point corresponding to the target limb; a target point vector corresponding to the target limb point on the target limb is determined according to the target limb vector; and a limb vector map corresponding to the target limb is obtained according to the target point vectors corresponding to each target limb point. Wherein, the limb corresponds to a section of area, and the limb vector is used to indicate the direction of the limb, which can be determined according to the position of the joint point corresponding to the limb. For example, the starting point and the end point of the limb vector can be the joint point of the corresponding limb, respectively. The target limb vector is the limb vector corresponding to the target limb. The limb point is a point on the limb, such as a pixel point on the limb, and a limb can include multiple limb points. The point vector refers to the vector corresponding to the limb point. The point vector can be a corresponding limb vector, or it can be calculated based on the limb vector, for example, it can be a unit vector corresponding to the limb vector. The limb vector map can include each target point vector, and the limb vector map can include a horizontal map and a vertical map, wherein the horizontal map includes the horizontal coordinate information corresponding to each target point vector, and the vertical map includes the vertical coordinate information corresponding to each target point vector. The limb vector map can also be called a joint connection vector map or an affinity vector map.

[0082] S906 , training the initial two-dimensional joint point detection model according to the training two-dimensional object image and the image training label corresponding to the training two-dimensional object image to obtain a target two-dimensional joint point detection model.

[0083] Specifically, the terminal device can input the training two-dimensional object image into the initial two-dimensional joint detection model to obtain the result of the model output, calculate the model loss value corresponding to the initial two-dimensional joint detection model according to the difference between the image training label and the model output result, adjust the parameters of the initial two-dimensional joint detection model according to the model loss value, and obtain the target two-dimensional joint detection model. For example, the terminal device can use the gradient descent method to adjust the parameters of the initial two-dimensional joint detection model in the direction of decreasing the model loss value, and use the initial two-dimensional joint detection model after adjusting the parameters as the target two-dimensional joint detection model. Among them, the batchsize parameter can be set to 10 during training, the initial learning rate can be set to 0.001, and the exponential decay of the training iteration memory can be set to 0.96, and the optimizer can use Adam. Among them, the model output result is consistent with the type of the image training label. For example, when the image training label is a two-dimensional joint point, the model output result is also a two-dimensional joint point. When the image training label is a confidence map, the model output result is also a confidence map.

[0084] In the above embodiment, the initial two-dimensional joint point detection model is trained according to the training two-dimensional object images and the image training labels corresponding to the training two-dimensional object images to obtain the target two-dimensional joint point detection model. Since the image training labels are determined according to the labeled two-dimensional joint points in the labeled two-dimensional joint point set corresponding to the training two-dimensional object images, the model output results can be related to the two-dimensional joint points, so that the trained initial two-dimensional joint point detection model can be used to predict the two-dimensional joint points and the two-dimensional joint point related information.

[0085] In some embodiments, Fig.13 As shown, step S208 uses the target three-dimensional joint points corresponding to each target two-dimensional joint point to perform object fitting, and obtains a fitting three-dimensional object model corresponding to the target three-dimensional object model, including:

[0086] S1302, obtaining a standard three-dimensional object model and a standard three-dimensional joint point set corresponding to the standard three-dimensional object model, wherein the standard three-dimensional joint point set includes a plurality of standard three-dimensional joint points.

[0087] Specifically, the standard three-dimensional joint point set may include a plurality of standard three-dimensional joint points, and the standard three-dimensional joint points refer to the joint points of the standard three-dimensional object model. The number of standard three-dimensional joint points may be consistent with the number of target three-dimensional joint points, for example, both are 16 joint points, and the type of standard three-dimensional joint points may be consistent with the type of target three-dimensional joint points, for example, both are head joint points, neck joint points, left hand joint points, left elbow joint points and left shoulder joint points. The positions of each standard three-dimensional joint point in the standard three-dimensional joint point set are known.

[0088] S1304, obtaining target joint point constraints, the target joint point constraints include the difference between the limb length adjustment item and the target limb length, the limb length adjustment item is the difference between the first joint point adjustment item and the second joint point adjustment item; the first joint point adjustment item is the result of the operation between the position of the first standard three-dimensional joint point and the morphological adjustment parameter, the second joint point adjustment item is the result of the operation between the position of the second standard three-dimensional joint point and the morphological adjustment parameter; the first standard three-dimensional joint point and the second standard three-dimensional joint point correspond to the target limb type, and the target limb length is the position difference between the first target three-dimensional joint point corresponding to the target limb type and the corresponding second target three-dimensional joint point.

[0089] Specifically, the morphology adjustment parameter refers to a parameter for adjusting the morphology of a standard three-dimensional object model. The morphology adjustment parameter can adjust the position of one or more standard three-dimensional joint points, thereby adjusting the morphology. For example, the positional relationship between the joint points corresponding to the same limb can be adjusted, and the positional relationship between the joint points corresponding to different limbs can also be adjusted. The morphology adjustment parameter can be an independent parameter, or it can be a sub-parameter including multiple different functions, and each sub-parameter can be a sub-parameter for adjusting the distance between different joint points and the same joint point. For example, it can include a sub-parameter for adjusting the distance between a standard three-dimensional joint point A and a standard three-dimensional joint point B, and it can also include a sub-parameter for adjusting the distance between a standard three-dimensional joint point A and a standard three-dimensional joint point C. The length of different parts of the human body can be controlled by the morphology adjustment parameter, such as controlling the length of the arm and the length of the leg.

[0090] Limb types may include left forearm, right forearm, left upper arm, right upper arm, left calf, right calf, left thigh and right thigh, etc. The target limb type may be any limb type, for example, it may be left forearm. There may be multiple target limb types. The first standard three-dimensional joint point and the second standard three-dimensional joint point are joint points corresponding to the target limb type in the standard three-dimensional joint point set. The first target three-dimensional joint point and the second target three-dimensional joint point are joint points corresponding to the target limb type on the target three-dimensional object model.

[0091] The position difference between the first target three-dimensional joint point and the second target three-dimensional joint point (i.e., the target limb length) can be the distance between the first target three-dimensional joint point and the second target three-dimensional joint point, or the square of the distance between the first target three-dimensional joint point and the second target three-dimensional joint point. The first joint point adjustment item can be obtained by performing at least one of linear or nonlinear operations between the position of the first standard three-dimensional joint point and the morphological adjustment parameter, and the linear operation can be, for example, an addition operation and a multiplication operation. The second joint point adjustment item can be obtained by performing at least one of linear or nonlinear operations between the position of the second standard three-dimensional joint point and the morphological adjustment parameter. Each standard three-dimensional joint point can correspond to a relevant node adjustment item respectively, and the joint point adjustment item corresponding to the standard three-dimensional joint point is the result of the operation between the position of the standard three-dimensional joint point and the morphological adjustment parameter. Each standard three-dimensional joint point corresponds to the same morphological adjustment parameter.

[0092] The difference between the first joint point adjustment item and the second joint point adjustment item (i.e., the limb length adjustment item) may be a result obtained by performing at least one of an absolute value or a square (e.g., a norm square) on the result obtained by subtracting the first joint point adjustment item from the second joint point adjustment item. The difference between the limb length adjustment item and the target limb length may be a result obtained by performing at least one of an absolute value or a square (e.g., a norm square) on the result obtained by subtracting the limb length adjustment item from the target limb length. The difference between the limb length adjustment item and the target limb length (referred to as the limb length difference) is positively correlated with the target joint point constraint item.

[0093] In some embodiments, different target limb types may correspond to limb length adjustment items and target limb lengths, respectively. The terminal device may calculate the difference between the limb length adjustment items corresponding to different target limb types and the corresponding target limb lengths, obtain the limb length differences corresponding to each target limb type, and obtain the target joint point constraint items according to the limb length differences corresponding to each target limb type. The terminal device may obtain the target joint point constraint items according to the statistical results of the limb length differences corresponding to each target limb type.

[0094] In some embodiments, the terminal device may determine the morphology adjustment factor by calculating the minimization result corresponding to the statistical value of the limb length difference corresponding to each target limb type. For example, the terminal device may determine the morphology adjustment factor using the formula in formula (10). Where, ||J(β) i -J(β) j || 2 represents the limb length adjustment term, represents the target limb length, Indicates limb length difference, Represents the statistical value of the difference in limb length. Among them, i and j represent two different joints belonging to the same limb. J(β) i and J(β) j Corresponding to the joint points in the standard 3D object model, and Refers to the joint points in the target 3D object model. Formula (10) can also be called the energy equation.

[0095]

[0096] S1306, when determining that the target joint point constraint item satisfies the minimization constraint condition, the value corresponding to the morphological adjustment parameter is obtained to obtain the target position adjustment factor.

[0097] Specifically, the minimization constraint condition may include at least one of a value less than a preset value or a value equal to a fixed value. The preset value may be set as needed, for example, to 0.2, and the fixed value may be set as needed, for example, to 0.1. When the value corresponding to the morphological adjustment parameter changes, the value corresponding to the target joint point constraint item changes accordingly. The terminal device may determine that when the target joint point constraint item satisfies the minimization constraint condition, the value corresponding to the morphological adjustment parameter is used as the morphological adjustment factor. The target position adjustment factor may be obtained based on the morphological adjustment factor, for example, the target position adjustment factor may be a morphological adjustment factor, or the target position adjustment factor may be obtained based on the morphological adjustment factor and other adjustment factors, and other adjustment factors may be, for example, posture adjustment factors.

[0098] In some embodiments, the terminal device can set the morphology adjustment parameter in the target joint point constraint item to a specific value for calculation to obtain a calculation result. When the calculation result satisfies the minimization constraint condition, the corresponding specific value is used as the morphology adjustment factor, and the target position adjustment factor is obtained based on the morphology adjustment factor.

[0099] S1308: Use the target position adjustment factor to adjust the positions of the joint points of the standard three-dimensional object model to obtain a fitted three-dimensional object model.

[0100] Specifically, the terminal device can adjust the position of the joint point of the standard three-dimensional object model according to at least one of the morphology adjustment factor and the posture adjustment factor to obtain the fitted three-dimensional object model. The morphology adjustment factor can be used for adjustment first, and then the posture adjustment factor can be used for adjustment, or the posture adjustment factor can be used for adjustment first, and then the morphology adjustment factor can be used for adjustment, or the morphology adjustment factor and the posture adjustment factor can be used for adjustment at the same time.

[0101] In some embodiments, the terminal device can set the morphological adjustment parameters in the joint point adjustment items corresponding to each standard three-dimensional joint point as the morphological adjustment factor, obtain the joint point adjustment values ​​corresponding to each standard three-dimensional joint point, use the position corresponding to the joint point adjustment value as the update position of the standard three-dimensional joint point, adjust each standard three-dimensional joint point in the standard three-dimensional object model to the corresponding update position, and obtain the fitting three-dimensional object model. Since the limb length difference is positively correlated with the target joint point constraint item, the smaller the result of the target joint point constraint item, the smaller the limb length difference, that is, the smaller the difference between the limb length adjustment item and the target limb length, the smaller the difference between the limb length corresponding to the limb length adjustment item and the limb length corresponding to the target limb length. Therefore, adjusting each standard three-dimensional joint point in the standard three-dimensional object model to the corresponding update position can make the limb length of the adjusted standard three-dimensional object model consistent with the limb length of the target three-dimensional object model, so that the posture of the fitting three-dimensional object model is consistent with that of the target three-dimensional object model.

[0102] In the above embodiment, the target position adjustment factor is used to adjust the position of the joint points of the standard three-dimensional object model to obtain a fitted three-dimensional object model, so that the posture of the fitted three-dimensional object model is consistent with that of the target three-dimensional object model, thereby improving the accuracy of the fitted three-dimensional object model.

[0103] In some embodiments, the target joint point constraint item also includes a third joint point adjustment item corresponding to the first standard three-dimensional joint point, and the difference between the position of the first target three-dimensional joint point, the third joint point adjustment item is the result of calculation of the posture adjustment parameter and the first joint point adjustment item; the joint point type of the first target three-dimensional joint point is consistent with that of the first standard three-dimensional joint point.

[0104] Specifically, the posture adjustment parameter refers to a parameter for adjusting the posture of the standard three-dimensional object model. The posture adjustment parameter can adjust the position of one or more standard three-dimensional joint points, thereby adjusting the posture. The third joint point adjustment item can be the result of performing at least one of linear operations or nonlinear operations on the posture adjustment parameter and the first joint point adjustment item, for example, it can be the result after multiplying the posture adjustment parameter by the first joint point adjustment item. The consistency of the joint point type of the first target three-dimensional joint point and the first standard three-dimensional joint point means that the two are the same type of joint points, for example, the first target three-dimensional joint point and the first standard three-dimensional joint point are both left-hand joint points. The difference between the position of the third joint point adjustment item and the first target three-dimensional joint point can be the result of calculating at least one of the absolute value, square or norm of the result after subtracting the position of the third joint point adjustment item from the first target three-dimensional joint point.

[0105] In some embodiments, the terminal device can calculate the difference between the third joint point adjustment item corresponding to the first standard three-dimensional joint point corresponding to the target joint point type and the position of the first target three-dimensional joint point (recorded as the joint point position difference), and obtain the joint point position difference corresponding to the target joint point type. According to the statistical results of the joint point position differences corresponding to each target joint point type, the target joint point constraint item is obtained, that is, the target joint point constraint item can include the statistical results of the joint point position differences corresponding to different joint point types. There is a positive correlation between the target joint point constraint item and the joint point position difference. Among them, the target joint point type can be determined as needed, or it can be any joint point type. For example, the terminal device can determine the numerical value corresponding to the morphological adjustment parameter and the numerical value corresponding to the posture adjustment parameter according to formula (11), and obtain the target position adjustment factor. Among them, R (θ) is the rotation and translation matrix corresponding to the posture adjustment factor.

[0106]

[0107] In some embodiments, the terminal device can obtain the target joint point constraint items based on the statistical results of the joint point position differences corresponding to each target joint point type, and the statistical results of the limb length differences corresponding to each target limb type.

[0108] In some embodiments, step S1306 determines that when the target joint point constraint item satisfies the minimization constraint condition, the numerical value corresponding to the morphology adjustment parameter, and obtains the target position adjustment factor, including: when determining that the target joint point constraint item satisfies the minimization constraint condition, the numerical value corresponding to the morphology adjustment parameter and the numerical value corresponding to the posture adjustment parameter, and obtains the target position adjustment factor.

[0109] Specifically, the terminal device can determine that when the target joint point constraint item satisfies the minimization constraint condition, the value corresponding to the morphology adjustment parameter is recorded as the morphology adjustment factor, and the value corresponding to the posture adjustment parameter is recorded as the posture adjustment factor, and the morphology adjustment factor and the posture adjustment factor are used as the target position adjustment factor. Since there is a positive correlation between the target joint point constraint item and the joint point position difference, when it is determined that the target joint point constraint item satisfies the minimization constraint condition, the value corresponding to the morphology adjustment parameter and the value corresponding to the posture adjustment parameter can obtain the target position adjustment factor that satisfies the joint point position difference and the limb length difference as a whole, so that the morphology adjustment factor that makes the limb length of the fitting three-dimensional object model the same as the limb length of the target three-dimensional object model can be obtained, and the posture adjustment factor that makes the positions of the same type of joints between the fitting three-dimensional object model and the target three-dimensional object model as consistent as possible can be obtained, that is, the target position adjustment factor that can make the posture and morphology of the fitting three-dimensional object model consistent with the target three-dimensional object model can be obtained.

[0110] In some embodiments, the terminal device can perform morphological adjustment on the standard three-dimensional object model according to the obtained morphological adjustment factor, and then perform posture adjustment on the three-dimensional object model after the morphological adjustment by the posture adjustment factor, to obtain the three-dimensional object model after the target position adjustment factor is adjusted (referred to as the initial three-dimensional object model), and obtain the fitting three-dimensional object model corresponding to the target three-dimensional object model according to the initial three-dimensional object model. Specifically, the terminal device can set the posture adjustment parameter in the third joint point adjustment item to the posture adjustment factor, set the morphological adjustment parameter therein to the morphological adjustment factor, obtain the result corresponding to the third joint point adjustment item, adjust the corresponding joint point in the standard three-dimensional object model to the position represented by the result corresponding to the third joint point adjustment item, and obtain the fitting three-dimensional object model.

[0111] like Fig.14 As shown, Fig.14 (a) is the effect diagram of morphological adjustment. Fig.14 (b) is a diagram showing the effect of posture adjustment. The terminal device can adjust the vertex positions of the initial three-dimensional object model according to the vertex positions of the target three-dimensional object model to obtain a non-rigid registration three-dimensional object model as the fitting three-dimensional object model corresponding to the target three-dimensional object model. Fig.15 As shown, Fig.15 (a) is a schematic diagram of the initial three-dimensional object model. Fig.15 (b) is a schematic diagram of a non-rigid registration 3D object model. Fig.16 As shown, a schematic diagram of the process of non-rigid registration is shown.

[0112] In the above embodiment, when it is determined that the target joint point constraint item satisfies the minimization constraint condition, the numerical value corresponding to the morphology adjustment parameter and the numerical value corresponding to the posture adjustment parameter can obtain the target position adjustment factor that minimizes both the joint point position difference and the limb length difference as a whole, that is, the target position adjustment factor that keeps the posture and morphology of the fitted three-dimensional object model consistent with the target three-dimensional object model can be obtained.

[0113] It should be understood that, although the steps in the flowcharts of the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts of the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] In some embodiments, Fig.17 As shown, a human body parameterized three-dimensional model deformation device based on neural network joint point estimation is provided, comprising: a target two-dimensional object image set obtaining module 1702, a target two-dimensional joint point set obtaining module 1704, a target three-dimensional joint point obtaining module 1706 and a fitting three-dimensional object model obtaining module 1708, wherein:

[0115] The target two-dimensional object image set obtaining module 1702 is used to obtain a target three-dimensional object model, perform two-dimensional image extraction on the target three-dimensional object model, and obtain a target two-dimensional object image set. The target three-dimensional object model is obtained by scanning the target object.

[0116] The target two-dimensional joint point set obtaining module 1704 is used to obtain a target two-dimensional joint point detection model, and use the target two-dimensional joint point detection model to perform two-dimensional joint point detection on the target two-dimensional object images in the target two-dimensional object image set to obtain the target two-dimensional joint point sets corresponding to each target two-dimensional object image.

[0117] The target 3D joint point obtaining module 1706 is used to obtain the target 3D joint point corresponding to the target 2D joint point according to the spatial mapping relationship corresponding to each target 2D joint point in the target 2D joint point set corresponding to the target 2D object image.

[0118] The fitting 3D object model obtaining module 1708 is used to perform object fitting using the target 3D joint points corresponding to each target 2D joint point, and obtain a fitting 3D object model corresponding to the target 3D object model, so as to control the movement of the fitting 3D object model.

[0119] In some embodiments, the target two-dimensional joint point set obtaining module 1704 includes:

[0120] The initial two-dimensional joint point detection model acquisition unit is used to acquire an initial two-dimensional joint point detection model and train a two-dimensional object image.

[0121] The image training label determination unit is used to determine the image training label corresponding to the training two-dimensional object image, and the image training label is determined according to the labeled two-dimensional joint points in the labeled two-dimensional joint point set corresponding to the training two-dimensional object image.

[0122] The target two-dimensional joint point detection model obtaining unit is used to train the initial two-dimensional joint point detection model according to the training two-dimensional object image and the image training label corresponding to the training two-dimensional object image to obtain the target two-dimensional joint point detection model.

[0123] In some embodiments, the image training label includes a confidence map corresponding to each labeled two-dimensional joint point in the labeled two-dimensional joint point set; the device also includes:

[0124] The target feature position acquisition module is used to determine the target feature position of the target feature point corresponding to the marked two-dimensional joint point according to the feature information corresponding to the marked two-dimensional joint point.

[0125] The confidence map determination module is used to obtain the associated feature position of the associated feature point corresponding to the target feature point, and determine the confidence map corresponding to the annotated two-dimensional joint point based on the difference between the target feature position and the associated feature position.

[0126] In some embodiments, the image training label includes a limb vector image corresponding to the target limb, and the joint point corresponding to the target limb is a labeled two-dimensional joint point in the labeled two-dimensional joint point set; the device also includes:

[0127] The target limb vector acquisition module is used to acquire the target limb vector corresponding to the target limb, and the target limb vector is determined according to the position of the joint point corresponding to the target limb.

[0128] The target point vector determination module is used to determine the target point vector corresponding to the target limb point on the target limb according to the target limb vector.

[0129] The limb vector map determination module is used to obtain the limb vector map corresponding to the target limb according to the target point vectors corresponding to each target limb point.

[0130] In some embodiments, the fitting three-dimensional object model obtaining module 1708 includes:

[0131] The standard three-dimensional joint point set acquisition unit is used to acquire a standard three-dimensional object model and a standard three-dimensional joint point set corresponding to the standard three-dimensional object model; the standard three-dimensional joint point set includes multiple standard three-dimensional joint points.

[0132] A target joint point constraint item acquisition unit is used to acquire target joint point constraint items, wherein the target joint point constraint items include the difference between the limb length adjustment item and the target limb length, wherein the limb length adjustment item is the difference result between the first joint point adjustment item and the second joint point adjustment item; the first joint point adjustment item is the result of the operation between the position of the first standard three-dimensional joint point and the morphological adjustment parameter, and the second joint point adjustment item is the result of the operation between the position of the second standard three-dimensional joint point and the morphological adjustment parameter; the first standard three-dimensional joint point and the second standard three-dimensional joint point correspond to the target limb type, and the target limb length is the position difference between the first target three-dimensional joint point corresponding to the target limb type and the corresponding second target three-dimensional joint point.

[0133] The target position adjustment factor obtaining unit is used to determine the value corresponding to the morphological adjustment parameter when the target joint point constraint item satisfies the minimization constraint condition, and obtain the target position adjustment factor.

[0134] The fitting three-dimensional object model obtaining unit is used to adjust the positions of the joint points of the standard three-dimensional object model by using the target position adjustment factor to obtain the fitting three-dimensional object model.

[0135] In some embodiments, the target joint point constraint item also includes a third joint point adjustment item corresponding to the first standard three-dimensional joint point, and the difference between the position of the first target three-dimensional joint point, the third joint point adjustment item is the result of calculation of the posture adjustment parameter and the first joint point adjustment item; the joint point type of the first target three-dimensional joint point is consistent with that of the first standard three-dimensional joint point.

[0136] In some embodiments, the target position adjustment factor obtaining unit is also used to determine the values ​​corresponding to the morphology adjustment parameters and the posture adjustment parameters when the target joint point constraint item satisfies the minimization constraint condition, so as to obtain the target position adjustment factor.

[0137] For the specific definition of the human body parametric three-dimensional model deformation device based on neural network joint point estimation, please refer to the definition of the human body parametric three-dimensional model deformation method based on neural network joint point estimation in the above text, which will not be repeated here. Each module in the above-mentioned human body parametric three-dimensional model deformation device based on neural network joint point estimation can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0138] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Fig.18As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a human body parameterized three-dimensional model deformation method based on neural network joint point estimation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0139] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above-mentioned method for deforming a parametric three-dimensional model of a human body based on neural network joint point estimation are implemented.

[0141] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned human body parametric three-dimensional model deformation method based on neural network joint point estimation are implemented.

[0142] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for deforming a parametric three-dimensional human body model based on neural network joint point estimation, characterized in that: The method comprises: Acquire a target three-dimensional object model, perform two-dimensional image extraction on the target three-dimensional object model, and obtain a target two-dimensional object image set, wherein the target three-dimensional object model is obtained by scanning the target object; Acquire a target two-dimensional joint point detection model, and use the target two-dimensional joint point detection model to perform two-dimensional joint point detection on the target two-dimensional object images in the target two-dimensional object image set to obtain a target two-dimensional joint point set corresponding to each of the target two-dimensional object images; According to the spatial mapping relationship corresponding to each target two-dimensional joint point in the target two-dimensional joint point set corresponding to the target two-dimensional object image, the target three-dimensional joint point corresponding to the target two-dimensional joint point is obtained; Acquire a standard three-dimensional object model and a standard three-dimensional joint point set corresponding to the standard three-dimensional object model; the standard three-dimensional joint point set includes a plurality of standard three-dimensional joint points; Obtain a target joint point constraint item, wherein the target joint point constraint item includes a difference between a limb length adjustment item and a target limb length, wherein the limb length adjustment item is a difference result between a first joint point adjustment item and a second joint point adjustment item; the first joint point adjustment item is a result of an operation between a position of a first standard three-dimensional joint point and a morphological adjustment parameter, and the second joint point adjustment item is a result of an operation between a position of a second standard three-dimensional joint point and the morphological adjustment parameter; the first standard three-dimensional joint point and the second standard three-dimensional joint point correspond to a target limb type, and the target limb length is a position difference between a first target three-dimensional joint point corresponding to the target limb type and a corresponding second target three-dimensional joint point; the target joint point constraint item also includes a third joint point adjustment item corresponding to the first standard three-dimensional joint point, and a difference between the position of the first target three-dimensional joint point, wherein the third joint point adjustment item is a result of an operation on a posture adjustment parameter and the first joint point adjustment item; the first target three-dimensional joint point is consistent with the first standard three-dimensional joint point in joint point type; When it is determined that the target joint point constraint item satisfies the minimization constraint condition, the value corresponding to the morphology adjustment parameter and the value corresponding to the posture adjustment parameter are used to obtain the target position adjustment factor; The target position adjustment factor is used to adjust the positions of the joint points of the standard three-dimensional object model to obtain a fitted three-dimensional object model.

2. The method according to claim 1, characterized in that The acquiring of the target two-dimensional joint point detection model comprises: Obtain an initial two-dimensional joint point detection model and a training two-dimensional object image; Determine an image training label corresponding to the training two-dimensional object image, wherein the image training label is determined according to annotated two-dimensional joint points in a set of annotated two-dimensional joint points corresponding to the training two-dimensional object image; The initial two-dimensional joint point detection model is trained according to the training two-dimensional object image and the image training label corresponding to the training two-dimensional object image to obtain the target two-dimensional joint point detection model.

3. The method according to claim 2, characterized in that The image training label includes a confidence map corresponding to each labeled two-dimensional joint point in the labeled two-dimensional joint point set; the method further includes: Determining a target feature position of a target feature point corresponding to the marked two-dimensional joint point according to feature information corresponding to the marked two-dimensional joint point; The associated feature position of the associated feature point corresponding to the target feature point is obtained, and the confidence map corresponding to the annotated two-dimensional joint point is determined according to the difference between the target feature position and the associated feature position.

4. The method according to any one of claims 2 to 3, characterized in that The image training label includes a limb vector image corresponding to the target limb, and the joint points corresponding to the target limb are the annotated two-dimensional joint points in the annotated two-dimensional joint point set; The method further comprises: Obtaining a target limb vector corresponding to the target limb, wherein the target limb vector is determined according to a position of a joint point corresponding to the target limb; Determine a target point vector corresponding to a target limb point on the target limb according to the target limb vector; According to the target point vectors corresponding to each of the target limb points, a limb vector map corresponding to the target limb is obtained.

5. A human body parametric three-dimensional model deformation device based on neural network joint point estimation, characterized in that: The device comprises: A target two-dimensional object image set obtaining module is used to obtain a target three-dimensional object model, perform two-dimensional image extraction on the target three-dimensional object model, and obtain a target two-dimensional object image set, wherein the target three-dimensional object model is obtained by scanning the target object; a target two-dimensional joint point set obtaining module, used to obtain a target two-dimensional joint point detection model, and use the target two-dimensional joint point detection model to perform two-dimensional joint point detection on the target two-dimensional object images in the target two-dimensional object image set to obtain target two-dimensional joint point sets corresponding to each of the target two-dimensional object images; A target three-dimensional joint point obtaining module is used to obtain a target three-dimensional joint point corresponding to the target two-dimensional joint point according to a spatial mapping relationship corresponding to each target two-dimensional joint point in a target two-dimensional joint point set corresponding to the target two-dimensional object image; Fitting the three-dimensional object model to obtain the module includes: A standard three-dimensional joint point set acquisition unit, used to acquire a standard three-dimensional object model and a standard three-dimensional joint point set corresponding to the standard three-dimensional object model; the standard three-dimensional joint point set includes a plurality of standard three-dimensional joint points; A target joint point constraint item acquisition unit is used to acquire a target joint point constraint item, wherein the target joint point constraint item includes a difference between a limb length adjustment item and a target limb length, wherein the limb length adjustment item is a difference result between a first joint point adjustment item and a second joint point adjustment item; the first joint point adjustment item is a result of an operation between a position of a first standard three-dimensional joint point and a morphological adjustment parameter, and the second joint point adjustment item is a result of an operation between a position of a second standard three-dimensional joint point and the morphological adjustment parameter; the first standard three-dimensional joint point and the second standard three-dimensional joint point correspond to a target limb type, and the target limb length is a position difference between a first target three-dimensional joint point corresponding to the target limb type and a corresponding second target three-dimensional joint point; the target joint point constraint item also includes a third joint point adjustment item corresponding to the first standard three-dimensional joint point, and a difference between the position of the first target three-dimensional joint point, wherein the third joint point adjustment item is a result of an operation between a posture adjustment parameter and the first joint point adjustment item; the first target three-dimensional joint point is consistent with the first standard three-dimensional joint point in joint point type; a target position adjustment factor obtaining unit, used to determine the value corresponding to the morphology adjustment parameter and the value corresponding to the posture adjustment parameter when the target joint point constraint item satisfies the minimization constraint condition, and obtain the target position adjustment factor; The fitting three-dimensional object model obtaining unit is used to adjust the positions of the joint points of the standard three-dimensional object model using the target position adjustment factor to obtain the fitting three-dimensional object model.

6. The device according to claim 5, characterized in that The target two-dimensional joint point set obtaining module includes: An initial two-dimensional joint point detection model acquisition unit, used to acquire an initial two-dimensional joint point detection model and train a two-dimensional object image; An image training label determination unit, used to determine an image training label corresponding to the training two-dimensional object image, wherein the image training label is determined according to annotated two-dimensional joint points in a set of annotated two-dimensional joint points corresponding to the training two-dimensional object image; The target two-dimensional joint point detection model obtaining unit is used to train the initial two-dimensional joint point detection model according to the training two-dimensional object image and the image training label corresponding to the training two-dimensional object image to obtain the target two-dimensional joint point detection model.

7. The device according to claim 6, characterized in that The image training label includes confidence maps corresponding to each labeled two-dimensional joint point in the labeled two-dimensional joint point set; the device also includes: A target feature position acquisition module, used to determine the target feature position of the target feature point corresponding to the marked two-dimensional joint point according to the feature information corresponding to the marked two-dimensional joint point; The confidence map determination module is used to obtain the associated feature position of the associated feature point corresponding to the target feature point, and determine the confidence map corresponding to the annotated two-dimensional joint point according to the difference between the target feature position and the associated feature position.

8. The device according to claim 6 or 7, characterized in that The image training label includes a limb vector image corresponding to the target limb, and the joint points corresponding to the target limb are the annotated two-dimensional joint points in the annotated two-dimensional joint point set; the device also includes: A target limb vector acquisition module is used to acquire a target limb vector corresponding to the target limb, wherein the target limb vector is determined according to the position of the joint point corresponding to the target limb; A target point vector determination module, used to determine a target point vector corresponding to a target limb point on the target limb according to the target limb vector; The limb vector map determining module is used to obtain the limb vector map corresponding to the target limb according to the target point vectors corresponding to each of the target limb points.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Human posture recognition device and method based on full convolutional neural network

    CN108898063A

  • Motion control method, device and apparatus for virtual three-dimensional model

    CN109191593A

  • A global three-dimensional human posture credible estimation method based on two views

    CN109377513A

  • A human body three-dimensional posture estimation method based on image retrieval

    CN109949368A

  • Optical unmarked three-dimensional human body motion capture method

    CN110633005A