Three-dimensional human body reconstruction method, device, electronic equipment and storage medium
In the three-dimensional human body reconstruction technology, the parameters optimization and adjustment of the two-dimensional human body image and reference human model are used, and the coordinate conversion of face key points and grid vertices are combined, the poor effect and low efficiency of face reconstruction in the existing technology are solved. The problem of this has achieved more efficient three-dimensional human body reconstruction.
Patent Information
- Application Number
- CN202311814296.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-26
AI Technical Summary
The existing three-dimensional human body reconstruction technology has shortcomings in the face reconstruction effect, and it is necessary to combine multiple models, resulting in complex data processing and affecting efficiency.
By obtaining the first model parameters based on the two-dimensional human image and reference human model, and using the coordinate conversion of face key points and grid vertices, optimize and adjust model parameters to improve the face reconstruction effect without combining multiple models, simplifying Data processing.
The effect of face reconstruction is improved, while reducing the overall efficiency effect of three-dimensional human body reconstruction, avoiding the negative impact of multi-model combination scheme on efficiency.
Smart Images

Figure CN117953150B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of three-dimensional model processing, and specifically to a three-dimensional human body reconstruction method, device, electronic device and storage medium. Background Art
[0002] At present, the application of three-dimensional human body reconstruction technology is becoming more and more widespread. Three-dimensional human body reconstruction can be performed based on a parametric human body model. However, the reconstructed three-dimensional human body model can only achieve a good reconstruction effect on body posture, but cannot achieve a good reconstruction effect on the face.
[0003] In the related art, in order to obtain a better reconstruction effect on the face, a parametric human body model is usually combined with a special head model to perform 3D human body reconstruction. The problem with the related art is that it needs to combine multiple models, the processed data is more complicated, and the efficiency of 3D human body reconstruction is affected. Summary of the invention
[0004] The embodiments of the present application provide a three-dimensional human body reconstruction method, device, electronic device and storage medium. When performing three-dimensional human body reconstruction, firstly obtain first model parameters and a first human body model according to a two-dimensional human body image and a reference human body model, and optimize and adjust the first model parameters according to the coordinates of facial key points in the two-dimensional human body image and the coordinates of target mesh vertices in the first human body model to optimize the facial reconstruction effect without combining multiple models, which is beneficial to improving the facial reconstruction effect and reducing the impact on the efficiency of three-dimensional human body reconstruction.
[0005] A first aspect of an embodiment of the present application provides a three-dimensional human body reconstruction method, the method comprising:
[0006] Reconstructing a three-dimensional model according to the two-dimensional human body image of the target object and a preset reference human body model to obtain first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image;
[0007] Adjusting the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model that matches the target object;
[0008] Obtaining the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image;
[0009] Determine the target mesh vertex corresponding to the facial key point in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertex;
[0010] According to the coordinate conversion information, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted into the same coordinate system, and the target distances between the corresponding facial key points and the target mesh vertices in the same coordinate system are obtained;
[0011] The first model parameters are adjusted according to the target distance to obtain second model parameters, and the positions of the mesh vertices of the first human body model are adjusted according to the second model parameters to obtain the second human body model.
[0012] A second aspect of an embodiment of the present application provides a three-dimensional human body reconstruction device, the device comprising:
[0013] A first model parameter acquisition module is used to reconstruct a three-dimensional model according to the two-dimensional human body image of the target object and a preset reference human body model, and obtain first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image;
[0014] A first model reconstruction module, used for adjusting the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model matching the target object;
[0015] A facial key point detection module, used to obtain the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human image;
[0016] A target mesh vertex matching module, used to determine the target mesh vertices corresponding to the facial key points in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertices;
[0017] A target distance calculation module, used to convert the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices into the same coordinate system according to the coordinate conversion information, and obtain the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system;
[0018] The second model reconstruction module is used to adjust the first model parameters according to the target distance to obtain second model parameters, and adjust the positions of the mesh vertices of the first human body model according to the second model parameters to obtain a second human body model.
[0019] In some optional embodiments, the first model parameter acquisition module is specifically used to:
[0020] Performing three-dimensional human body reconstruction on the two-dimensional human body image to obtain third model parameters matching the target object, wherein the third model parameters include posture information and shape information of the target object;
[0021] Acquire adjusted shape information according to the shape information in the third model parameter, the preset shape adjustment parameters and the model parameters of the reference human body model;
[0022] Constructing the first model parameters according to the posture information in the third model parameters and the adjusted shape information;
[0023] The coordinate transformation information between the two-dimensional human body image and the reference human body model is obtained.
[0024] In some optional embodiments, the first model parameter acquisition module is further specifically used for:
[0025] Performing human body detection on the two-dimensional human body image to obtain a human body detection frame of the target object;
[0026] A three-dimensional human body is reconstructed according to the image content within the human body detection frame in the two-dimensional human body image to obtain third model parameters matching the target object.
[0027] In some optional embodiments, the facial key point detection module is specifically used for:
[0028] Performing face detection on the two-dimensional human body image to obtain a face detection frame of the target object;
[0029] Performing facial key point recognition on the image content within the facial detection frame in the two-dimensional human image to obtain facial key points of the target object in the two-dimensional human image;
[0030] The two-dimensional coordinates of the facial key points in the two-dimensional coordinate system where the two-dimensional human body image is located are obtained.
[0031] In some optional embodiments, the target mesh vertex matching module is specifically used for:
[0032] According to preset key point index value association data, obtaining vertex index values of mesh vertices associated with the facial key points in the reference human body model;
[0033] Determining, according to the first human body model and the vertex index values, target mesh vertices corresponding to the facial key points in the first human body model;
[0034] The three-dimensional coordinates of the target mesh vertices in the three-dimensional coordinate system where the first human body model is located are obtained.
[0035] In some optional embodiments, the target distance calculation module is specifically used for:
[0036] According to the coordinate conversion information, the three-dimensional coordinates of the target mesh vertices are converted into the two-dimensional coordinate system where the two-dimensional human body image is located, to obtain the target two-dimensional coordinates of the target mesh vertices, wherein the first human body model and the reference human body model are in the same three-dimensional coordinate system;
[0037] According to the two-dimensional coordinates of the facial key points and the target two-dimensional coordinates of the target mesh vertices corresponding to the facial key points, the target distance between the corresponding facial key points and the target mesh vertices is obtained.
[0038] In some optional embodiments, the first model parameter includes posture information and shape information of a plurality of joint point mesh vertices, and the joint point mesh vertices are mesh vertices corresponding to the joint points of the target object;
[0039] The second model reconstruction module is specifically used to adjust the posture information corresponding to the head joint mesh vertices and the neck joint mesh vertices in the first model parameters according to the target distance until the second model parameters that meet the preset iteration termination conditions are obtained.
[0040] A third aspect of an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in the three-dimensional human body reconstruction method provided in the first aspect of the embodiment of the present application.
[0041] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores multiple instructions. The instructions are suitable for a processor to load to execute the steps in the three-dimensional human body reconstruction method provided in the first aspect of the embodiments of the present application.
[0042] By adopting the scheme of the embodiment of the present application, a three-dimensional model can be reconstructed based on the two-dimensional human body image of the target object and the preset reference human body model, and first model parameters and coordinate conversion information matching the target object can be obtained, wherein the coordinate conversion information is used to characterize the conversion relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image; the positions of the mesh vertices of the reference human body model are adjusted according to the first model parameters to obtain the first human body model matching the target object; the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image are obtained; the target mesh vertices corresponding to the facial key points in the first human body model are determined, and the three-dimensional coordinates of the target mesh vertices are obtained; according to the coordinate conversion information, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted to the same coordinate system to obtain the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system; the first model parameters are adjusted according to the target distance to obtain the second model parameters, and the positions of the mesh vertices of the first human body model are adjusted according to the second model parameters to obtain the second human body model.
[0043] In this way, when performing three-dimensional human reconstruction, first obtain the first model parameters and the first human model based on the two-dimensional human image and the reference human model, and then perform coordinate transformation on the two-dimensional coordinates of the key points of the face in the two-dimensional human image and / or the three-dimensional coordinates of the target mesh vertices in the first human model, and transform them into the same coordinate system to determine the corresponding target distance, thereby optimizing and adjusting the first model parameters according to the target distance to achieve optimization of the facial reconstruction effect, without combining multiple models, avoiding the significant impact of the three-dimensional human reconstruction scheme combining multiple models on the three-dimensional human reconstruction efficiency, which is conducive to improving the facial reconstruction effect and reducing the impact on the three-dimensional human reconstruction efficiency. Compared with the scheme of combining multiple models for three-dimensional human reconstruction in the related art, the embodiment of the present application can improve the efficiency of three-dimensional human reconstruction on the basis of improving the facial reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0045] Figure 1 This is a schematic diagram of the timing of data interaction when a server performs three-dimensional human body reconstruction provided by an embodiment of the present application;
[0046] Figure 2 is a flow chart of a three-dimensional human body reconstruction method provided in an embodiment of the present application;
[0047] Figure 3 is a schematic diagram of the position of a target mesh vertex provided in an embodiment of the present application;
[0048] Figure 4 It is a specific flow chart of a three-dimensional human body reconstruction method provided in an embodiment of the present application;
[0049] Figure 5 is a structural block diagram of a three-dimensional human body reconstruction device provided in an embodiment of the present application;
[0050] Figure 6 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0052] The embodiments of the present application provide a three-dimensional human body reconstruction method, device, electronic device and storage medium. Specifically, the three-dimensional human body reconstruction method of the embodiments of the present application can be executed by a computer device, wherein the computer device can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a personal computer (PC, Personal Computer), a personal digital assistant (PDA, Personal Digital Assistant), etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0053] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0054] Figure 1 is a schematic diagram of the data interaction timing when a server performs three-dimensional human body reconstruction according to an embodiment of the present application. In the embodiment of the present application, the client and the server communicate according to Figure 1 Steps S1 to S7 shown perform data exchange.
[0055] like Figure 1As shown, the user can upload a two-dimensional human image containing the target object to the server through the client. After the server obtains the two-dimensional human image, it reconstructs the three-dimensional model according to the image and the reference human model pre-set in the server to obtain the first model parameter and coordinate conversion information. Further, the server adjusts the position of the mesh vertex of the reference human model according to the first model parameter to obtain the first human model matching the target object. Then, the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human image are obtained, and the target mesh vertices and their three-dimensional coordinates corresponding to the facial key points in the first human model are determined. According to the coordinate conversion information, the two-dimensional coordinates and the three-dimensional coordinates are converted to the same coordinate system, so as to obtain the target distance between the corresponding facial key points and the target mesh vertex. Further, the first model parameter is adjusted according to the target distance to obtain the second parameter, so as to optimize the facial reconstruction effect and obtain the second human model matching the target object with better facial reconstruction effect.
[0056] It should be noted that Figure 1 The three-dimensional human body reconstruction by the server is taken as an example for specific description. In actual use, the three-dimensional human body reconstruction can also be directly performed by the client used by the user, which is not specifically limited here.
[0057] The first aspect of the present application embodiment provides a three-dimensional human body reconstruction method. Figure 2 , Figure 2 : is a flow chart of a 3D human body reconstruction method provided in an embodiment of the present application. The specific flow of the 3D human body reconstruction method can be as follows:
[0058] 201. Reconstruct a three-dimensional model based on a two-dimensional human body image of a target object and a preset reference human body model to obtain first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize a transformation relationship between a three-dimensional coordinate system of the reference human body model and a two-dimensional coordinate system of the two-dimensional human body image.
[0059] The above-mentioned two-dimensional human body image is an image of a human body subject of a target object, and the two-dimensional human body image can be input or selected by a user, which is not specifically limited here. The above-mentioned preset reference human body model is a pre-set parameterized human body model, for example, a preset SMPL (skinned multi-person linear model) model can be used, which is not specifically limited here. The above-mentioned first model parameter is used to indicate the posture characteristics and shape characteristics of the parameterized human body model.
[0060] In some embodiments of the present application, the three-dimensional model is reconstructed based on the two-dimensional human body image of the target object and the preset reference human body model to obtain the first model parameters and coordinate conversion information matching the target object, including:
[0061] Performing three-dimensional human body reconstruction on the two-dimensional human body image to obtain third model parameters matching the target object, wherein the third model parameters include posture information and shape information of the target object;
[0062] Acquire adjusted shape information according to the shape information in the third model parameter, the preset shape adjustment parameters and the model parameters of the reference human body model;
[0063] Constructing the first model parameters according to the posture information in the third model parameters and the adjusted shape information;
[0064] The coordinate transformation information between the two-dimensional human body image and the reference human body model is obtained.
[0065] It should be noted that, in the embodiment of the present application, for the parameterized three-dimensional human body model (including the reference human body model, the first human body model and the second human body model), the corresponding model parameters include posture information and shape information, the above-mentioned posture information is used to indicate the posture characteristics, and the above-mentioned shape information is used to indicate the shape characteristics.
[0066] The above-mentioned preset shape adjustment parameters are control parameters pre-set to adjust the shape information in the calculated third model parameters. It should be noted that in an application scenario, in order to obtain a better shape control effect, the shape information in the third model parameters may not be adjusted, and the third model parameters may be directly used as the first model parameters, which is not specifically limited here.
[0067] In some embodiments of the present application, the three-dimensional human body reconstruction is performed on the two-dimensional human body image to obtain a third model parameter matching the target object, including:
[0068] Performing human body detection on the two-dimensional human body image to obtain a human body detection frame of the target object;
[0069] A three-dimensional human body is reconstructed according to the image content within the human body detection frame in the two-dimensional human body image to obtain third model parameters matching the target object.
[0070] In this way, when performing three-dimensional human body reconstruction, the human body detection frame in the two-dimensional human body image can be determined first. The above-mentioned human body detection frame is used to indicate the position of the human body in the image, so that when performing three-dimensional human body reconstruction, only the image content within the human body detection frame can be considered, thereby reducing the required calculation amount and improving data processing efficiency.
[0071] In a specific application scenario, a two-dimensional human image (such as image A) is obtained, and a human body detection algorithm is used to perform human body detection on the two-dimensional human body image to obtain a human body detection frame bbox_body = [X1, Y1, X2, Y2] corresponding to the main body of the person in the image. The above-mentioned human body detection algorithm can be set and adjusted according to actual needs. For example, the YOLOV7 algorithm or the DETR algorithm can be used, and other algorithms can also be used, which are not specifically limited here. The bbox_body includes the two-dimensional coordinates of the upper left corner vertex and the two-dimensional coordinates of the lower right corner vertex of the human body detection frame, which are used to indicate the position of the human body detection frame.
[0072] The above-mentioned image A and the human body detection box bbox_body are used as inputs of the three-dimensional human body reconstruction algorithm to obtain the third model parameters (including the posture information pose and shape information shape of the human body in the human body detection box), as well as the camera information camera, and the above-mentioned camera information is used as the coordinate conversion information. Among them, the above-mentioned three-dimensional human body reconstruction algorithm is a pre-set three-dimensional human body reconstruction algorithm based on the SMPL human body model, which can be set and adjusted according to actual needs. For example, the HMR algorithm or the CLIFF algorithm can be used, and other algorithms can also be used, which are not specifically limited here. In this embodiment, the HMR algorithm is used, and the corresponding posture information pose is a 216-dimensional vector (216=24x3x3), which is used to represent the rotation matrix information of 24 human body bone points, the shape information shape is a 10-dimensional vector, and the camera information camera is a 3-dimensional vector.
[0073] In some embodiments of the present application, the third model parameters may include posture information corresponding to a large number of joints of the target object. In order to reduce the amount of calculation during subsequent data processing, only the posture information corresponding to some of the nodes (i.e., the joints corresponding to the human body) is selected. For example, according to the index values of the joints such as "pelvis", "spine1" (first spine point), "spine2" (second spine point), "spine3" (third spine point), "neck", and "head", the posture information of the corresponding joints is obtained, that is, 6 rotation matrices R1, R2, R3, R4, R5, and R6, to construct the first model parameters based on the posture information of these 6 joints and the adjusted shape information. It should be noted that in actual use, the posture information of other joints can also be used, which is not specifically limited here.
[0074] Further, the shape information in the third model parameter is adjusted to obtain the corresponding first model parameter. Specifically, the preset shape adjustment parameter blend shape parameter (specific dimension is 6890x3x10) for shape information control is obtained, the SMPL model average mesh point shape information MeanMesh (specific dimension is 6890x3) pre-set in the reference human body model is obtained, and the shape information Mesh (dimension is 6890x3) of the adjusted mesh vertex is calculated according to the following formula (1):
[0075] Mesh = Mean Mesh + blend shape * shape (1)
[0076] In this way, the first model parameters are constructed according to the adjusted shape information and the posture information in the third model parameters, so that a better shape control effect can be obtained.
[0077] 202. Adjust positions of mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model matching the target object.
[0078] Specifically, the mesh vertices of the reference human body model are adjusted according to the first model parameters, so as to obtain a first human body model obtained by three-dimensionally reconstructing the target object in the two-dimensional human body image. It should be noted that the posture and shape of the first human body model match the main body of the target object in the two-dimensional human body image, but the matching effect of the face is poor. Therefore, in the embodiment of the present application, the first human body model is further optimized to obtain a second human body model with better facial reconstruction effect.
[0079] 203. Obtain the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image.
[0080] The facial key points are key points obtained by performing key point recognition on the facial area of the target object, and can be used to indicate the positions of the facial features of the target object.
[0081] In some embodiments of the present application, the step of obtaining the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human image includes:
[0082] Performing face detection on the two-dimensional human body image to obtain a face detection frame of the target object;
[0083] Performing facial key point recognition on the image content within the facial detection frame in the two-dimensional human image to obtain facial key points of the target object in the two-dimensional human image;
[0084] The two-dimensional coordinates of the facial key points in the two-dimensional coordinate system where the two-dimensional human body image is located are obtained.
[0085] In this way, the face detection frame in the two-dimensional human image is first identified, and facial key point recognition can be performed only on the image content in the face detection frame, which can reduce the amount of data processing and improve the efficiency and accuracy of key point recognition.
[0086] In one application scenario, the above image A is used as input data of a face detection algorithm, and face detection is performed to obtain the face detection box bbox_face = [X3, Y3, X4, Y4] of the target object in image A, corresponding to the coordinates of the upper left corner point and the coordinates of the lower right corner point of the face detection box. The above face detection algorithm can be set and adjusted according to actual needs. For example, the RetinaFace algorithm or the Pyramid box algorithm can be used, and other algorithms can also be used, which are not specifically limited here.
[0087] The above image A and the face detection box bbox_face are used as input data of the face key point detection algorithm to perform face key point recognition to obtain the face key points and their two-dimensional coordinates of the target object. The above face key point detection algorithm can use the SDUNets algorithm or the 3DDFA algorithm, and other algorithms can also be used according to actual needs, which are not specifically limited here.
[0088] It should be noted that multiple key points of the face can be obtained through the face key point detection algorithm. In order to reduce the amount of data that needs to be calculated subsequently and improve the efficiency of three-dimensional human body reconstruction, in the embodiment of the present application, some key points that can indicate the positions of the facial features are selected from the detected key points of the face as target key points.
[0089] In one application scenario, the left and right eye corners of the left eye, the left and right eye corners of the right eye, the nose point, the left and right mouth corners and the chin point are selected as the above target key points, but this is not a specific limitation. The two-dimensional coordinates corresponding to the above target key points are P_face = [[x1, y1],
[0090] [x2,y2],…,[x8,y8]].
[0091] 204. Determine the target mesh vertex corresponding to the facial key point in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertex.
[0092] Specifically, for the preset reference human body model, each mesh vertex is set with an index value and the meaning of the mesh vertex. For example, the mesh vertex with an index value of a represents the nose point, that is, there is a corresponding relationship between the vertex and the nose point. In the process of adjusting the position of the mesh vertices of the reference human body model, the index values and key point meanings corresponding to the mesh vertices do not change, and the three-dimensional coordinate system corresponding to the three-dimensional human body model does not change. Therefore, in the embodiment of the present application, the three-dimensional coordinates of the target mesh vertex can be determined according to the index value corresponding to the target mesh vertex in the first human body model.
[0093] In some embodiments of the present application, the step of determining the target mesh vertex corresponding to the facial key point in the first human body model and obtaining the three-dimensional coordinates of the target mesh vertex includes:
[0094] According to preset key point index value association data, obtaining vertex index values of mesh vertices associated with the facial key points in the reference human body model;
[0095] Determining, according to the first human body model and the vertex index values, target mesh vertices corresponding to the facial key points in the first human body model;
[0096] The three-dimensional coordinates of the target mesh vertices in the three-dimensional coordinate system where the first human body model is located are obtained.
[0097] The preset key point index value associated data is data corresponding to the reference human body model and used to store mesh vertex index values and meanings.
[0098] It should be noted that, in the embodiment of the present application, for each selected facial key point, the index value of the mesh vertex matching it is determined, so that the mesh vertex with the same index value as the first human body model value is used as the target mesh vertex corresponding to the facial key point.
[0099] Figure 3 is a schematic diagram of the position of a target mesh vertex provided in an embodiment of the present application, such as Figure 3 As shown, in an embodiment of the present application, a three-dimensional geometry processing system (such as MeshLab) is used to obtain the index numbers V_Index=[Index1, Index2, Index3, Index4, Index5, Index6, Index7, Index8] corresponding to the left and right eye corners of the left eye, the left and right eye corners of the right eye, the nose point, the left and right mouth corners, and the chin point from the 6890 vertices of the SMPL basic human body model, thereby determining the three-dimensional coordinates of the corresponding target mesh vertex in the first human body model according to the index number.
[0100] 205. According to the coordinate conversion information, convert the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices into the same coordinate system, and obtain the target distances between the corresponding facial key points and the target mesh vertices in the same coordinate system.
[0101] Specifically, the two-dimensional coordinates of the facial key points may be converted into three-dimensional coordinates in a three-dimensional coordinate system, or the three-dimensional coordinates of the target mesh vertices may be converted into two-dimensional coordinates in a two-dimensional coordinate system, which is not specifically limited here.
[0102] In the embodiment of the present application, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted into the same coordinate system according to the coordinate conversion information, and the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system is obtained, including:
[0103] According to the coordinate conversion information, the three-dimensional coordinates of the target mesh vertices are converted into the two-dimensional coordinate system where the two-dimensional human body image is located, to obtain the target two-dimensional coordinates of the target mesh vertices, wherein the first human body model and the reference human body model are in the same three-dimensional coordinate system;
[0104] According to the two-dimensional coordinates of the facial key points and the target two-dimensional coordinates of the target mesh vertices corresponding to the facial key points, the target distance between the corresponding facial key points and the target mesh vertices is obtained.
[0105] In some embodiments of the present application, according to the SMPL linear skinning algorithm, a first human body model is obtained based on the first model parameters and a reference human body model. It should be noted that when the above-mentioned first model parameters are generated only according to the posture information of some joint points selected from the third model parameters, the linear skinning algorithm is improved to reduce the number of joint points required by the linear skinning algorithm.
[0106] For example, in a specific application scenario, the linear skinning algorithm is adjusted to only retain the skin weights corresponding to the joint points "pelvis", "spine1" (first spine point), "spine2" (second spine point), "spine3" (third spine point), "neck", and "head", and delete the skin weights corresponding to other joint points. That is, the skin weights of the linear skinning algorithm are reduced from the original 6890x24 dimension (the original linear skinning algorithm needs to use the information corresponding to 24 joint points) to 6890x6 dimension. At the same time, the joint point transformation only calculates these 6 nodes, reducing the weight matrix calculation amount in the original skinning algorithm from 6890x24x24x16 to 6890x6x6x16, and finally obtaining the first human body model composed of the new human mesh point Mesh'.
[0107] For the facial key points, the specific position of the target mesh vertex corresponding to each facial key point in the first human body model is determined according to the key point index value associated data V_Index, so as to obtain the three-dimensional coordinates of each target mesh vertex P_mesh_3d=[[x13, y13, z13],
[0108] [x23,y23,z23],…[x83,y83,z83]].
[0109] It should be noted that [x13, y13, z13] represents the three-dimensional coordinates corresponding to the first target mesh vertex, and x13 is used to distinguish from the two-dimensional horizontal coordinate x1 of the target key point, but it is not used as a specific limitation.
[0110] Use the coordinate transformation information to transform the three-dimensional coordinates into a two-dimensional coordinate system to obtain the target two-dimensional coordinates of each target mesh vertex in the two-dimensional coordinate system. Specifically, the above coordinate transformation information is the camera information, that is, the Camera Matrix, which is a 3x3 matrix M. In an application scenario, M = [[focal_length,0,center_x],[0,focal_length,center_y],[0,0,1]], where focal_length represents the pixel focal length in the camera perspective transformation, center_x represents the coordinate of the image center in the horizontal direction, and center_y represents the coordinate of the image center in the vertical direction.
[0111] Specifically, the coordinate transformation can be performed according to the following formula (2) to obtain the target two-dimensional coordinate P_mesh_2d:
[0112]
[0113] Among them, P_mesh_3d′ ij Represents the value of the i-th row and j-th column in the P_mesh_3d′ matrix; P_mesh_3d ij Represents the value of the i-th row and j-th column in the P_mesh_3d matrix; P_mesh_3d i3 Represents the value of the 3rd column of the i-th row in the P_mesh_3d matrix; P_mesh_3d′ is the value of P_mesh_3d′ ijThe matrix composed of represents the matrix form of 3D mesh point coordinates; P_mesh_3d″ represents the matrix after the transposition of P_mesh_3d′; the superscript T represents the transposition of the matrix; P_mesh_2d represents the target two-dimensional coordinates after the conversion; M represents the above-mentioned camera information. Further, for each facial key point and its corresponding target mesh vertex, the target vertex distance is obtained according to the Euclidean distance between the corresponding two-dimensional coordinate and the target two-dimensional coordinate, wherein the above-mentioned target distance may be the sum or the sum of squares of the Euclidean distances.
[0114] In some embodiments of the present application, based on the function corresponding to the preset L2 norm, the target function L corresponding to the target distance is determined according to the following formula (3):
[0115] L = L2_norm(P_face – P_mesh_2d) (3)
[0116] Among them, (P_face–P_mesh_2d) represents the subtraction of the two-dimensional coordinates of the corresponding target key point and the target two-dimensional coordinates of the target mesh vertex.
[0117] 206. Adjust the first model parameters according to the target distance to obtain second model parameters, and adjust the positions of the mesh vertices of the first human body model according to the second model parameters to obtain a second human body model.
[0118] Specifically, the first model parameters can be optimized and adjusted with the minimum target distance as the optimization target to obtain the corresponding second model parameters. It should be noted that during the optimization process, the optimization can be performed only once or multiple optimization iterations can be performed, which is not specifically limited here.
[0119] In some embodiments of the present application, according to the optimization target, the first model parameter is adjusted by using a gradient descent optimization method until a second model parameter that satisfies a preset iteration termination condition is obtained. The above iteration termination condition can be set and adjusted according to actual needs. For example, it can be set to when the number of iterations reaches a number threshold, or when the target distance reaches a distance threshold, or when the target distance converges. Other conditions can also be set, which are not specifically limited here.
[0120] It should be noted that the first model parameters include posture information and shape information corresponding to multiple joint points. During the optimization process, only the posture information of the joint points that will affect the head can be optimized to reduce the amount of data that needs to be processed while obtaining a better head posture reconstruction effect.
[0121] Specifically, the first model parameter includes posture information and shape information of a plurality of joint point mesh vertices, and the joint point mesh vertices are mesh vertices corresponding to the joint points of the target object;
[0122] The step of adjusting the first model parameters according to the target distance to obtain the second model parameters, and adjusting the positions of the mesh vertices of the first human body model according to the second model parameters to obtain the second human body model includes:
[0123] The posture information corresponding to the mesh vertices of the head joint points and the mesh vertices of the neck joint points in the first model parameters is adjusted according to the target distance until a second model parameter that meets a preset iteration termination condition is obtained.
[0124] In the embodiment of the present application, a gradient descent optimization method is used to perform gradient optimization on the head and neck postures (i.e., R5 and R6 parameters). Other posture information, shape information, and camera information remain unchanged and do not participate in the gradient descent update. After 50 rounds of iteration, the optimized head and neck posture information is obtained.
[0125] According to the optimized head and neck posture information, the posture information corresponding to the head and neck in the human posture information pose in the original first model parameters is replaced to obtain the final human body reconstruction information (human body posture information pose, shape information shape and camera information camera), wherein the posture information pose and shape information shape of the reconstructed human body constitute the second model parameters, and according to the second model parameters, based on the above-mentioned improved linear skinning algorithm, the optimized second human body model is obtained. In this way, the facial posture reconstruction effect of the obtained optimized second human body model is better than that of the first human body model.
[0126] Figure 4 is a specific flow chart of a three-dimensional human body reconstruction method provided in an embodiment of the present application, such as Figure 4 As shown, in an embodiment of the present application, a two-dimensional human body image (image A) of the target object is obtained. It should be noted that image A can be input or selected by the user. For image A, face detection is performed to determine the face detection frame, and facial key point recognition is performed on the face detection frame area to obtain the facial key points. For image A, human body detection is performed to obtain a human body detection frame, and SMPL human body reconstruction is performed on the human body detection frame area to obtain the first model parameters. According to the facial key points, the corresponding target mesh vertices in the first model parameters are determined, and fitting optimization is performed according to the coordinates of the facial key points and the target mesh vertices to obtain the optimized second model parameters, thereby obtaining the optimized second human body model. For the specific processing and optimization process, please refer to the specific description above, which will not be repeated here.
[0127] In this way, the human face posture can be reconstructed more accurately while ensuring the accuracy of the human body posture, without the need to combine additional models (such as FLAME), label a large amount of data, or modify the SMPL human body reconstruction algorithm. At the same time, only 6 joint points are used for linear skinning, which greatly improves the speed of calculation and processing compared to the original 24 joint point solution, which is conducive to improving the efficiency of 3D human body reconstruction.
[0128] By adopting the scheme of the embodiment of the present application, a three-dimensional model can be reconstructed based on the two-dimensional human body image of the target object and the preset reference human body model, and first model parameters and coordinate conversion information matching the target object can be obtained, wherein the coordinate conversion information is used to characterize the conversion relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image; the positions of the mesh vertices of the reference human body model are adjusted according to the first model parameters to obtain the first human body model matching the target object; the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image are obtained; the target mesh vertices corresponding to the facial key points in the first human body model are determined, and the three-dimensional coordinates of the target mesh vertices are obtained; according to the coordinate conversion information, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted to the same coordinate system to obtain the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system; the first model parameters are adjusted according to the target distance to obtain the second model parameters, and the positions of the mesh vertices of the first human body model are adjusted according to the second model parameters to obtain the second human body model.
[0129] In this way, when performing three-dimensional human reconstruction, first obtain the first model parameters and the first human model based on the two-dimensional human image and the reference human model, and then perform coordinate transformation on the two-dimensional coordinates of the key points of the face in the two-dimensional human image and / or the three-dimensional coordinates of the target mesh vertices in the first human model, and transform them into the same coordinate system to determine the corresponding target distance, thereby optimizing and adjusting the first model parameters according to the target distance to achieve optimization of the facial reconstruction effect, without combining multiple models, avoiding the significant impact of the three-dimensional human reconstruction scheme combining multiple models on the three-dimensional human reconstruction efficiency, which is conducive to improving the facial reconstruction effect and reducing the impact on the three-dimensional human reconstruction efficiency. Compared with the scheme of combining multiple models for three-dimensional human reconstruction in the related art, the embodiment of the present application can improve the efficiency of three-dimensional human reconstruction on the basis of improving the facial reconstruction effect.
[0130] In view of the three-dimensional human body reconstruction method provided in the first aspect of the embodiment of the present application, the second aspect of the embodiment of the present application provides a three-dimensional human body reconstruction device, please refer to Figure 5 , Figure 5 : is a structural block diagram of a three-dimensional human body reconstruction device provided in an embodiment of the present application, the device comprising:
[0131] A first model parameter acquisition module 501 is used to reconstruct a three-dimensional model according to a two-dimensional human body image of a target object and a preset reference human body model, and obtain first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image;
[0132] A first model reconstruction module 502, configured to adjust the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model matching the target object;
[0133] A facial key point detection module 503 is used to obtain the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image;
[0134] A target mesh vertex matching module 504 is used to determine the target mesh vertex corresponding to the facial key point in the first human body model and obtain the three-dimensional coordinates of the target mesh vertex;
[0135] The target distance calculation module 505 is used to convert the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices into the same coordinate system according to the coordinate conversion information, and obtain the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system;
[0136] The second model reconstruction module 506 is used to adjust the first model parameters according to the target distance to obtain second model parameters, and adjust the positions of the mesh vertices of the first human body model according to the second model parameters to obtain a second human body model.
[0137] In some optional embodiments, the first model parameter acquisition module 501 is specifically used for:
[0138] Performing three-dimensional human body reconstruction on the two-dimensional human body image to obtain third model parameters matching the target object, wherein the third model parameters include posture information and shape information of the target object;
[0139] Acquire adjusted shape information according to the shape information in the third model parameter, the preset shape adjustment parameters and the model parameters of the reference human body model;
[0140] Constructing the first model parameters according to the posture information in the third model parameters and the adjusted shape information;
[0141] The coordinate transformation information between the two-dimensional human body image and the reference human body model is obtained.
[0142] In some optional embodiments, the first model parameter acquisition module 501 is further specifically used for:
[0143] Performing human body detection on the two-dimensional human body image to obtain a human body detection frame of the target object;
[0144] A three-dimensional human body is reconstructed according to the image content within the human body detection frame in the two-dimensional human body image to obtain third model parameters matching the target object.
[0145] In some optional embodiments, the facial key point detection module 503 is specifically used for:
[0146] Performing face detection on the two-dimensional human body image to obtain a face detection frame of the target object;
[0147] Performing facial key point recognition on the image content within the facial detection frame in the two-dimensional human image to obtain facial key points of the target object in the two-dimensional human image;
[0148] The two-dimensional coordinates of the facial key points in the two-dimensional coordinate system where the two-dimensional human body image is located are obtained.
[0149] In some optional embodiments, the target mesh vertex matching module 504 is specifically used for:
[0150] According to preset key point index value association data, obtaining vertex index values of mesh vertices associated with the facial key points in the reference human body model;
[0151] Determining, according to the first human body model and the vertex index values, target mesh vertices corresponding to the facial key points in the first human body model;
[0152] The three-dimensional coordinates of the target mesh vertices in the three-dimensional coordinate system where the first human body model is located are obtained.
[0153] In some optional embodiments, the target distance calculation module 505 is specifically used for:
[0154] According to the coordinate conversion information, the three-dimensional coordinates of the target mesh vertices are converted into the two-dimensional coordinate system where the two-dimensional human body image is located, to obtain the target two-dimensional coordinates of the target mesh vertices, wherein the first human body model and the reference human body model are in the same three-dimensional coordinate system;
[0155] According to the two-dimensional coordinates of the facial key points and the target two-dimensional coordinates of the target mesh vertices corresponding to the facial key points, the target distance between the corresponding facial key points and the target mesh vertices is obtained.
[0156] In some optional embodiments, the first model parameter includes posture information and shape information of a plurality of joint point mesh vertices, and the joint point mesh vertices are mesh vertices corresponding to the joint points of the target object;
[0157] The second model reconstruction module 506 is specifically used to adjust the posture information corresponding to the mesh vertices of the head joint points and the mesh vertices of the neck joint points in the first model parameters according to the target distance, until the second model parameters satisfying the preset iteration termination condition are obtained.
[0158] The embodiment of the present application discloses a three-dimensional human body reconstruction device, which reconstructs a three-dimensional model according to a two-dimensional human body image of a target object and a preset reference human body model through a first model parameter acquisition module 501, and obtains first model parameters and coordinate transformation information matching the above target object, wherein the above coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the above reference human body model and the two-dimensional coordinate system of the above two-dimensional human body image; adjusts the positions of the mesh vertices of the above reference human body model according to the above first model parameters through a first model reconstruction module 502, and obtains a first human body model matching the above target object; obtains facial key point detection module 503 The method comprises the steps of: determining the target mesh vertices corresponding to the facial key points in the first human body model through the target mesh vertex matching module 504, and obtaining the three-dimensional coordinates of the target mesh vertices; converting the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices to the same coordinate system according to the coordinate conversion information through the target distance calculation module 505, and obtaining the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system; adjusting the first model parameters according to the target distance to obtain the second model parameters through the second model reconstruction module 506, and adjusting the positions of the mesh vertices of the first human body model according to the second model parameters to obtain the second human body model.
[0159] In this way, when performing three-dimensional human reconstruction, first obtain the first model parameters and the first human model based on the two-dimensional human image and the reference human model, and then perform coordinate transformation on the two-dimensional coordinates of the key points of the face in the two-dimensional human image and / or the three-dimensional coordinates of the target mesh vertices in the first human model, and transform them into the same coordinate system to determine the corresponding target distance, thereby optimizing and adjusting the first model parameters according to the target distance to achieve optimization of the facial reconstruction effect, without combining multiple models, avoiding the significant impact of the three-dimensional human reconstruction scheme combining multiple models on the three-dimensional human reconstruction efficiency, which is conducive to improving the facial reconstruction effect and reducing the impact on the three-dimensional human reconstruction efficiency. Compared with the scheme of combining multiple models for three-dimensional human reconstruction in the related art, the embodiment of the present application can improve the efficiency of three-dimensional human reconstruction on the basis of improving the facial reconstruction effect.
[0160] It should be noted that the division of each module in the above device can be determined according to actual needs and is not specifically limited here.
[0161] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0162] Accordingly, the embodiment of the present application also provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 6 As shown, Figure 6 600 includes a processor 601 having one or more processing cores, a memory 602 having one or more computer-readable storage media, and a computer program stored in the memory 602 and executable on the processor. The processor 601 is electrically connected to the memory 602. It will be appreciated by those skilled in the art that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.
[0163] The processor 601 is the control center of the electronic device 600, and uses various interfaces and lines to connect various parts of the entire electronic device 600. By running or loading software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, the processor 601 executes various functions of the electronic device 600 and processes data, thereby monitoring the electronic device 600 as a whole. The processor 601 can be a central processing unit CPU, a graphics processing unit GPU, a network processor (NP, Network Processor), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.
[0164] In the embodiment of the present application, the processor 601 in the electronic device 600 will load instructions corresponding to the processes of one or more application programs into the memory 602 according to the following steps, and the processor 601 will run the application programs stored in the memory 602 to implement various functions, such as:
[0165] Reconstructing a three-dimensional model according to the two-dimensional human body image of the target object and a preset reference human body model to obtain first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image;
[0166] Adjusting the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model that matches the target object;
[0167] Obtaining the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image;
[0168] Determine the target mesh vertex corresponding to the facial key point in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertex;
[0169] According to the coordinate conversion information, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted into the same coordinate system, and the target distances between the corresponding facial key points and the target mesh vertices in the same coordinate system are obtained;
[0170] The first model parameters are adjusted according to the target distance to obtain second model parameters, and the positions of the mesh vertices of the first human body model are adjusted according to the second model parameters to obtain the second human body model.
[0171] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0172] Optional, such as Figure 6As shown, the electronic device 600 further includes: a touch screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607, respectively. Those skilled in the art can understand that Figure 6 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0173] The touch display screen 603 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 603 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 601, and can receive the command sent by the processor 601 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 601 to determine the type of touch event, and then the processor 601 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 603 to realize the input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 603 can also be used as a part of the input unit 606 to realize the input function.
[0174] The radio frequency circuit 604 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.
[0175] The audio circuit 605 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 605 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 605 and converted into audio data, and then the audio data is output to the processor 601 for processing, and then sent to another electronic device through the radio frequency circuit 604, or the audio data is output to the memory 602 for further processing. The audio circuit 605 may also include an earplug jack to provide communication between an external headset and an electronic device.
[0176] The input unit 606 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0177] The power supply 607 is used to supply power to various components of the electronic device 600. Optionally, the power supply 607 can be logically connected to the processor 601 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 607 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0178] although Figure 6 Not shown, the electronic device 600 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.
[0179] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0180] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0181] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any one of the three-dimensional human body reconstruction methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:
[0182] Reconstructing a three-dimensional model according to the two-dimensional human body image of the target object and a preset reference human body model to obtain first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image;
[0183] Adjusting the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model that matches the target object;
[0184] Obtaining the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image;
[0185] Determine the target mesh vertex corresponding to the facial key point in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertex;
[0186] According to the coordinate conversion information, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted into the same coordinate system, and the target distances between the corresponding facial key points and the target mesh vertices in the same coordinate system are obtained;
[0187] The first model parameters are adjusted according to the target distance to obtain second model parameters, and the positions of the mesh vertices of the first human body model are adjusted according to the second model parameters to obtain the second human body model.
[0188] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0189] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0190] Since the computer program stored in the storage medium can execute the steps in any one of the three-dimensional human body reconstruction methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the three-dimensional human body reconstruction methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0191] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.
[0192] The above is a detailed introduction to the three-dimensional human body reconstruction method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A three-dimensional human body reconstruction method, characterized in that: include: Reconstructing a three-dimensional model according to the two-dimensional human body image of the target object and a preset reference human body model, obtaining first model parameters and coordinate transformation information matching the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image, and the first model parameters include posture information of a plurality of joint point mesh vertices, and the joint point mesh vertices are mesh vertices corresponding to the joint points of the target object; Adjusting the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model matching the target object; Acquire the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image; Determine a target mesh vertex corresponding to the facial key point in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertex; According to the coordinate conversion information, the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices are converted into the same coordinate system, and the target distances between the corresponding facial key points and the target mesh vertices in the same coordinate system are obtained; According to the target distance, the posture information corresponding to the mesh vertices of the head joint points and the mesh vertices of the neck joint points in the first model parameters is adjusted until the second model parameters that meet the preset iteration termination conditions are obtained, and the positions of the mesh vertices of the first human body model are adjusted according to the second model parameters to obtain the second human body model.
2. The three-dimensional human body reconstruction method according to claim 1, characterized in that: The three-dimensional model reconstruction is performed according to the two-dimensional human body image of the target object and the preset reference human body model to obtain first model parameters and coordinate conversion information matching the target object, including: Performing three-dimensional human body reconstruction on the two-dimensional human body image to obtain third model parameters matching the target object, wherein the third model parameters include posture information and shape information of the target object; Acquire adjusted shape information according to the shape information in the third model parameters, the preset shape adjustment parameters and the model parameters of the reference human body model; Constructing the first model parameters according to the posture information in the third model parameters and the adjusted shape information; Coordinate transformation information between the two-dimensional human body image and the reference human body model is obtained.
3. The three-dimensional human body reconstruction method according to claim 2, characterized in that: The performing three-dimensional human body reconstruction on the two-dimensional human body image to obtain a third model parameter matching the target object includes: Performing human body detection on the two-dimensional human body image to obtain a human body detection frame of the target object; A three-dimensional human body is reconstructed according to the image content within the human body detection frame in the two-dimensional human body image to obtain third model parameters matching the target object.
4. The three-dimensional human body reconstruction method according to claim 1, characterized in that: The step of obtaining the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human body image includes: Performing face detection on the two-dimensional human body image to obtain a face detection frame of the target object; Performing facial key point recognition on the image content within the face detection frame in the two-dimensional human image to obtain facial key points of the target object in the two-dimensional human image; The two-dimensional coordinates of the facial key points in the two-dimensional coordinate system where the two-dimensional human body image is located are obtained.
5. The three-dimensional human body reconstruction method according to claim 1, characterized in that: The step of determining a target mesh vertex corresponding to the facial key point in the first human body model and acquiring a three-dimensional coordinate of the target mesh vertex includes: According to preset key point index value association data, obtaining vertex index values of mesh vertices associated with the facial key points in the reference human body model; Determining, according to the first human body model and the vertex index value, a target mesh vertex corresponding to the facial key point in the first human body model; The three-dimensional coordinates of the target mesh vertices in the three-dimensional coordinate system where the first human body model is located are obtained.
6. The three-dimensional human body reconstruction method according to claim 1, characterized in that: The step of converting the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices into the same coordinate system according to the coordinate conversion information, and obtaining the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system includes: Converting the three-dimensional coordinates of the target mesh vertices into the two-dimensional coordinate system where the two-dimensional human body image is located according to the coordinate conversion information to obtain target two-dimensional coordinates of the target mesh vertices, wherein the first human body model and the reference human body model are in the same three-dimensional coordinate system; According to the two-dimensional coordinates of the facial key points and the target two-dimensional coordinates of the target mesh vertices corresponding to the facial key points, a target distance between the corresponding facial key points and the target mesh vertices is obtained.
7. A three-dimensional human body reconstruction device, characterized in that: include: A first model parameter acquisition module is used to reconstruct a three-dimensional model according to a two-dimensional human body image of a target object and a preset reference human body model, and obtain first model parameters and coordinate transformation information that match the target object, wherein the coordinate transformation information is used to characterize the transformation relationship between the three-dimensional coordinate system of the reference human body model and the two-dimensional coordinate system of the two-dimensional human body image, and the first model parameters include posture information of a plurality of joint point mesh vertices, and the joint point mesh vertices are mesh vertices corresponding to the joint points of the target object; A first model reconstruction module, used for adjusting the positions of the mesh vertices of the reference human body model according to the first model parameters to obtain a first human body model matching the target object; A facial key point detection module, used to obtain the two-dimensional coordinates of the facial key points of the target object in the two-dimensional human image; A target mesh vertex matching module, used to determine the target mesh vertex corresponding to the facial key point in the first human body model, and obtain the three-dimensional coordinates of the target mesh vertex; A target distance calculation module, used to convert the two-dimensional coordinates of the facial key points and the three-dimensional coordinates of the target mesh vertices into the same coordinate system according to the coordinate conversion information, and obtain the target distance between the corresponding facial key points and the target mesh vertices in the same coordinate system; The second model reconstruction module is used to adjust the posture information corresponding to the mesh vertices of the head joint points and the mesh vertices of the neck joint points in the first model parameters according to the target distance until the second model parameters that meet the preset iteration termination conditions are obtained, and adjust the positions of the mesh vertices of the first human body model according to the second model parameters to obtain the second human body model.
8. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the three-dimensional human body reconstruction method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the three-dimensional human body reconstruction method according to any one of claims 1 to 6.
