Three-dimensional human body reconstruction method, device, computer equipment and storage medium

Abstract

The present application discloses a three-dimensional human body reconstruction method, device, computer equipment and storage medium. The method includes obtaining a human body image of a target human body, and inputting it into a first network model for processing to obtain a 3D surface reconstruction result; wherein the first network model includes an image encoder and a fully connected layer. Afterwards, the 3D surface reconstruction result is voxelized using a voxel grid of a preset size to obtain 3D surface sampling points, and the obtained 3D surface sampling points are processed by a second network model to obtain a color prediction result; wherein the second network model includes the image encoder and a multi-layer perceptron. The present application uses an asymmetric network model to fully extract image features and achieve high-quality three-dimensional surface reconstruction. Based on grid sampling of voxel coverage, the coverage of 3D surface sampling points is maximized while ensuring computational efficiency, thereby optimizing the three-dimensional human body reconstruction result.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a three-dimensional human body reconstruction method, device, computer equipment and storage medium. Background Art

[0002] 3D reconstruction is a hot research topic in the field of computer vision. The task of image-based 3D reconstruction is to infer the 3D geometry and structure of objects and scenes from one or more 2D images. Currently, 3D reconstruction has been widely used in various fields such as autonomous driving, robotics, object recognition and scene understanding, AR, VR, 3D modeling and animation, industrial manufacturing, etc. 3D reconstruction based on the human body is an important direction in the field of 3D reconstruction. The current trend is to apply it to augmented reality, medical treatment, personal measurement, clothing and virtual clothing fitting, etc.

[0003] In the prior art, for example, methods based on voxel representation require the space to be divided into voxels. Due to the limitation of voxel resolution, it is difficult for methods based on voxel representation to restore high-quality geometric details. Methods based on parameterized models have certain difficulties in retaining the detailed information of the input image. It is difficult for existing algorithms to simultaneously meet these difficulties, and most methods have too high a time complexity in practical applications. Summary of the invention

[0004] Based on this, the embodiments of the present application provide a three-dimensional human body reconstruction method, apparatus, computer equipment and storage medium, which can solve the problem of optimizing the three-dimensional human body reconstruction results while ensuring computing efficiency.

[0005] In a first aspect, a three-dimensional human body reconstruction method is provided, the method comprising:

[0006] Acquire a human body image of a target human body;

[0007] Processing the human body image through a first network model to obtain a 3D surface reconstruction result; the first network model includes an image encoder and a fully connected layer;

[0008] voxelize the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points;

[0009] The 3D surface sampling points are processed by a second network model to obtain a color prediction result; the second network model includes the image encoder and a multi-layer perceptron.

[0010] In one embodiment, the first network model is represented by I1=F CN (F(x),D(X));

[0011] Where I1∈R represents the 3D surface reconstruction result, F CNrepresents the fully connected layer, F(x) represents the output of the image encoder, D(X) represents the depth value in the camera coordinate space, X represents the 3D point in the human body image, and x represents the 2D projection of the 3D point in the human body image.

[0012] In one embodiment, the first network model measures the 3D surface reconstruction result by a first loss function, and the first loss function includes:

[0013]

[0014] Wherein, ΔF represents the error between the predicted value and the true value of the 3D surface.

[0015] In one embodiment, voxelizing the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points includes:

[0016] The voxel grid size is (g x ,g y ,g z ) voxelizing the 3D surface reconstruction result; mapping each three-dimensional point in the 3D surface to a voxel index;

[0017] And label all non-empty voxels as V i , randomly select several voxels from all the non-empty voxels as the central voxel V c , select the central voxel V c A random point in the c ;

[0018] Update the central voxel V c , each central voxel V c The center point P c Through the K nearest neighbor algorithm, the voxel grid V neigh Find k neighboring points P k As sampling points, finally all the 3D surface sampling points P are obtained sample .

[0019] In one embodiment, the central voxel V is updated c ,include:

[0020] Central voxel V c The difference between the updated voxel coverage and the unupdated coverage is calculated. If the difference is greater than 0, the central voxel is updated, otherwise it is not updated.

[0021] The difference calculation includes:

[0022]

[0023] Among them, m netx Represents the number of central voxels after update, m percent Indicates the number of current central voxels, V neigh (V next ) represents the updated central voxel V next The non-empty neighborhood voxels of neigh (V percent ) represents the current center voxel V percent The function f represents the calculation of the coverage of each central voxel.

[0024] In one embodiment, the second network model is represented by I2=F MLP (F C (x i ),D(X i ));

[0025] Among them, I2∈R represents the 3D surface color prediction result, F MLP represents the multilayer perceptron, X i represents the i-th 3D surface sampling point in the human body image, x i represents the 2D projection of the i-th 3D surface sampling point in the human body image, F C (x i ) represents x i At the output of the image encoder, D(X i ) represents the depth value in the camera coordinate space.

[0026] In one embodiment, the second network model measures the color prediction result by a second loss function, and the second loss function includes:

[0027]

[0028] Among them, C gt (X i ) represents the 3D surface sampling point X i The true color value.

[0029] In a second aspect, a three-dimensional human body reconstruction device is provided, the device comprising:

[0030] An acquisition module, used for acquiring a human body image of a target human body;

[0031] A first network module, used for processing the human body image through a first network model to obtain a 3D surface reconstruction result; the first network model includes an image encoder and a fully connected layer;

[0032] A voxelization module, used for voxelizing the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points;

[0033] The second network module is used to process the 3D surface sampling points through a second network model to obtain a color prediction result; the second network model includes the image encoder and a multi-layer perceptron.

[0034] In a third aspect, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the three-dimensional human body reconstruction method described in any one of the first aspects is implemented.

[0035] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional human body reconstruction method described in any one of the first aspects is implemented.

[0036] The technical solution provided in the embodiment of the present application obtains a human body image of the target human body, and inputs it into a first network model for processing to obtain a 3D surface reconstruction result; wherein, the first network model includes an image encoder and a fully connected layer. Afterwards, the 3D surface reconstruction result is voxelized using a voxel grid of a preset size to obtain 3D surface sampling points, and the obtained 3D surface sampling points are processed by a second network model to obtain a color prediction result; wherein, the second network model includes the image encoder and a multi-layer perceptron. It can be seen that compared with the prior art, the asymmetric network model of the present application realizes the full extraction of image features and realizes high-quality three-dimensional surface reconstruction. The grid sampling based on the voxel coverage range maximizes the coverage of the 3D surface sampling points while ensuring computational efficiency, thereby optimizing the three-dimensional human body reconstruction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a three-dimensional human body reconstruction method provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a three-dimensional human body reconstruction method provided in an embodiment of the present application;

[0039] Figure 3 A block diagram of a three-dimensional human body reconstruction device provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0042] Please refer to Figure 1 , which shows a flow chart of a 3D human body reconstruction method provided in an embodiment of the present application. The 3D human body reconstruction method may include the following steps:

[0043] Step 101, obtaining a human body image of a target human body.

[0044] In the embodiment of the present application, the human body image of the target human body refers to a single-view RGB image containing the human body. The RGB color mode is a color standard in the industry, which is obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other to obtain various colors. RGB represents the colors of the three channels of red, green, and blue.

[0045] Step 102: Processing the human body image through the first network model to obtain a 3D surface reconstruction result.

[0046] The first network model includes an image encoder and a fully connected layer. In an embodiment of the present application, the first network model consists of an image encoder and a continuous implicit function FCN represented by a fully connected layer.

[0047] Specifically, the image encoder may include five encoding layers, the human body image is input into the first encoding layer for encoding to obtain a first image feature; the first image feature is input into the second encoding layer for encoding to obtain a second image feature; the second image feature is input into the third encoding layer for encoding to obtain a third image feature; the third image feature is input into the fourth encoding layer for encoding to obtain a fourth image feature; the fourth image feature is input into the fifth encoding layer for encoding to obtain an output F(x) of the image encoder;

[0048] Among them, the structure in each coding layer is the same. The embodiment of the present application includes a method for extracting image features based on an asymmetric convolutional network. Specifically, in the first coding layer, a single-view RGB image (human image) containing a human body is input; the human body image is input into a two-dimensional convolutional network with a convolution kernel of 1x3 to extract image feature f11; the image feature f11 is input into a two-dimensional convolutional network with a convolution kernel of 3x1 to extract image feature f12; the human body image is input into a two-dimensional convolutional network with a convolution kernel of 3x1 to extract image feature f21; the image feature f21 is input into a two-dimensional convolutional network with a convolution kernel of 1x3 to extract image feature f22; the image feature f12 is added to the image feature f22 for feature fusion to obtain the first image feature. The subsequent coding layer uses the output of the previous coding layer as input, repeats four times, and obtains the output F(x) of the image encoder.

[0049] The output 3D surface of the first network model can be represented as a level set, which includes implicit functions based on the image encoder and the fully connected layer representation, namely:

[0050] I1=F CN (F(x),D(X))

[0051] Among them, I1∈R represents the 3D surface reconstruction result, F CN represents a fully connected layer, F(x) represents the output of the image encoder, D(X) represents the depth value in the camera coordinate space, X represents a 3D point in the human image, and x represents the 2D projection of the 3D point in the human image.

[0052] The real surface is represented as a continuous 3D occupancy domain with a 0.5 level set F gt (X). If X is on the mesh surface, then F gt (X) = 1; if X is not on the grid surface, then F gt (X)=0.

[0053] The error between the predicted value and the true value of the 3D surface is expressed as ΔF = F CN (F(x),D(X))-F gt (X). The 3D surface reconstruction result is measured by the first loss function, which includes:

[0054]

[0055] Wherein, ΔF represents the error between the predicted value and the true value of the 3D surface.

[0056] Step 103 : voxelize the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points.

[0057] In the embodiment of the present application, the sampling strategy will largely affect the final reconstruction effect, and a grid sampling strategy is used to sample the three-dimensional surface, including grid sampling based on voxel coverage.

[0058] Step 1031, using a voxel grid size of (g x ,g y ,g z ) voxelize the 3D surface reconstruction result;

[0059] Step 1032, map each 3D point in the 3D surface to a voxel index

[0060]

[0061] Among them, x, y, z represent the coordinates of the three-dimensional point, g x , g y , gz Represents the voxel grid size, and Map represents the mapping function.

[0062] Step 1033, mark all non-empty voxels as V i , randomly select several voxels from all non-empty voxels as the central voxel V c , since the central voxel V c Contains multiple points, select the central voxel V c A random point in the c ;

[0063] Central voxel V c The neighborhood voxel grid is set to center on the voxel V c All 5x5 voxel grids centered at neigh .

[0064] Step 1034, update the central voxel V c , each central voxel V c The center point P c Through the K nearest neighbor algorithm, the voxel grid V neigh Find k neighboring points P k As sampling points, all 3D surface sampling points P are finally obtained sample .

[0065] In this embodiment of the present application, the central voxel V c The update standard is to calculate the difference between the updated voxel coverage and the unupdated coverage. If the difference is greater than 0, the central voxel is updated, otherwise it is not updated. Specifically, it includes:

[0066] Central voxel V c The difference between the updated voxel coverage and the unupdated coverage is calculated. If the difference is greater than 0, the central voxel is updated, otherwise it is not updated.

[0067] The difference calculation includes:

[0068]

[0069] Among them, m netx Represents the number of central voxels after update, m percent Indicates the number of current central voxels, V neigh (V next ) represents the updated central voxel V next The non-empty neighborhood voxels of neigh (V percent ) represents the current center voxel V percent The function f represents the calculation of the coverage of each central voxel.

[0070] Step 104: Process the 3D surface sampling points through the second network model to obtain a color prediction result.

[0071] The second network model includes an image encoder and a multi-layer perceptron. In the embodiment of the present application, the output of the second network model is defined as an RGB vector field to predict the RGB color on the 3D surface. The second network model is represented by I2=F MLP (F C (x i ),D(X i ));

[0072] Among them, I2∈R represents the 3D surface color prediction result, F MLP represents a multi-layer perceptron, X i represents the i-th 3D surface sampling point in the human body image, x i represents the 2D projection of the i-th 3D surface sampling point in the human body image, F C is an encoder, which in the embodiment of the present application may be the same as the above encoder, and has five encoding layers, and the specific encoding process is not repeated here; F C (x i ) represents x i At the output of the image encoder, D(X i ) represents the depth value in the camera coordinate space. Calculate the error between the color prediction value and the true color value of the 3D surface sampling point, expressed as ΔC = F MLP (F c (x i ),D(X i ))-C gt (X i ).

[0073] The second network model measures the color prediction results through the second loss function, which includes:

[0074]

[0075] Among them, C gt (X i ) represents the 3D surface sampling point X i The true color value of . By minimizing the loss function, the optimal 3D surface color prediction is achieved.

[0076] In an embodiment of the present application, two loss functions are designed to measure the predicted value and true value errors of the surface and color respectively. In another optional embodiment of the present application, the loss function L1 and the loss function L2 are added to obtain the loss function L, and the optimal 3D surface reconstruction result is achieved by minimizing the loss function L.

[0077] This application designs an asymmetric convolutional network, which makes full use of the different focuses of feature extraction of different convolution kernels to fully extract image features; designs a 3D reconstruction strategy based on implicit functions, which makes full use of image features and color inference to achieve high-quality 3D surface reconstruction; designs a grid sampling based on voxel coverage, which maximizes the coverage of 3D surface sampling points while ensuring computational efficiency, thereby optimizing the 3D human body reconstruction results; designs two loss functions to measure the predicted value and true value errors of 3D surface and color respectively, and achieves surface detail reconstruction by minimizing the loss function.

[0078] It can be seen that by making full use of the different emphases of different convolution kernels on feature extraction, the full extraction of image features is achieved. Then, the present invention makes full use of image features and color inference to achieve high-quality three-dimensional surface reconstruction. Then, the present invention maximizes the coverage of 3D surface sampling points while ensuring computational efficiency, thereby optimizing the three-dimensional human body reconstruction results. Finally, the present invention designs two loss functions to measure the predicted value and true value errors of the surface and color respectively, and achieves surface detail reconstruction by minimizing the loss function.

[0079] Please refer to Figure 3 , which shows a block diagram of a 3D human body reconstruction device 300 provided in an embodiment of the present application. Figure 3 As shown, the device 300 may include: an acquisition module 301 , a first network module 302 , a voxelization module 303 , and a second network module 304 .

[0080] An acquisition module 301 is used to acquire a human body image of a target human body;

[0081] The first network module 302 is used to process the human body image through the first network model to obtain a 3D surface reconstruction result; the first network model includes an image encoder and a fully connected layer;

[0082] A voxelization module 303 is used to voxelize the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points;

[0083] The second network module 304 is used to process the 3D surface sampling points through a second network model to obtain a color prediction result; the second network model includes an image encoder and a multi-layer perceptron.

[0084] For the specific definition of the three-dimensional human body reconstruction device, please refer to the definition of the three-dimensional human body reconstruction method above, which will not be repeated here. Each module in the above-mentioned three-dimensional human body reconstruction device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0085] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a display and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for three-dimensional human body reconstruction data. The network interface of the computer device is used to communicate with an external terminal through a network connection, and the display is used to display the three-dimensional human body reconstruction result. When the computer program is executed by the processor, a three-dimensional human body reconstruction method is implemented.

[0086] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional human body reconstruction method are implemented.

[0088] The computer-readable storage medium provided in this embodiment has similar implementation principles and technical effects to those of the above method embodiments, and will not be described in detail here.

[0089] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A three-dimensional human body reconstruction method, characterized in that: The method comprises: Acquire a human body image of a target human body; Processing the human body image through a first network model to obtain a 3D surface reconstruction result; the first network model includes an image encoder and a fully connected layer; voxelize the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points; Processing the 3D surface sampling points through a second network model to obtain a color prediction result; the second network model includes the image encoder and a multi-layer perceptron; The first network model is represented by I1=F CN (F(x),D(x)); Where I1∈R represents the 3D surface reconstruction result, F CN represents the fully connected layer, F(x) represents the output of the image encoder, D(X) represents the depth value in the camera coordinate space, X represents the 3D point in the human image, and x represents the 2D projection of the 3D point in the human image; The first network model measures the 3D surface reconstruction result by a first loss function, and the first loss function includes: Among them, ΔF represents the error between the predicted value and the true value of the 3D surface; The step of voxelizing the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points includes: The voxel grid size is (g x ,g y ,g z ) voxelizing the 3D surface reconstruction result; mapping each 3D point in the 3D surface to a voxel index; And label all non-empty voxels as V i , randomly select several voxels from all the non-empty voxels as the central voxel V c , select the central voxel V c A random point in the c ; Update the central voxel V c , each central voxel V c The center point P c Through the K nearest neighbor algorithm, the voxel grid V neigh Find k neighboring points P k As sampling points, finally all the 3D surface sampling points P are obtained sample .

2. The method according to claim 1, updating the central voxel V c ,include: Central voxel V c The difference between the updated voxel coverage and the unupdated coverage is calculated. If the difference is greater than 0, the central voxel is updated, otherwise it is not updated. The difference calculation includes: Among them, m netx Represents the number of central voxels after update, m percent Indicates the number of current central voxels, V neigh (V next ) represents the non-empty neighboring voxels of the updated central voxel Vnext, V neigh (V percent ) represents the current center voxel V percent The function f represents the calculation of the coverage of each central voxel.

3. The method according to claim 1, characterized in that The second network model is expressed as I2=F MLP (F C (x i ),D(X i )); Among them, I2∈R represents the 3D surface color prediction result, F MLP represents the multilayer perceptron, X i represents the i-th 3D surface sampling point in the human body image, x i represents the 2D projection of the i-th 3D surface sampling point in the human body image, F C (x i ) represents x i At the output of the image encoder, D(X i ) represents the depth value in the camera coordinate space.

4. The method according to claim 3, characterized in that: The second network model measures the color prediction result by a second loss function, and the second loss function includes: Among them, C gt (X i ) represents the 3D surface sampling point X i The true color value.

5. A three-dimensional human body reconstruction device, characterized in that: The device comprises: An acquisition module, used for acquiring a human body image of a target human body; A first network module, used for processing the human body image through a first network model to obtain a 3D surface reconstruction result; the first network model includes an image encoder and a fully connected layer; A voxelization module, used for voxelizing the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points; A second network module, used for processing the 3D surface sampling points through a second network model to obtain a color prediction result; the second network model includes the image encoder and a multi-layer perceptron; The first network model is represented by I1=F CN (F(x),D(x)); Where I1∈R represents the 3D surface reconstruction result, F CN represents the fully connected layer, F(x) represents the output of the image encoder, D(X) represents the depth value in the camera coordinate space, X represents the 3D point in the human image, and x represents the 2D projection of the 3D point in the human image; The first network model measures the 3D surface reconstruction result by a first loss function, and the first loss function includes: Among them, ΔF represents the error between the predicted value and the true value of the 3D surface; The step of voxelizing the 3D surface reconstruction result using a voxel grid of a preset size to obtain 3D surface sampling points includes: The voxel grid size is (g x ,g y ,g z ) voxelizing the 3D surface reconstruction result; mapping each 3D point in the 3D surface to a voxel index; And label all non-empty voxels as V i , randomly select several voxels from all the non-empty voxels as the central voxel V c , select the central voxel V c A random point in the c ; Update the central voxel V c , each central voxel V c The center point P c Through the K nearest neighbor algorithm, the voxel grid V neigh Find k neighboring points P k As sampling points, finally all the 3D surface sampling points P are obtained sample .

6. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the three-dimensional human body reconstruction method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the three-dimensional human body reconstruction method as described in any one of claims 1 to 4 is implemented.

Images