Human body three-dimensional reconstruction method, system, equipment, medium and product

Abstract

The invention discloses a human body three-dimensional reconstruction method, system and device, a medium and a product, and relates to the technical field of computer vision and sparse visual angle three-dimensional reconstruction, and the method comprises the steps: obtaining a multi-visual angle RGB image of a to-be-shot human body; the multi-view RGB image is preprocessed, and a preprocessed multi-view RGB image is obtained; inputting the preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; fusing the plurality of main view angle human body depth prediction maps to obtain a fused main view angle human body depth prediction map; based on the fused main view angle human body depth prediction map, performing human body three-dimensional reconstruction by adopting a three-dimensional reconstruction method to obtain a human body three-dimensional reconstruction result; the human body three-dimensional reconstruction result is a human body surface grid. According to the invention, the surface smoothness of the human body three-dimensional reconstruction result is improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and sparse-view three-dimensional reconstruction, and in particular to a method, system, device, medium and product for three-dimensional reconstruction of a human body. Background Art

[0002] With the popularization of mobile computing devices with high-quality image acquisition capabilities and the rapid development of virtual reality technology, people have an unprecedented high-intensity demand for creating 3D content, but how to quickly and conveniently create high-precision 3D content remains an unresolved problem. For the popularization of 3D content creation, cheap equipment and simple operation are inevitable requirements. However, existing methods, such as 3D scanning and multi-view stereo algorithms, can measure and reconstruct physical objects with high precision, but they are limited by expensive capture equipment and rely on complex capture systems with strict environmental restrictions (such as controlled lighting). It is difficult for non-professionals to use, and the high cost of use is also prohibitive, which greatly limits the application scope of three-dimensional reconstruction of objects.

[0003] High-precision 3D reconstruction of the human body is an important research topic in 3D content creation. It is one of the key technologies in application fields such as medical imaging, games and VR. Modern film shooting and game production have become increasingly inseparable from this technology. In order to obtain a truly immersive experience, animators need to spend a lot of effort to manually draw virtual characters or use expensive capture systems, which increases the cost of use and strictly limits the scope of use. If the 3D reconstruction technology of the human body can use the high-quality image sensors of mobile devices to obtain high-quality digital human models as easily as taking pictures with a mobile phone, it will have a revolutionary impact on game production and virtual reality technology. In recent years, people have increasingly shifted their research focus to 3D reconstruction of the human body using deep learning models. Although most of these methods are still far below the level that professional capture systems can achieve, they still show great potential. The current 3D reconstruction scheme is designed based on discretized voxels, and there is a problem of rough reconstructed surface, which is far from the commercial stage.

[0004] Therefore, a human body three-dimensional reconstruction method is urgently needed to solve the above problems. Summary of the invention

[0005] The purpose of this application is to provide a method, system, device, medium and product for three-dimensional reconstruction of the human body, which improves the surface smoothness of the three-dimensional reconstruction results of the human body.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for three-dimensional reconstruction of a human body, the method comprising:

[0008] Acquire a multi-view RGB image of a human body to be photographed; the multi-view RGB image includes: a main-view RGB image and a top-view RGB image;

[0009] Preprocessing the multi-view RGB image to obtain a preprocessed multi-view RGB image;

[0010] The preprocessed multi-view RGB image is input into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map;

[0011] Fusing multiple main-view human body depth prediction images to obtain a fused main-view human body depth prediction image;

[0012] Based on the fused main-view human body depth prediction map, a three-dimensional reconstruction method is used to perform three-dimensional reconstruction of the human body to obtain a three-dimensional reconstruction result of the human body; the three-dimensional reconstruction result of the human body is a human body surface mesh.

[0013] Optionally, obtaining a multi-view RGB image of a human body to be photographed specifically includes:

[0014] A plurality of RGBD cameras are used to surround the human body to be photographed at preset angle intervals to perform real-time synchronous photography of the human body to be photographed, thereby obtaining a multi-view RGB image of the human body to be photographed.

[0015] Optionally, preprocessing the multi-view RGB image to obtain a preprocessed multi-view RGB image specifically includes:

[0016] The contrast and saturation of the multi-view RGB image are adjusted in sequence to obtain an adjusted multi-view RGB image;

[0017] The artificial blurring method is used to perform low-resolution simulation on the adjusted multi-view RGB image to obtain the preprocessed multi-view RGB image.

[0018] Optionally, the CNN-based deep learning network includes a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence.

[0019] Optionally, the training process of the multi-view human body depth map prediction model specifically includes:

[0020] Get the training dataset;

[0021] The preprocessed sample multi-view RGB image is input into the CNN-based deep learning network, and the sample main view human body depth prediction map is output;

[0022] A loss function is constructed according to the sample main-view human body depth prediction map and the sample main-view human body depth map, and the network parameters of the CNN-based deep learning network are iteratively optimized according to the loss function until the iterative optimization round reaches a maximum value or the loss function reaches a minimum value, and the iterative optimization is stopped to obtain the multi-view human body depth map prediction model.

[0023] Optionally, the three-dimensional reconstruction method is Poisson reconstruction.

[0024] In a second aspect, the present application provides a human body 3D reconstruction system, the human body 3D reconstruction system is used to implement the human body 3D reconstruction method, the human body 3D reconstruction system comprises:

[0025] An image acquisition unit is used to acquire a multi-view RGB image of a human body to be photographed; the multi-view RGB image includes: a main-view RGB image and a top-view RGB image;

[0026] A preprocessing unit, used to preprocess the multi-view RGB image to obtain a preprocessed multi-view RGB image;

[0027] A main-view human body depth prediction map determining unit is used to input the preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map;

[0028] An image fusion unit, used for fusing a plurality of main-view human body depth prediction images to obtain a fused main-view human body depth prediction image;

[0029] The human body 3D reconstruction result determination unit is used to perform 3D reconstruction of the human body using a 3D reconstruction method based on the fused main-view human body depth prediction map to obtain a 3D reconstruction result of the human body; the 3D reconstruction result of the human body is a human body surface mesh.

[0030] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-described methods for three-dimensional reconstruction of a human body.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for three-dimensional reconstruction of the human body.

[0032] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for three-dimensional reconstruction of the human body.

[0033] According to the specific embodiments provided in this application, this application has the following technical effects:

[0034] The present application discloses a method, system, device, medium and product for three-dimensional reconstruction of a human body. The method comprises the following steps: inputting a preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map, and fusing multiple main-view human body depth prediction maps to obtain a fused main-view human body depth prediction map. Based on the fused main-view human body depth prediction map, a three-dimensional reconstruction method is used to perform three-dimensional reconstruction of the human body to obtain a three-dimensional reconstruction result of the human body. The main-view human body depth prediction map predicted by the multi-view human body depth map prediction model has a smoother surface depth, thereby improving the smoothness of subsequent three-dimensional reconstruction of the human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of a process flow of a human body three-dimensional reconstruction method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a training process of a CNN-based deep learning network provided in one embodiment of the present application;

[0038] Figure 3 A schematic diagram of functional modules of a human body three-dimensional reconstruction system provided in one embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0040] Reference numerals:

[0041] Image acquisition unit-1, preprocessing unit-2, main view human body depth prediction map determination unit-3, image fusion unit-4, human body three-dimensional reconstruction result determination unit 5. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] In an exemplary embodiment, Figure 1 As shown, a method for three-dimensional reconstruction of a human body is provided, comprising the following steps S1 to S5. Among them:

[0045] Step S1, obtaining a multi-view RGB image of a human body to be photographed; the multi-view RGB image includes: a main-view RGB image and a top-view RGB image.

[0046] As an optional implementation, step S1 specifically includes:

[0047] Multiple RGBD cameras are used to surround the human body to be photographed at preset angle intervals to synchronously shoot the human body to be photographed in real time, and obtain a multi-view RGB image of the human body to be photographed. The specific shooting scene is that 8 RGBD cameras evenly surround the human body to be photographed at 45-degree intervals, face the center, and shoot synchronously in real time in a hardware synchronization manner.

[0048] Step S2: preprocess the multi-view RGB image to obtain a preprocessed multi-view RGB image.

[0049] As an optional implementation, step S2 specifically includes:

[0050] Step S21, adjusting the contrast and saturation of the multi-view RGB image in sequence to obtain an adjusted multi-view RGB image.

[0051] Step S22, using an artificial blur method, performs low-resolution simulation on the adjusted multi-view RGB image to obtain a pre-processed multi-view RGB image, simulating the image conditions captured by different devices.

[0052] Step S3, input the preprocessed multi-view RGB image into the multi-view human body depth map prediction model to obtain the main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map.

[0053] As an optional implementation, in step S3, the CNN-based deep learning network includes a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence.

[0054] Specifically, Figure 2 As shown, step 1): use a multi-level image feature encoder with self-attention (i.e. Figure 2 The Deform FPN shown in the figure extracts high-level semantic features of the input multi-view RGB image (including: high-level semantic features of the main-view RGB image and high-level semantic features of the top-view RGB image); Step 2): Then, in the loss space construction layer with local semantic features, the high-level semantic features of the multi-view RGB image are back-projected into 3D voxels according to camera parameters (i.e., internal and external parameters of the camera, the external parameters are the camera pose, and the internal parameters include the camera focal length and distortion coefficient), and the variance of the back-projected image features of each voxel at different viewpoints is calculated to obtain the feature differences at different viewpoints, and each voxel at different viewpoints is back-projected into 3D voxels. The variance of the back-projected image features is combined with the high-level semantic features of the main view image to obtain a loss space with local semantic features (i.e., 3D loss space), which is used to maintain spatial continuity and obtain a smoother surface depth. This solves the problem of different voxel discretization and obvious noise in the depth map obtained in the traditional method, and a depth map with a smooth surface is obtained based on the high-level semantic features; step 3): on the loss space with local semantic features, under the guidance of the local semantic features, a sparse convolution module is used to perform sparse 3D convolution that maintains the original voxel adjacency relationship to obtain a first-resolution depth map of the main view.

[0055] Figure 2 In the figure, W represents feature deformation, which is used to back-project image features onto the rays emitted by each pixel of the main view, and calculate the variance of image features of different views. Because the first level is uniformly sampled on the pixel rays, the space is fully covered, and ordinary 3D convolution is used later; F represents voxel feature fusion, which fuses the variance of the main view feature and the multi-view feature, and directly splices them together to obtain a loss space with local semantic features, and then performs 3D convolution. The convolution result is the probability that each point is a surface point. The weighted sum of the point probability on a ray and the depth of the point is the estimated depth of the point. 3D represents 3D convolution; S represents 3D sparse convolution, that is, only the part with value in the voxel is convolved. Figure 2 On the right are three depth predictions at different resolutions (i.e. Figure 2 Pred Depth in , they use multi-view image features of different resolutions on the left, and the three levels are cascaded together; Figure 2 The far right side (i.e. GT-One-Hot) represents MSE Loss, which is used to supervise the probability that each spatial point is a surface point.

[0056] The traditional method is to directly extract the sampling points for 3D convolution, but the relative position relationship of the points will change. This application chooses to put the points in the original position in space, use sparse convolution, only calculate the points with values, and retain the relative position relationship of the points. The rest of the process is the same as the first level.

[0057] Furthermore, the calculation formula of the variance of the back-projected image features of each voxel at different viewing angles is as follows:

[0058]

[0059] Where N is the number of viewing angles, V i is the image feature of the i-th perspective, is the mean of the image features back-projected to this point; Var is the variance of the image features back-projected to this point.

[0060] The calculation formula of the loss space with local semantic features is as follows:

[0061] C=Concate(Var,V s )(2)

[0062] Among them, C is the loss space with local semantic features; Concate means splicing in the channel dimension; Vs is the high-level semantic feature of the main view image.

[0063] Then, the first resolution depth map of the main view is interpolated and upsampled to obtain the second resolution depth map of the main view, and the second resolution depth map of the main view is used as the basis to construct the loss space with local semantic features corresponding to the second resolution and focused on the base depth (i.e., the depth estimated at the previous level), and then sparse 3D convolution is continued to obtain the third resolution depth map of the main view, and then interpolation upsampling is continued-to obtain the fourth resolution depth map-to construct the loss space with local semantic features corresponding to the fourth resolution-sparse 3D convolution-to obtain the fifth resolution depth map of the main view, that is, repeat this step until the target resolution depth map of the main view is obtained; wherein the target resolution is the resolution of the corresponding sample main view human body depth map, and the resolution of the corresponding sample main view human body depth map is consistent with the resolution of the input main view RGB image, and the first resolution is less than the second resolution, the second resolution is less than the third resolution, the third resolution is less than the fourth resolution, and the fourth resolution is less than the fifth resolution, that is, compared with the first resolution depth map of the main view, a higher resolution depth map of the main view is obtained through continuous training. Through multi-resolution depth map prediction, the computational complexity and video memory requirements are reduced.

[0064] Specifically, in step 1), a multi-level image feature encoder with self-attention is used to extract high-level semantic features of the input multi-view RGB image, specifically: using deformable convolution, real-time perception of the receptive field range and perception direction required to obtain the detailed features of the current feature points, and real-time adjustment of the convolution kernel shape to solve the limitation of the fixed receptive field of the traditional convolution algorithm. Among them, high-dimensional features are used for convolution operations to predict finer receptive field ranges and perception directions, and obtain finer image features, including local image features and more extensive semantic features.

[0065] In the process of constructing a multi-level image feature encoder with self-attention, the image feature extraction layer using self-attention is embedded into the pyramid feature extractor (i.e. Figure 2 In the deformable convolutional pyramid feature extraction network on the far left, which is used to extract multi-level image features with semantics and output image features of different resolutions, the self-attention image feature layer can use higher-dimensional image features to obtain a finer receptive field range and perception direction. At the same time, the pyramid feature extractor can also use the semantic information obtained from the self-attention image feature layer to obtain multi-level image features with higher information density.

[0066] In step 3), in the second stage and above stages using multi-resolution depth map prediction, the traditional method subtracts the basic depth when constructing the loss space, causing the voxels to lose their original adjacency relationship. The present application adopts a sparse convolution method in the second stage and above to maintain the original adjacency relationship of the voxels, and introduces local high-level semantic features in each voxel to guide the original 3D convolution to obtain a smoother surface depth.

[0067] As an optional implementation, in step S3, the training process of the multi-view human body depth map prediction model specifically includes:

[0068] Step S31, obtaining a training data set.

[0069] In the process of multi-view RGB image acquisition, the training data set is obtained by using data augmentation: different lighting is applied to the human body to be photographed, and RGB images taken under different lighting conditions are obtained to meet the system's adaptability to collecting photos under different lighting conditions during the application process.

[0070] In addition, each pixel of the sample main-view human body depth map and the RGB image is aligned. The depth map is obtained by shooting with a depth camera, and the sample main-view human body depth map and the RGB image are shot synchronously in real time in a hardware synchronization manner.

[0071] Step S32, input the preprocessed sample multi-view RGB image into a CNN-based deep learning network, and output the sample main-view human body depth prediction map.

[0072] Step S33, constructing a loss function according to the sample main-view human body depth prediction map and the sample main-view human body depth map, and iteratively optimizing the network parameters of the CNN-based deep learning network according to the loss function until the iterative optimization round reaches a maximum value or the loss function reaches a minimum value, stopping the iterative optimization, and obtaining the multi-view human body depth map prediction model.

[0073] Step S4, fuse the multiple main-view human body depth prediction maps to obtain a fused main-view human body depth prediction map. Specifically, the multiple main-view human body depth prediction maps are obtained by replacing the main-view RGB image with the different top-view RGB images in step S3 to obtain multiple main-view human body depth prediction maps. The multiple main-view human body depth prediction maps are fused to obtain a smooth and complete grid, reducing the interference of single-view depth errors.

[0074] Step S5, based on the fused main-view human body depth prediction map, a three-dimensional reconstruction method is used to perform three-dimensional reconstruction of the human body to obtain a three-dimensional reconstruction result of the human body; the three-dimensional reconstruction result of the human body is a human body surface mesh.

[0075] Specifically, the fused main-view human depth prediction map is projected into 3D space, and a smooth human surface mesh is reconstructed using methods such as Poisson reconstruction to obtain a three-dimensional reconstruction result of the human body. That is to say, the fused main-view human depth prediction map is projected into 3D space according to camera parameters to obtain a multi-view point cloud, and point clouds of different viewpoints are fused according to back-projection, and point clouds with large back-projection errors are removed, and then a smooth human surface mesh is reconstructed using methods such as Poisson reconstruction.

[0076] Beneficial effects of this application:

[0077] In the multi-view human depth map prediction model constructed by this application, sparse convolution and high-level semantic information injection are used to obtain a more precise, smoother, and more visually compliant human surface mesh from sparse RGB images, solving the problems of high equipment requirements, high labor costs, and rough surface of human 3D reconstruction results in traditional methods. The system can be used to assist in 3D character modeling in games, construction of virtual reality digital twin characters, etc.

[0078] Based on the same inventive concept, the embodiment of the present application also provides a human body 3D reconstruction system for implementing the human body 3D reconstruction method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more human body 3D reconstruction system embodiments provided below can refer to the limitations of the human body 3D reconstruction method above, and will not be repeated here.

[0079] In an exemplary embodiment, Figure 3 As shown, a human body 3D reconstruction system is provided, including: an image acquisition unit 1, a preprocessing unit 2, a main-view human body depth prediction map determination unit 3, an image fusion unit 4 and a human body 3D reconstruction result determination unit 5.

[0080] The image acquisition unit 1 is used to acquire a multi-view RGB image of a human body to be photographed; the multi-view RGB image includes: a main-view RGB image and a top-view RGB image.

[0081] The preprocessing unit 2 is used to preprocess the multi-view RGB image to obtain a preprocessed multi-view RGB image.

[0082] The main-view human body depth prediction map determination unit 3 is used to input the preprocessed multi-view RGB image into the multi-view human body depth map prediction model to obtain the main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map.

[0083] The image fusion unit 4 is used to fuse multiple main-view human body depth prediction images to obtain a fused main-view human body depth prediction image.

[0084] The human body 3D reconstruction result determination unit 5 is used to perform 3D reconstruction of the human body using a 3D reconstruction method based on the fused main-view human body depth prediction map to obtain a 3D reconstruction result of the human body; the 3D reconstruction result of the human body is a human body surface mesh.

[0085] In an exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for three-dimensional reconstruction of a human body.

[0086] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a method for three-dimensional reconstruction of a human body is implemented.

[0087] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements a method for three-dimensional reconstruction of a human body when executed by a processor.

[0088] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for three-dimensional reconstruction of the human body is implemented.

[0089] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0092] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for three-dimensional reconstruction of a human body, characterized in that: The human body three-dimensional reconstruction method comprises: Acquire a multi-view RGB image of a human body to be photographed; the multi-view RGB image includes: a main-view RGB image and a top-view RGB image; Preprocessing the multi-view RGB image to obtain a preprocessed multi-view RGB image; The preprocessed multi-view RGB image is input into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map; Fusing multiple main-view human body depth prediction images to obtain a fused main-view human body depth prediction image; Based on the fused main-view human body depth prediction map, a three-dimensional reconstruction method is used to perform three-dimensional reconstruction of the human body to obtain a three-dimensional reconstruction result of the human body; the three-dimensional reconstruction result of the human body is a human body surface mesh.

2. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: Obtain a multi-view RGB image of the human body to be photographed, specifically including: A plurality of RGBD cameras are used to surround the human body to be photographed at preset angle intervals to perform real-time synchronous photography of the human body to be photographed, thereby obtaining a multi-view RGB image of the human body to be photographed.

3. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: Preprocessing the multi-view RGB image to obtain a preprocessed multi-view RGB image specifically includes: The contrast and saturation of the multi-view RGB image are adjusted in sequence to obtain an adjusted multi-view RGB image; The artificial blurring method is used to perform low-resolution simulation on the adjusted multi-view RGB image to obtain the preprocessed multi-view RGB image.

4. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The CNN-based deep learning network consists of a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module.

5. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The training process of the multi-view human depth map prediction model specifically includes: Get the training dataset; The preprocessed sample multi-view RGB image is input into the CNN-based deep learning network, and the sample main view human body depth prediction map is output; A loss function is constructed according to the sample main-view human body depth prediction map and the sample main-view human body depth map, and the network parameters of the CNN-based deep learning network are iteratively optimized according to the loss function until the iterative optimization round reaches a maximum value or the loss function reaches a minimum value, and the iterative optimization is stopped to obtain the multi-view human body depth map prediction model.

6. The method for three-dimensional reconstruction of a human body according to claim 1, characterized in that: The three-dimensional reconstruction method is Poisson reconstruction.

7. A human body three-dimensional reconstruction system, characterized in that: The human body 3D reconstruction system is used to implement the human body 3D reconstruction method according to any one of claims 1 to 6, and the human body 3D reconstruction system comprises: An image acquisition unit is used to acquire a multi-view RGB image of a human body to be photographed; the multi-view RGB image includes: a main-view RGB image and a top-view RGB image; A preprocessing unit, used to preprocess the multi-view RGB image to obtain a preprocessed multi-view RGB image; A main-view human body depth prediction map determining unit is used to input the preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map; An image fusion unit, used for fusing a plurality of main-view human body depth prediction images to obtain a fused main-view human body depth prediction image; The human body 3D reconstruction result determination unit is used to perform 3D reconstruction of the human body using a 3D reconstruction method based on the fused main-view human body depth prediction map to obtain a 3D reconstruction result of the human body; the 3D reconstruction result of the human body is a human body surface mesh.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional reconstruction method of a human body according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for three-dimensional reconstruction of a human body according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for three-dimensional reconstruction of a human body according to any one of claims 1 to 6 is implemented.

Images