A human three-dimensional reconstruction method, system, device, medium and product

Abstract

The application discloses a human body three-dimensional reconstruction method, system, device, medium and product, relates to the technical field of computer vision and sparse view three-dimensional reconstruction, and the method comprises the steps of acquiring multi-view RGB images of a human body to be photographed; preprocessing the multi-view RGB images to obtain preprocessed multi-view RGB images; inputting the preprocessed multi-view RGB images into a multi-view human body depth map prediction model to obtain main-view human body depth prediction maps; fusing the multi-view human body depth prediction maps to obtain fused main-view human body depth prediction maps; based on the fused main-view human body depth prediction maps, performing human body three-dimensional reconstruction by adopting a three-dimensional reconstruction method to obtain a human body three-dimensional reconstruction result; and the human body three-dimensional reconstruction result is a human body surface mesh. The application improves the surface smoothness of the human body three-dimensional reconstruction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and sparse view three-dimensional reconstruction, and particularly relates to a human body three-dimensional reconstruction method, system, device, medium and product. BACKGROUND

[0002] With the popularity of mobile computing devices with high-quality image acquisition capabilities and the rapid development of virtual reality technology, people have an unprecedented high demand for creating 3D content, but how to quickly and conveniently create high-precision 3D content is still a difficult problem to be solved. For the popularization of 3D content creation, device cheapness and operation simplification are an inevitable requirement. However, existing methods such as 3D scanning and multi-view stereo algorithm can measure and reconstruct physical objects with high precision, but are limited by high-priced capture devices and rely on complex capture systems with strict environmental restrictions (such as controlled lighting), making it difficult for non-professionals to use. The high use cost also makes people hesitate, which greatly limits the application range of object three-dimensional reconstruction.

[0003] High-precision human body three-dimensional reconstruction is an important research topic in 3D content creation and is one of the key technologies in application fields such as medical imaging, games and VR. Modern film shooting and game production have increasingly relied on this technology. In order to achieve a truly immersive experience, animators need to spend a lot of effort to manually draw virtual characters or use expensive capture systems, which increases the use cost and strictly limits the use range. If human body three-dimensional reconstruction technology can use high-quality image sensors of mobile devices to obtain high-quality digital human body models as simply as taking pictures with a mobile phone, it will have a revolutionary impact on game production and virtual reality technology. In recent years, people have increasingly shifted their research focus to human body three-dimensional reconstruction using deep learning models. Although most of these methods are still far below the level that can be achieved by professional capture systems, they still show great potential. Current 3D reconstruction schemes are designed based on discretized voxels, and there is a problem of rough reconstructed surface, which is far from commercial stage.

[0004] Therefore, there is an urgent need for a human body three-dimensional reconstruction method to solve the above problems. SUMMARY

[0005] The purpose of the present application is to provide a human body three-dimensional reconstruction method, system, device, medium and product, which improves the surface smoothness of human body three-dimensional reconstruction results.

[0006] To achieve the above purpose, the present application provides the following solutions:

[0007] In a first aspect, the present application provides a human body three-dimensional reconstruction method, which comprises:

[0008] obtaining a multi-view RGB image of a human body to be photographed; the multi-view RGB image comprises a main-view RGB image and a top-view RGB image;

[0009] preprocessing the multi-view RGB image to obtain a preprocessed multi-view RGB image;

[0010] inputting the preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set comprises a preprocessed sample multi-view RGB image and a corresponding sample main-view human body depth map;

[0011] fusing a plurality of main-view human body depth prediction maps to obtain a fused main-view human body depth prediction map;

[0012] based on the fused main-view human body depth prediction map, performing human body three-dimensional reconstruction using a three-dimensional reconstruction method to obtain a human body three-dimensional reconstruction result; the human body three-dimensional reconstruction result is a human body surface mesh.

[0013] Optionally, obtaining a multi-view RGB image of a human body to be photographed specifically comprises:

[0014] surrounding the human body to be photographed at a preset angle interval using a plurality of RGBD cameras to perform real-time and synchronous shooting on the human body to be photographed to obtain a multi-view RGB image of the human body to be photographed.

[0015] Optionally, preprocessing the multi-view RGB image to obtain a preprocessed multi-view RGB image specifically comprises:

[0016] adjusting the contrast and saturation of the multi-view RGB image in sequence to obtain an adjusted multi-view RGB image;

[0017] using an artificial blur method to perform low-resolution simulation on the adjusted multi-view RGB image to obtain a preprocessed multi-view RGB image.

[0018] Optionally, the CNN-based deep learning network comprises a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence.

[0019] Optionally, the training process of the multi-view human body depth map prediction model specifically comprises:

[0020] obtaining a training data set;

[0021] input the preprocessed multi-view RGB image of the sample into the deep learning network based on CNN, and output a human body depth prediction map of a main view of the sample;

[0022] A loss function is constructed according to the human body depth prediction map of the main view of the sample and the human body depth map of the main view of the sample, and network parameters of the deep learning network based on CNN are iteratively optimized according to the loss function until the iteration optimization round reaches a maximum value or the loss function reaches a minimum value, and the iteration optimization is stopped, thereby obtaining the multi-view human body depth map prediction model.

[0023] Optionally, the three-dimensional reconstruction method is Poisson reconstruction.

[0024] In a second aspect, the present application provides a human body three-dimensional reconstruction system, which is used to implement the human body three-dimensional reconstruction method, and includes:

[0025] An image acquisition unit is configured to acquire multi-view RGB images of a human body to be photographed, wherein the multi-view RGB images include a main view RGB image and a top view RGB image.

[0026] A preprocessing unit is configured to preprocess the multi-view RGB images to obtain preprocessed multi-view RGB images.

[0027] A human body depth prediction map of a main view determination unit is configured to input the preprocessed multi-view RGB images into a multi-view human body depth map prediction model to obtain a human body depth prediction map of a main view, wherein the multi-view human body depth map prediction model is obtained by training a deep learning network based on CNN using a training data set, and the training data set includes preprocessed multi-view RGB images of samples and corresponding human body depth maps of main views of the samples.

[0028] An image fusion unit is configured to fuse a plurality of human body depth prediction maps of main views to obtain a fused human body depth prediction map of a main view.

[0029] A human body three-dimensional reconstruction result determination unit is configured to perform human body three-dimensional reconstruction using a three-dimensional reconstruction method based on the fused human body depth prediction map of a main view to obtain a human body three-dimensional reconstruction result, wherein the human body three-dimensional reconstruction result is a human body surface mesh.

[0030] In a third aspect, the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the human body three-dimensional reconstruction method of any one of the above aspects.

[0031] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the human three-dimensional reconstruction method of any one of the above aspects.

[0032] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the human three-dimensional reconstruction method of any one of the above aspects.

[0033] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0034] The present application discloses a human three-dimensional reconstruction method, system, device, medium and product. The pre-processed multi-view RGB image is input into a multi-view human depth map prediction model to obtain a main-view human depth prediction map. A plurality of main-view human depth prediction maps are fused to obtain a fused main-view human depth prediction map. Based on the fused main-view human depth prediction map, a three-dimensional reconstruction method is used to perform human three-dimensional reconstruction to obtain a human three-dimensional reconstruction result. The main-view human depth prediction map predicted by the multi-view human depth map prediction model has a smoother surface depth, thereby improving the smoothness of subsequent human three-dimensional reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Figure 1 A human three-dimensional reconstruction method flowchart is provided for an embodiment of the present application.

[0037] Figure 2 A training process diagram of a CNN-based deep learning network is provided for an embodiment of the present application.

[0038] Figure 3 A functional module diagram of a human three-dimensional reconstruction system is provided for an embodiment of the present application.

[0039] Figure 4 A structural diagram of a computer device is provided for an embodiment of the present application.

[0040] REFERENCE NUMERALS:

[0041] Image acquisition unit-1, preprocessing unit-2, main-view human depth prediction map determination unit-3, image fusion unit-4, human three-dimensional reconstruction result determination unit 5. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0043] In order to make the above objectives, characteristics and advantages of the present application more apparent, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0044] In one exemplary embodiment, as shown in FIG. 1, a human three-dimensional reconstruction method is provided, including the following steps S1 to S5. Wherein: Figure 1

[0045] Step S1, acquiring multi-view RGB images of a human to be photographed; the multi-view RGB images include a main-view RGB image and a top-view RGB image.

[0046] As an optional implementation, step S1 specifically includes:

[0047] A plurality of RGBD cameras are used to surround the human to be photographed at a preset angle interval, to perform real-time synchronous photographing on the human to be photographed, so as to obtain the multi-view RGB images of the human to be photographed. Specifically, eight RGBD cameras are used to uniformly surround the human to be photographed at an interval of 45 degrees, face the center, and perform real-time synchronous photographing in a hardware synchronization manner.

[0048] Step S2, pre-processing the multi-view RGB images to obtain pre-processed multi-view RGB images.

[0049] As an optional implementation, step S2 specifically includes:

[0050] Step S21, adjusting the contrast and saturation of the multi-view RGB images in sequence to obtain adjusted multi-view RGB images.

[0051] Step S22, using an artificial blur method to perform low-resolution simulation on the adjusted multi-view RGB images to obtain pre-processed multi-view RGB images. The simulation is performed on the image conditions of different devices.

[0052] ​Step S3: input the preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set includes the preprocessed sample multi-view RGB image and the corresponding sample main-view human body depth map.

[0053] As an optional implementation, in step S3, the CNN-based deep learning network includes a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence.

[0054] Specifically, if Figure 2 As shown, step 1): using a multi-level image feature encoder with self-attention (i.e. Figure 2 The Deform FPN shown in FIG3 is used to extract high-level semantic features of the input multi-view RGB image (including: high-level semantic features of the main view RGB image and high-level semantic features of the top view RGB image); Step 2): Then, in the loss space with local semantic features, the high-level semantic features of the multi-view RGB image are back-projected into 3D voxels according to the camera parameters (i.e., the internal and external parameters of the camera, the external parameters are the camera pose, and the internal parameters include the camera focal length and distortion coefficient). The variance of the back-projected image features of each voxel at different viewpoints is calculated to obtain the feature differences of different viewpoints, and the variance of the back-projected image features of each voxel at different viewpoints is calculated. The variance of the back-projected image features is combined with the high-level semantic features of the main view image to obtain a loss space with local semantic features (i.e., 3D loss space), which is used to maintain spatial continuity and obtain a smoother surface depth. This solves the problem of different voxel discretization in traditional methods and the obvious noise in the depth map. A depth map with a smooth surface is obtained based on the high-level semantic features; Step 3): On the loss space with local semantic features, under the guidance of the local semantic features, a sparse convolution module is used to perform sparse 3D convolution that maintains the original voxel adjacency relationship to obtain the first-resolution depth map of the main view.

[0055] Figure 2 In the figure, W represents feature deformation, which is used to back-project image features onto the rays emitted by each pixel of the primary view and calculate the variance of image features from different viewpoints. Because the first level uniformly samples the pixel rays, the space is fully covered, and ordinary 3D convolution is used later. F represents voxel feature fusion, which fuses the variances of the primary view features and multi-view features and directly splices them together to obtain a loss space with local semantic features. 3D convolution is then performed. The convolution result is the probability that each point is a surface point. The weighted sum of the point probability and the depth of the point on a ray is the estimated depth of the point. 3D represents 3D convolution; S represents 3D sparse convolution, which only convolves the voxels with values.Figure 2 Right side is three different resolution depth prediction (i.e. Figure 2 Pred Depth in), which uses left side different resolution multi-view image features, three levels are cascaded together; Figure 2 Rightmost in (i.e. GT-One-Hot) represents the MSE Loss, which is used to supervise the probability of each spatial point being a surface point.

[0056] The traditional method is to directly extract the sampling points for 3D convolution, but the relative position relationship of the points is changed. The present application puts the points into the original position in the space, uses sparse convolution, only calculates the points with values, and preserves the relative position relationship of the points. The remaining process is the same as the first level.

[0057] Further, the calculation formula of the variance of each voxel in the image features of different views is as follows:

[0058]

[0059] Where N is the number of views, V i is the image feature of the i-th view, is the average of the image features projected to this point; Var is the variance of the image features projected to the point.

[0060] The calculation formula of the loss space with local semantic features is as follows:

[0061] C = Concate (Var, V s )(2)

[0062] Where C is the loss space with local semantic features; Concate represents concatenation in the channel dimension; Vs is the high-level semantic feature of the main view image.

[0063] Then, the first resolution depth map of the main view is interpolated and up-sampled to obtain a second resolution depth map of the main view, and a second resolution loss space with local semantic features focused on the base depth (i.e. the depth estimated in the previous step) is constructed based on the second resolution depth map of the main view, and sparse 3D convolution is continued to obtain a third resolution depth map of the main view, and then interpolation and up-sampling is continued to obtain a fourth resolution depth map, and then a fourth resolution loss space with local semantic features is constructed, and then sparse 3D convolution is continued to obtain a fifth resolution depth map of the main view, i.e. this step is repeated until a target resolution depth map of the main view is obtained; wherein the target resolution is the resolution of the corresponding sample human body depth map of the main view, and the resolution of the corresponding sample human body depth map of the main view is consistent with the resolution of the input RGB image of the main view, and the first resolution is less than the second resolution, the second resolution is less than the third resolution, the third resolution is less than the fourth resolution, and the fourth resolution is less than the fifth resolution, i.e. compared with the first resolution depth map of the main view, a higher resolution depth map of the main view is obtained through continuous training. Through multi-level resolution depth map prediction, the computational complexity and memory requirement are reduced.

[0064] Specifically, in step 1), a multi-level image feature encoder with self-attention is used to extract high-level semantic features of the input multi-view RGB image, specifically: using deformable convolution, real-time perception is used to obtain the receptive field range and perception direction required by the detailed features of the current feature point, and the shape of the convolution kernel is adjusted in real time to solve the limitation of fixed receptive field of traditional convolution algorithm. Wherein, high-dimensional features are used for convolution operation to predict finer receptive field range and perception direction, obtain finer image features, including local picture features and more extensive semantic features.

[0065] Wherein, in the process of constructing the multi-level image feature encoder with self-attention, the image feature extraction layer with self-attention is embedded into the pyramid feature extractor (i.e. Figure 2 The leftmost deformable convolution pyramid feature extraction network in the pyramid feature extractor is used to extract multi-level image features with semantics, and outputs image features with different resolutions) to extract multi-level image features with semantics, and the image feature layer with self-attention can use higher-dimensional image features to obtain finer receptive field range and perception direction, and the pyramid feature extractor can also use the semantic information obtained by the image feature layer with self-attention to obtain image features with higher information density at multiple levels.

[0066] In step 3), in the second stage and above stages of the multi-resolution depth map prediction, the traditional method subtracts the basic depth when constructing the loss space, which causes the voxels to lose the original adjacency relationship. In the second stage and above stages, the method of sparse convolution is used to maintain the original adjacency relationship of the voxels, and the local high-level semantic features are introduced into each voxel to guide the original 3D convolution, so that a more smooth surface depth is obtained.

[0067] As an optional implementation, in step S3, the training process of the multi-view human body depth map prediction model specifically includes:

[0068] In step S31, a training data set is obtained.

[0069] In the multi-view RGB image acquisition process, the training data set is obtained by using data augmentation: different illuminations are marked on the human body to be photographed, and RGB images are obtained under different illumination conditions, so as to adapt to the acquisition of photos under different illumination conditions in the application process.

[0070] In addition, each pixel of the sample main-view human body depth map and the RGB image is aligned, and the depth map is obtained by a depth camera. The sample main-view human body depth map and the RGB image are synchronously photographed in real time by using a hardware synchronization method.

[0071] In step S32, the preprocessed sample multi-view RGB image is input into the CNN-based deep learning network, and a sample main-view human body depth prediction map is output.

[0072] In step S33, a loss function is constructed according to the sample main-view human body depth prediction map and the sample main-view human body depth map, and the network parameters of the CNN-based deep learning network are iteratively optimized according to the loss function until the maximum number of iterations is reached or the loss function reaches the minimum value, and the iterative optimization is stopped, thereby obtaining the multi-view human body depth map prediction model.

[0073] In step S4, a plurality of main-view human body depth prediction maps are fused to obtain a fused main-view human body depth prediction map. Specifically, the plurality of main-view human body depth prediction maps are obtained by replacing the main-view RGB image with different overhead view RGB images in step S3. The plurality of main-view human body depth prediction maps are fused to obtain a surface smooth and complete mesh, and the single-view depth error interference is reduced.

[0074] In step S5, based on the fused main-view human body depth prediction map, a three-dimensional reconstruction method is used for human body three-dimensional reconstruction, and a human body three-dimensional reconstruction result is obtained; the human body three-dimensional reconstruction result is a human body surface mesh.

[0075] Specifically, the fused main-view human depth prediction map is projected to a 3D space, and a surface smooth human surface mesh is reconstructed by using a Poisson reconstruction method or the like to obtain a human three-dimensional reconstruction result. That is, the fused main-view human depth prediction map is projected to a 3D space according to camera parameters to obtain multi-view point clouds, different view point clouds are fused according to back projection, and point clouds with large back projection errors are removed, and a surface smooth human surface mesh is reconstructed by using a Poisson reconstruction method or the like subsequently.

[0076] The beneficial effects of the present application are as follows:

[0077] In the multi-view human depth map prediction model constructed by the present application, sparse convolution and advanced semantic information injection are used to obtain a human surface mesh with higher precision, smoother surface and better visual effect from a sparse RGB image, thereby solving the problems of high equipment requirement, high labor cost and rough surface of the human three-dimensional reconstruction result of the traditional method. The system can be used to assist in game 3D character modeling, virtual reality digital twin character construction and the like.

[0078] Based on the same inventive concept, the present application also provides a human three-dimensional reconstruction system for implementing the human three-dimensional reconstruction method described above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more human three-dimensional reconstruction system embodiments provided below can be referred to the limitations of the human three-dimensional reconstruction method described above, which will not be described here again.

[0079] In one exemplary embodiment, as shown in Figure 3 A human three-dimensional reconstruction system is provided, which includes an image acquisition unit 1, a preprocessing unit 2, a main-view human depth prediction map determination unit 3, an image fusion unit 4 and a human three-dimensional reconstruction result determination unit 5.

[0080] The image acquisition unit 1 is configured to acquire multi-view RGB images of a human body to be photographed, wherein the multi-view RGB images include a main-view RGB image and a top-view RGB image.

[0081] The preprocessing unit 2 is configured to preprocess the multi-view RGB images to obtain preprocessed multi-view RGB images.

[0082] The main-view human depth prediction map determination unit 3 is configured to input the preprocessed multi-view RGB images into a multi-view human depth map prediction model to obtain a main-view human depth prediction map, wherein the multi-view human depth map prediction model is obtained by training a CNN-based deep learning network using a training data set, and the training data set includes preprocessed sample multi-view RGB images and corresponding sample main-view human depth maps.

[0083] The image fusion unit 4 is configured to fuse the multiple main-view human body depth prediction maps to obtain a fused main-view human body depth prediction map.

[0084] The human three-dimensional reconstruction result determination unit 5 is configured to perform human three-dimensional reconstruction based on the fused main-view human body depth prediction map to obtain a human three-dimensional reconstruction result, wherein the human three-dimensional reconstruction result is a human surface mesh.

[0085] In an exemplary embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the human three-dimensional reconstruction method.

[0086] In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the human three-dimensional reconstruction method.

[0087] In an exemplary embodiment, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement the human three-dimensional reconstruction method.

[0088] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram of the computer device can be as shown in Figure 4 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a human three-dimensional reconstruction method.

[0089] Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0091] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0092] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0093] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0094] The principles and implementations of the present application are described in detail herein with specific examples. The above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method of three-dimensional reconstruction of a human body, characterized in that, The human body three-dimensional reconstruction method comprises: obtaining multi-view RGB images of a human body to be photographed; the multi-view RGB images comprise a main-view RGB image and a top-view RGB image; preprocessing the multi-view RGB images to obtain preprocessed multi-view RGB images; inputting the preprocessed multi-view RGB images into a multi-view human body depth map prediction model to obtain main-view human body depth prediction maps; the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set; the training data set comprises preprocessed sample multi-view RGB images and corresponding sample main-view human body depth maps; fusing a plurality of main-view human body depth prediction maps to obtain fused main-view human body depth prediction maps; performing human body three-dimensional reconstruction based on the fused main-view human body depth prediction maps using a three-dimensional reconstruction method to obtain a human body three-dimensional reconstruction result; the human body three-dimensional reconstruction result is a human body surface mesh; The CNN-based deep learning network comprises a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence; The multi-level image feature encoder with self-attention is used to extract high-level semantic features of the input multi-view RGB images; in the loss space construction layer with local semantic features, the high-level semantic features of the multi-view RGB images are back-projected into 3D voxels according to camera parameters, the variance of the back-projected image features of each voxel at different views is calculated, and the variance of the back-projected image features of each voxel at different views is combined with the high-level semantic features of the main-view image to obtain a loss space with local semantic features; in the loss space with local semantic features, under the guidance of local semantic features, the sparse 3D convolution is performed using the sparse convolution module to maintain the original voxel adjacency relationship, and a first resolution depth map of the main view is obtained; The calculation formula of the variance of the back-projected image features of each voxel at different views is as follows: where N is the number of views, V i is the mean of the image features projected back to the voxel for the i-th view, is the mean of the image features projected back to the voxel; Var is the variance of the back-projected image features projected back to the voxel. The calculation formula of the loss space with local semantic features is as follows: C = Concate(Var, V s ); Wherein, C is the loss space with local semantic features; Concate represents concatenation in the channel dimension; V s is the high-level semantic feature of the main view image; The first resolution depth map of the main view is used for interpolation upsampling to obtain a second resolution depth map of the main view, and a second resolution loss space with local semantic features focusing on the basic depth is constructed based on the second resolution depth map of the main view, and sparse 3D convolution is continued to obtain a third resolution depth map of the main view, and interpolation upsampling is continued to obtain a fourth resolution depth map, a fourth resolution loss space with local semantic features is constructed, sparse 3D convolution is performed to obtain a fifth resolution depth map of the main view, and the operation is repeated until a target resolution depth map of the main view is obtained; wherein the target resolution is the resolution of the corresponding sample main-view human body depth map, the resolution of the corresponding sample main-view human body depth map is consistent with the resolution of the input main-view RGB image, and the first resolution is less than the second resolution, the second resolution is less than the third resolution, the third resolution is less than the fourth resolution, and the fourth resolution is less than the fifth resolution.

2. The method of claim 1, wherein, The multi-view RGB image of the human body to be photographed is obtained, specifically comprising: A plurality of RGBD cameras are used to surround the human body to be photographed at preset angle intervals to perform real-time synchronous shooting on the human body to be photographed, thereby obtaining the multi-view RGB image of the human body to be photographed.

3. The method of claim 1, wherein, The multi-view RGB image is preprocessed to obtain the preprocessed multi-view RGB image, specifically comprising: The contrast and saturation of the multi-view RGB image are adjusted in sequence to obtain the adjusted multi-view RGB image; An artificial blur method is used to simulate low resolution of the adjusted multi-view RGB image to obtain the preprocessed multi-view RGB image.

4. The method of claim 1, wherein, The CNN-based deep learning network comprises a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence.

5. The method of claim 1, wherein, The training process of the multi-view human body depth map prediction model comprises: Obtaining a training data set; The preprocessed sample multi-view RGB image is input into the CNN-based deep learning network, and a sample main-view human body depth prediction map is output; A loss function is constructed according to the sample main-view human body depth prediction map and the sample main-view human body depth map, and the network parameters of the CNN-based deep learning network are iteratively optimized according to the loss function until the iteration optimization round reaches a maximum value or the loss function reaches a minimum value, and the iteration optimization is stopped to obtain the multi-view human body depth map prediction model.

6. The method of claim 1, wherein, The three-dimensional reconstruction method is Poisson reconstruction.

7. A human three-dimensional reconstruction system, characterized by, The human body three-dimensional reconstruction system is used to implement the human body three-dimensional reconstruction method of any one of claims 1-6, and comprises: An image acquisition unit is configured to acquire a multi-view RGB image of a human body to be photographed, wherein the multi-view RGB image comprises a main-view RGB image and a top-view RGB image; A preprocessing unit is configured to preprocess the multi-view RGB image to obtain a preprocessed multi-view RGB image; A main-view human body depth prediction map determination unit is configured to input the preprocessed multi-view RGB image into a multi-view human body depth map prediction model to obtain a main-view human body depth prediction map, wherein the multi-view human body depth map prediction model is obtained by training a CNN-based deep learning network using a training data set, and the training data set comprises preprocessed sample multi-view RGB images and corresponding sample main-view human body depth maps; An image fusion unit is configured to fuse a plurality of main-view human body depth prediction maps to obtain a fused main-view human body depth prediction map; A human body three-dimensional reconstruction result determination unit is configured to perform human body three-dimensional reconstruction based on the fused main-view human body depth prediction map using a three-dimensional reconstruction method to obtain a human body three-dimensional reconstruction result, wherein the human body three-dimensional reconstruction result is a human body surface mesh; The CNN-based deep learning network comprises a multi-level image feature encoder with self-attention, a loss space construction layer with local semantic features, and a sparse convolution module connected in sequence. The high-level semantic features of the input multi-view RGB image are extracted by using a multi-level image feature encoder with self-attention; the high-level semantic features of the multi-view RGB image are back-projected into 3D voxels according to camera parameters in a loss space construction layer with local semantic features, the variance of the back-projected image features of each voxel in different views is calculated, and the variance of the back-projected image features of each voxel in different views is combined with the high-level semantic features of the main view image to obtain a loss space with local semantic features; in the loss space with local semantic features, under the guidance of local semantic features, sparse 3D convolution is performed by using a sparse convolution module to maintain the original voxel adjacency relationship, to obtain a first resolution depth map of the main view; The calculation formula of the variance of the back-projected image features of each voxel in different views is as follows: where N is the number of views, V i is the mean of the image features of the i-th view, is the mean of the image features of the i-th view, The calculation formula of the loss space with local semantic features is as follows: C = Concate(Var, V s ); Wherein, C is the loss space with local semantic features; Concate represents concatenation in the channel dimension; V s is the high-level semantic feature of the main view image; The first resolution depth map of the main view is interpolated and up-sampled to obtain a second resolution depth map of the main view, and a loss space with local semantic features focused on the basic depth near the second resolution is constructed based on the second resolution depth map of the main view, and sparse 3D convolution is continued to obtain a third resolution depth map of the main view, and interpolation up-sampling is continued to obtain a fourth resolution depth map, a loss space with local semantic features corresponding to the fourth resolution is constructed, sparse 3D convolution is continued, and a fifth resolution depth map of the main view is obtained, and the operation is repeated until a target resolution depth map of the main view is obtained. The target resolution is the resolution of the corresponding sample main view human body depth map, the resolution of the corresponding sample main view human body depth map is consistent with the resolution of the input main view RGB image, and the first resolution is less than the second resolution, the second resolution is less than the third resolution, the third resolution is less than the fourth resolution, and the fourth resolution is less than the fifth resolution.

8. A computer device comprising: The memory, the processor, and the computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the human body three-dimensional reconstruction method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the human body three-dimensional reconstruction method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the human body three-dimensional reconstruction method of any one of claims 1-6.

Images