Three-dimensional reconstruction method and device, storage medium, program product and computer equipment

Abstract

The invention discloses a three-dimensional reconstruction method and device, a storage medium, a program product and computer equipment, and the method comprises the steps: determining a corresponding rear view image on the basis of collecting a front view image, i.e., not needing to collect images at multiple angles of view, thereby improving the efficiency of three-dimensional reconstruction, improving the application range of the three-dimensional reconstruction, and reducing the cost of the three-dimensional reconstruction. The image features and the normal features of all the images can be used as prior constraints to constrain reconstruction, and the depth implicit function corresponding to the object is determined in combination with random sampling points of the object in a three-dimensional space, so that the depth implicit function is used for assisting three-dimensional reconstruction processing of the front view image and the rear view image; and the precision of three-dimensional reconstruction is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a three-dimensional reconstruction method, apparatus, storage medium, program product, and computer equipment. Background Technology

[0002] 3D reconstruction technology aims to reconstruct three-dimensional geometric models from two-dimensional image information. Compared to two-dimensional images, 3D models contain more parametric information, resulting in a greater sense of depth and thus a more visually impactful experience for users. Compared to manual modeling, 3D reconstruction technology can significantly reduce costs and supports real-time modeling.

[0003] In related technologies, multi-view 3D reconstruction methods can be used. For example, by calibrating and aligning the acquired multi-view 2D images, object feature points can be extracted from the aligned 2D images, and then the object feature points can be converted into 3D spatial feature points using computer vision algorithms.

[0004] However, multi-view 3D reconstruction technology requires capturing multiple 2D images of the object from different angles. This is time-consuming and labor-intensive, resulting in low reconstruction efficiency. Furthermore, the acquisition angles are difficult to control precisely, leading to inaccurate key point features and consequently low reconstruction accuracy. In addition, in many application scenarios, it is difficult to acquire multi-view images of the object in real time, making 3D reconstruction impossible and limiting its applicability. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a three-dimensional reconstruction method, apparatus, storage medium, program product, and computer equipment, which can improve the efficiency, accuracy, and applicability of three-dimensional reconstruction.

[0006] In a first aspect, embodiments of this application provide a three-dimensional reconstruction method, including: Acquire a front view image, wherein the front view image corresponds to the object to be reconstructed; Determine the rear view image corresponding to the front view image; Determine the image features of the front view image and the rear view image respectively, and determine the normal features of the front view image and the rear view image respectively; Based on the image features, the normal features, and the random sampling points of the object in three-dimensional space, the implicit depth function corresponding to the object is determined; Based on the front view image, the rear view image, and the implicit depth function, a 3D reconstruction process is performed to generate a 3D model corresponding to the object.

[0007] Optionally, determining the rear view image corresponding to the front view image includes: Determine the two-dimensional contour information corresponding to the front view image; The front view image and the two-dimensional contour information are input into an image conversion network to obtain the rear view image output by the image conversion network, wherein the image conversion network is constructed based on a conditional generative adversarial algorithm.

[0008] Optionally, determining the depth latent function corresponding to the object based on the image features, the normal features, and random sampling points of the object in three-dimensional space includes: Gaussian perturbation is added to the random sampling points to obtain the target sampling points; The image features and the normal features are used as prior constraints for the depth implicit function, and the depth implicit function is constructed based on the target sampling points.

[0009] Optionally, the three-dimensional reconstruction process includes surface reconstruction and texture domain reconstruction; The step of performing 3D reconstruction processing based on the front view image, the rear view image, and the implicit depth function to generate a 3D model corresponding to the object includes: Based on the aforementioned depth implicit function, the multilayer perceptron is invoked to obtain the volume occupancy probability of the three-dimensional points corresponding to the object. Surface reconstruction is performed based on the three-dimensional point volume occupancy probability to generate an initial three-dimensional model corresponding to the object; The texture domain is reconstructed based on the front view image, the rear view image, and the initial 3D model to generate the 3D model.

[0010] Optionally, the step of reconstructing the texture domain based on the front view image, the rear view image, and the initial 3D model to generate the 3D model includes: Determine each grid face of the object; Based on the normal vector corresponding to each of the grid faces and the camera ray vector corresponding to the front view image, the view corresponding to each of the grid faces is determined; Based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, the texture domain of each mesh surface is reconstructed to generate the three-dimensional model.

[0011] Optionally, the view corresponding to each of the grid surfaces is a front view, a rear view, or a boundary view; The process of reconstructing the texture domain of each mesh surface based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, includes: For each of the aforementioned mesh faces When the corresponding view is the front view, the texture domain of the mesh surface is reconstructed based on the front view image; When the corresponding view is a rear view, the texture domain of the mesh surface is reconstructed based on the rear view image; When the corresponding view is a boundary view, the texture domain is reconstructed for the mesh surface based on the front view image and the rear view image.

[0012] Secondly, embodiments of this application provide a three-dimensional reconstruction apparatus, comprising: A front view acquisition module is used to acquire a front view image, wherein the front view image corresponds to the object to be reconstructed; The rear view determination module is used to determine the rear view image corresponding to the front view image; The feature determination module is used to determine the image features of the front view image and the rear view image respectively, and to determine the normal features of the front view image and the rear view image respectively; The implicit function calculation module is used to determine the depth implicit function corresponding to the object based on the image features, the normal features, and random sampling points of the object in three-dimensional space. The reconstruction module is used to perform three-dimensional reconstruction processing based on the front view image, the rear view image and the depth implicit function to generate a three-dimensional model corresponding to the object.

[0013] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0014] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0015] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0016] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, the corresponding rear view image can be determined based on the acquired front view image, i.e., it is not required to acquire images from multiple perspectives, thereby improving the efficiency and applicability of 3D reconstruction. Furthermore, the image features and normal features of each image can be used as prior constraints to constrain the reconstruction, and the implicit depth function corresponding to the object can be determined by combining the random sampling points of the object in 3D space. This implicit depth function can then be used to assist the front view image and the rear view image in 3D reconstruction processing, thereby improving the accuracy of 3D reconstruction. Attached Figure Description

[0017] Figure 1 This is a schematic flowchart of the three-dimensional reconstruction method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the image conversion network provided in an embodiment of this application; Figure 3 This is a schematic diagram of the discriminator in the image conversion network provided in the embodiments of this application; Figure 4 This is a schematic diagram of the three-dimensional reconstruction provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the three-dimensional reconstruction device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments / examples are only a part of the embodiments / examples of this application, and not all of the embodiments / examples. Based on the embodiments / examples in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment / example" means "at least one embodiment / example"; the term "another embodiment / example" means "at least one additional embodiment / example"; the term "some embodiments / examples" means "at least some embodiments / examples."

[0020] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] In related technologies, single-view Figure 3 Reconstruction schemes can be categorized into the following three types.

[0023] 1. Parametric 3D Reconstruction.

[0024] By collecting a large amount of object structure data and learning its statistical distribution, the regularization of 3D shape surfaces can be mapped to a low-dimensional parameter domain. By decomposing the statistical model into several sets of low-dimensional parameter vectors, the shape of an object can be described based on low-dimensional feature embeddings in the deformation space. Parametric 3D reconstruction methods are generally suitable for reconstructing objects of a given shape category, such as faces and bodies. Human body parametric models can include SCAPE, SMPL, SMPL-X, etc., each providing a baseline human body structure. Human body shape parameters and human pose parameters are used to control human body deformation, transforming the 3D human body reconstruction problem into a problem of predicting human body parameters.

[0025] 2. 3D reconstruction based on point clouds.

[0026] 3D shapes can be represented by an unordered set of 3D points. This point-based representation is relatively simple and generally suitable for representing objects with fine details, but it has high memory requirements. Furthermore, the irregular nature of point cloud models allows point cloud-based 3D reconstruction schemes to handle various shapes and structures more flexibly, making them more widely applicable. Through preprocessing, registration, and model optimization of point cloud data, high-quality 3D models can be generated.

[0027] 3. 3D reconstruction based on implicit functions.

[0028] The implicit function-based 3D reconstruction scheme constructs a probability field to predict whether a target point is inside or outside a continuous space. Isosurfaces with a probability value of 0.5 are extracted from this probability field to synthesize the surface of a 3D model. This scheme uses a convolutional neural network to establish a classification relationship between image features extracted from a 2D image and 3D labels. The implicit function model trained based on this relationship can generalize to various objects or scenes, exhibiting strong adaptability and generalization capabilities.

[0029] However, single-view technology in related technologies Figure 3 The difficulties in 3D reconstruction mainly come from two aspects. First, the single view results in a lack of information such as image depth, body shape, and back details, making it difficult to obtain accurate physical parameters. Second, different objects have different morphological characteristics, making it difficult to use a universal baseline model as a remodeling template.

[0030] Parametric reconstruction schemes can establish a spatial mapping between 2D images and low-dimensional parameter vectors by regularizing the solution space, resulting in a relatively reliable object shape. Their statistical properties can, to some extent, offset the depth ambiguity caused by monocular images. However, this reconstruction scheme is generally only suitable for reconstructing a specific type of object; for example, using a parametric human body model to reconstruct a 3D human body. Furthermore, the statistical distribution of the parametric model only satisfies the low-dimensional assumption. Irregular surfaces and other shapes of objects inherently belong to a high-dimensional space and cannot be simply represented by low-dimensional parameters. Therefore, parametric models cannot be directly used to reconstruct objects with irregular shapes.

[0031] Point cloud-based reconstruction schemes capture the shape and appearance of objects by using a series of points distributed in three-dimensional space. Each point in a point cloud carries not only its spatial coordinates but also attributes such as color and intensity, thus accurately describing the three-dimensional data structure of an object's surface or scene. However, point cloud data often contains noise, requiring noise filtering algorithms to remove these noise points, which makes it difficult to guarantee the efficiency and accuracy of point cloud data processing. Furthermore, large-scale dense point clouds require processing a large number of data points, therefore point cloud-based reconstruction schemes are typically highly complex, computationally intensive, and place high demands on hardware.

[0032] Implicit function-based reconstruction schemes can use a scalar field to define the isosurface of a 3D mesh, seamlessly fusing global features learned from coarse inference with image features learned from high-resolution input images. This not only results in high reconstruction efficiency but also the ability to reconstruct high-fidelity 3D geometry, including surface roughness and wrinkles. However, due to the lack of prior constraints, this reconstruction scheme is prone to artifacts when reconstructing images with occlusion, making it difficult to achieve a balance between reconstruction efficiency and accuracy.

[0033] In view of the above, embodiments of this application provide a three-dimensional reconstruction method, apparatus, storage medium, program product, and computer equipment, which aim to at least partially solve the shortcomings existing in the above-mentioned related technologies.

[0034] Firstly, see [the following] Figure 1 The diagram shows a flowchart of a three-dimensional reconstruction method provided in an embodiment of this application. The three-dimensional reconstruction method can be applied to a computer device with data processing capabilities. The method includes S101-S105, as detailed below.

[0035] S101, Obtain a front view image, wherein the front view image corresponds to the object to be reconstructed.

[0036] In some examples, the front view image may be obtained by taking a picture of the object to be reconstructed. In this case, the shooting angle of the front view image can be used as the front view angle of the object.

[0037] In some examples, the front view image may include a single image containing the complete frontal outline of the object, and / or multiple images containing the frontal outline of the object that can cover the complete frontal outline.

[0038] S102, determine the rear view image corresponding to the front view image.

[0039] In some examples, a conditional generative adversarial network can be used to generate a rear view image corresponding to the front view image.

[0040] S103, determine the image features of the front view image and the rear view image respectively, and determine the normal features of the front view image and the rear view image respectively.

[0041] In some examples, the front view image and the rear view image can be input into an image feature extraction network separately to obtain the image features of the front view image and the rear view image output by the image feature extraction network. The image feature extraction network can be a ResNet-50 network.

[0042] In some examples, normal features can refer to the surface normal map of an image.

[0043] In some examples, the front view image and the rear view image can be input into the image encoder separately to obtain the image features of the front view image and the rear view image output by the image encoder.

[0044] S104, Based on the image features, the normal features, and the random sampling points of the object in three-dimensional space, determine the implicit depth function corresponding to the object.

[0045] In some examples, the image features and normal features of the front view image and the rear view image can be used as constraints of the implicit function, and then the implicit function can be used to calculate the corresponding depth implicit function based on the random sampling point.

[0046] S105, perform 3D reconstruction processing based on the front view image, the rear view image and the depth implicit function to generate a 3D model corresponding to the object.

[0047] In some examples, this implicit depth function can be used to correlate the pixels of the front and rear view images with the depth of the three-dimensional points in space. The front and rear view images can be used to provide texture information of the object. Therefore, combining them can achieve accurate three-dimensional reconstruction.

[0048] In this embodiment, the corresponding rear view image can be determined based on the acquired front view image, thus eliminating the need for multi-view image acquisition. This improves the efficiency and applicability of 3D reconstruction. Furthermore, the implicit depth function corresponding to the object can be determined based on the image features and normal features of each image, combined with random sampling points of the object in 3D space. This function assists in 3D reconstruction processing of the front and rear view images, improving the accuracy of 3D reconstruction. In addition, this embodiment can also reconstruct general objects from images from a single viewpoint, reducing reconstruction errors caused by information loss in single-view images and preserving the complete object structure.

[0049] In one optional implementation, determining the rear view image corresponding to the front view image includes: Determine the two-dimensional contour information corresponding to the front view image; The front view image and the two-dimensional contour information are input into an image conversion network to obtain the rear view image output by the image conversion network, wherein the image conversion network is constructed based on a conditional generative adversarial algorithm.

[0050] In some cases, it's easy to understand that predicting the precise geometry of the back face of an object under single-view conditions can be considered an ill-posed problem, since the back face image cannot be directly observed from the input single view. Since the front and back views of a typical object generally share the same contour, and many visual features are spatially aligned, this embodiment can generate the corresponding back view image from the input front view image using an image transformation network to improve reconstruction quality and fidelity. Furthermore, normal features can be extracted from the front and back view images of the object respectively as additional input to the image transformation network for its training.

[0051] In some examples, the image transformation network can be a pix2pixHD network built on a conditional generative adversarial algorithm. The pix2pixHD network is an improved version of the conditional generative adversarial network, adding extra conditional parameters to the inputs of the generator and discriminator. This ensures that the image generated by the generator can only pass the discriminator if it is sufficiently realistic and conforms to the conditions. Therefore, it can guide the network to generate images that meet the conditions, rather than simply generating random images.

[0052] For image conversion tasks, there can be many shared information components between the input and output images. Inspired by the characteristics of human vision, when humans observe the world, if part of the observed target is obscured by other objects, the human brain can infer the overall outline of the target based on the partial target information seen by the human eye.

[0053] Therefore, in some examples, this embodiment can use the two-dimensional outline of an object as a semantic label to instruct the image transformation network to infer the corresponding rear view image from the front view image of the object.

[0054] In addition, in other examples, semantic segmentation algorithms can be used to first convert the front view image of the object to the semantic label domain and then convert it back to the image domain, thereby completing the synthesis of the front view image.

[0055] In some examples, see Figure 2 The diagram shows a schematic of an image conversion network provided in an embodiment of this application. The overall structure of the network may include a generator G and a discriminator D.

[0056] The generator G can include a multi-scale generator composed of two residual networks, G1 and G2. These two residual networks can be structurally identical, employing a coarse-to-fine design strategy. G1 is a low-resolution image generation module and also a global generator, primarily consisting of a convolutional front-end, a set of residual blocks, and a transposed convolutional back-end. The generator G can first downsample the input image to obtain lower-resolution image data, using this low-resolution image data to train the G1 network. Then, a G2 network with the same structure is appended to the G1 network. The G2 network acts as a local enhancer, its input being the features extracted by the G2 network and the output of the last stage of the G1 network. The advantage of this coarse-to-fine generator structure is that the downsampled low-resolution image reduces the impact of noise in the original image, while the pixel grayscale values ​​in the residual image are usually concentrated, allowing the use of short codes to characterize these recurring values, thereby improving coding efficiency.

[0057] See also Figure 3 The diagram illustrates a discriminator D in an image conversion network provided in this application embodiment. Discriminator D may include a multi-scale discriminator, which may include three sub-discriminators with the same network structure but operating at different image scales (original image scale, image obtained by downsampling the original image scale by 2x, and image obtained by downsampling the original image scale by 4x). , , The input to the discriminator D can include the real image (front view image), the two-dimensional contour information of the real image (such as a two-dimensional contour map), and the image generated by the generator G. By downsampling the real image and the generated image by 2x and 4x respectively, these three sub-discriminators can be trained at three different scales, thereby achieving the ability to distinguish between real and generated images at three different scales. The sub-discriminator operating at the coarse scale has a more global receptive field, thus guiding the generator to generate globally consistent images, while the sub-discriminator operating at the fine scale can guide the generator to generate finer details, thereby further improving the image generation capability of the generator model.

[0058] Thus, the adversarial loss of this image translation network can be expressed by the following formula:

[0059] in, middle Time represents , , , Indicate that the generator G and GAN adversarial losses between them The weight coefficients representing the feature matching loss (a hyperparameter that can be used to adjust the proportion of feature matching loss in the total loss) The feature matching loss term (which can be used to make the generator G generate images, in...) represents the loss term for the feature matching loss term. In the intermediate feature layer, the features are made as close as possible to those of the real image to improve the detail fidelity of the generated image. That is, the feature matching loss term can be obtained from the image generated by generator G. (Determined by the difference between the features on the intermediate feature layer and the features of the real image). This represents the weighting coefficient of the VGG perceptual loss (a hyperparameter that can be used to adjust the proportion of perceptual loss). It can be represented as the VGG perceptual loss term (which can extract the VGG high-level semantic features of the real image and the generated image through the pre-trained VGG network, and make the VGG high-level semantic features of the two more similar to improve the global visual consistency of the generated image, that is, the VGG perceptual loss term is determined by the difference between the VGG high-level semantic features of the two).

[0060] It is understandable that, in this embodiment, after generating the rear view image of the object, normal features can be further extracted from the front and rear view images. The normal features obtained from RGB color mapping can also play a certain constraining role in the 3D reconstruction process. Using normal mapping to guide the inference of 3D geometry can enable the multilayer perceptron to output more accurate high-frequency features, and can also generate reliable outputs for the back of the object that cannot be directly observed in a single view.

[0061] In one optional implementation, determining the implicit depth function corresponding to the object based on the image features, the normal features, and random sampling points of the object in three-dimensional space includes: Gaussian perturbation is added to the random sampling points to obtain the target sampling points; The image features and the normal features are used as prior constraints for the depth implicit function, and the depth implicit function is constructed based on the target sampling points.

[0062] In some cases, since most vertices of general objects in space do not lie on isosurfaces, uniform sampling can easily result in too few samples, necessitating unnecessary adjustments to the weighting coefficients. Conversely, adaptive sampling directly on the object's surface can easily lead to severe overfitting. To avoid these two phenomena, this embodiment first randomly samples the object's surface in three-dimensional space to obtain random sampling points, and then adds a Gaussian perturbation to these random sampling points. For example, this Gaussian perturbation can be a normally distributed... N(0, σ) offset (e.g.) σ=5cm Here, because the sampling density of the Gaussian sphere is large near high curvature, the target sampling points obtained through this Gaussian perturbation can conform to the surface curvature of the model, significantly improving fidelity.

[0063] Understandably, the depth implicit function is an efficient representation of a three-dimensional surface in space, defining the surface using an occupancy probability function. f(p) The level set of the function represents whether a point in 3D space is inside or outside an object. Depth implicit functions do not require explicit storage of feature embeddings on the object's surface, making them more efficient than traditional voxel or point cloud representations.

[0064] To represent the surface of a specific object using a neural network, a deep implicit function can use a conditional variable along with vertex coordinates as input. For example, in the PIFu framework, image features and point coordinates are used as inputs to the deep implicit function. S(p) Thus, the deep implicit function is defined by the following formula.

[0065]

[0066]

[0067] in, E I To extract image features from the input image using an image encoder, π(p) A sampling point in space p Projection onto a two-dimensional feature map (i.e., image features), φ(.) To use bilinear interpolation pairs π(p) The sampling function for feature sampling at the location, Z(p) For point p Depth value in three-dimensional space.

[0068] The PIFu framework projects each pixel of an image into space as a ray, assuming that all 3D points on the same ray share the same image features. Therefore, the spatial features learned using implicit functions are pixel-aligned. However, reconstructing object structures directly based on image features is fraught with uncertainty, especially for parts that are difficult to observe directly from the input image. This often leads to reconstruction artifacts in the PIFu framework's reconstruction results. Furthermore, the PIFu framework is customized for the human body, thus generally supporting the reconstruction of human models but difficult to generalize to ordinary objects.

[0069] This embodiment can acquire image features from the front and rear view images of an object, as well as the normal features from the front view image. F n Normal features of the rear view image B n This serves as a priori constraint for the deep implicit function, allowing for further rewriting of the input to the deep implicit function. S(p) .

[0070]

[0071] For a 3D model represented by a 3D mesh, the orientation of each mesh face is determined by the normal of that face. Therefore, the normal map can serve as a geometric proxy for the 3D model in space. Providing the normal map as a local feature to the multilayer perceptron model makes it easier for the multilayer perceptron to perceive local details, and the generated 3D model can better represent the roughness of the object's surface. Overly smooth surfaces cannot accurately represent the object's local features. Furthermore, predicting the back view details of an object based solely on the features of the input front view image has certain limitations. The back view image of the object generated by a conditional generative adversarial network provides the image features of the object's back side and the corresponding normal map, providing more reliable constraints on the hidden 3D geometry. This allows the network model in this embodiment to possess good multi-view reasoning capabilities even under single-view conditions.

[0072] The prediction of the occupancy probability of each grid point using a deep implicit function can be viewed as a binary classification task, where the sample label is defined as 0 and 1, where 0 represents that the vertex is outside the 3D grid and 1 represents that the vertex is inside the 3D grid. Therefore, the loss function applied to this task can be extended from the binary cross-entropy loss commonly used in binary classification tasks. Compared to L1 loss and L2 loss, cross-entropy loss is easier to converge and optimize in binary classification tasks. Therefore, in this embodiment, binary cross-entropy loss can be used during the training process of the deep implicit function model to reflect the difference between the predicted value and the true value.

[0073]

[0074] in, δ To participate in the loss function The training sample set is calculated. λ To balance the loss between inside and outside the mesh (this can be set as the ratio of the number of vertices inside and outside the mesh face), a weight parameter is used. This represents the true label of the 3D point p. This represents the prediction of the occupancy probability of each grid point by the depth implicit function.

[0075] In one optional implementation, the 3D reconstruction process includes surface reconstruction and texture domain reconstruction; The step of performing 3D reconstruction processing based on the front view image, the rear view image, and the implicit depth function to generate a 3D model corresponding to the object includes: Based on the aforementioned depth implicit function, the multilayer perceptron is invoked to obtain the volume occupancy probability of the three-dimensional points corresponding to the object. Surface reconstruction is performed based on the three-dimensional point volume occupancy probability to generate an initial three-dimensional model corresponding to the object; The texture domain is reconstructed based on the front view image, the rear view image, and the initial 3D model to generate the 3D model.

[0076] In some examples, after obtaining the 3D point volume occupancy probability, the Marching Cubes algorithm can be used to process the 3D point volume occupancy probability to extract the required mesh surface from the discrete data field (3D point volume occupancy probability), thereby completing the surface reconstruction and generating the initial 3D model.

[0077] In some examples, see Figure 4 The diagram illustrates a three-dimensional reconstruction provided in an embodiment of this application. This multilayer perceptron can convert the depth implicit function into the three-dimensional point volume occupancy probability.

[0078] In one optional implementation, the step of reconstructing the texture domain based on the front view image, the rear view image, and the initial 3D model to generate the 3D model includes: Determine each grid face of the object; Based on the normal vector corresponding to each of the grid faces and the camera ray vector corresponding to the front view image, the view corresponding to each of the grid faces is determined; Based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, the texture domain of each mesh surface is reconstructed to generate the three-dimensional model.

[0079] Understandably, after completing the surface reconstruction of the 3D model, the final step could be to reconstruct the texture domain of the 3D model. Texture domain reconstruction can be achieved by predicting an RGB vector field on the surface geometry of the reconstructed model, defining the surface texture as a vector function in the surface space. In this embodiment, the rear view image of the object is generated during the front and rear view normal inference, and the normal features of the front and rear view images are extracted respectively. Therefore, the color texture of each vertex of the 3D mesh can be inferred under any topology without needing to use the albedo color space to predict the model texture.

[0080] In some examples, image textures that can be directly traced in the front and rear view images can be sampled directly from the front and rear view images. Assume the camera ray vector is... c The normal vector corresponding to any mesh surface is n Then we can determine the dot product of the two vectors. c·n To determine the view (image view) corresponding to the grid vertex in the grid face.

[0081] In one alternative implementation, the view corresponding to each of the grid surfaces is a front view, a rear view, or a boundary view; The process of reconstructing the texture domain of each mesh surface based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, includes: For each of the aforementioned mesh faces When the corresponding view is the front view, the texture domain of the mesh surface is reconstructed based on the front view image; When the corresponding view is a rear view, the texture domain of the mesh surface is reconstructed based on the rear view image; When the corresponding view is a boundary view, the texture domain is reconstructed for the mesh surface based on the front view image and the rear view image.

[0082] In some examples, if the inner product value is positive, it means that the direction of the mesh face is the same as the direction of the camera ray, and the corresponding mesh vertex is located in the rear view of the object, requiring RGB color sampling from the rear view image; if the inner product value is negative, it means that the direction of the mesh face is opposite to the direction of the camera ray, and the mesh vertex is located in the front view of the object, requiring RGB color sampling from the front view image; if the inner product value is 0, it means that the surface direction is perpendicular to the direction of the camera ray, belonging to the boundary view, in which case the RGB color sampled from the front view image and the rear view image can be used as the color of the mesh vertex to prevent sharp boundaries from appearing on the reconstructed model surface.

[0083] It is understandable that in a standard spatial 3D reconstruction, there may be object surfaces that are not observable in either the front or rear view images. Forcibly fitting the output of occluded surfaces to the ground truth and requiring them to output completely identical color spaces can easily reduce the fitting rate and lead to overfitting. Therefore, a better approach to texture domain reconstruction in this embodiment is to allow the occluded area to output a texture similar to the ground truth based on the semantic relationship between the occluded area and its adjacent areas on the object, rather than requiring the output to be completely identical to the ground truth. For rays that cannot be directly traced from the front and rear view images, this embodiment can use the same method as the PIFu framework, employing a depth implicit function to interpolate the texture features at each vertex, thereby filling in the missing texture.

[0084] In this embodiment, considering that regressing a 3D model from a single 2D image is inherently an ill-posed problem—because the same 2D mask can be mapped from multiple 3D objects of different shapes from different angles—this embodiment can predict and generate the rear view image of an object from its front view image using an adversarial learning algorithm. Normal features and image features are extracted from the front and rear view images, respectively, and these extracted features are used as additional parameters of the depth latent function to improve the 3D reconstruction from a single view. Thus, this embodiment can, to some extent, offset the viewpoint limitations caused by a single view and reduce the probability of reconstruction artifacts in the reconstruction results. Furthermore, the trained latent function allows this embodiment to be applied to the reconstruction of any general object, while the normal features of the front and rear view images enable it to reconstruct a more accurate object surface. Moreover, this embodiment can infer the color texture of each vertex of the 3D mesh using the RGB colors and normal maps of the front and rear views without learning complex spatial features such as compensation light from the input image. This refines the texture domain reconstruction based on the spatial features of the image, and training can be completed on a general dataset.

[0085] In this embodiment, predicted rear view image features and normal features of the object's front and rear view images can be used as prior parameters to regulate the reconstruction process. Therefore, the reconstruction result is closer to the actual ground result, and the surface details are richer. Furthermore, the network structure for keypoint acquisition and texture domain reconstruction is lightweight, without overly complex structural operations. The model can be trained in general experimental environments and on general datasets. It can generate a corresponding 3D model by requiring only a single input RGB image containing the complete object from the front. It can be easily applied to mobile devices and other platforms, making it easy for general users to learn, even without any professional knowledge. It eliminates the need to purchase expensive equipment such as multi-view cameras and 3D scanners; images from any camera can be used to reconstruct the 3D model, saving hardware and data processing costs. Additionally, it can quickly perform 3D reconstruction based on a single view image, exhibiting high real-time performance, which can meet the needs of many scenarios requiring immediate feedback and processing, such as virtual try-on and augmented reality.

[0086] Secondly, correspondingly, this application also provides a three-dimensional reconstruction apparatus capable of implementing all the processes of the three-dimensional reconstruction method provided in the above embodiments.

[0087] See Figure 5 The diagram shows a structural schematic of a three-dimensional reconstruction device 500 provided in an embodiment of this application. The three-dimensional reconstruction device 500 includes: A front view acquisition module 501 is used to acquire a front view image, wherein the front view image corresponds to the object to be reconstructed; The rear view determination module 502 is used to determine the rear view image corresponding to the front view image; The feature determination module 503 is used to determine the image features of the front view image and the rear view image respectively, and to determine the normal features of the front view image and the rear view image respectively; The implicit function calculation module 504 is used to determine the depth implicit function corresponding to the object based on the image features, the normal features, and the random sampling points of the object in three-dimensional space. The reconstruction module 505 is used to perform three-dimensional reconstruction processing based on the front view image, the rear view image and the depth implicit function to generate a three-dimensional model corresponding to the object.

[0088] In one optional implementation, determining the rear view image corresponding to the front view image includes: Determine the two-dimensional contour information corresponding to the front view image; The front view image and the two-dimensional contour information are input into an image conversion network to obtain the rear view image output by the image conversion network, wherein the image conversion network is constructed based on a conditional generative adversarial algorithm.

[0089] In one optional implementation, determining the implicit depth function corresponding to the object based on the image features, the normal features, and random sampling points of the object in three-dimensional space includes: Gaussian perturbation is added to the random sampling points to obtain the target sampling points; The image features and the normal features are used as prior constraints for the depth implicit function, and the depth implicit function is constructed based on the target sampling points.

[0090] In one optional implementation, the 3D reconstruction process includes surface reconstruction and texture domain reconstruction; The step of performing 3D reconstruction processing based on the front view image, the rear view image, and the implicit depth function to generate a 3D model corresponding to the object includes: Based on the aforementioned depth implicit function, the multilayer perceptron is invoked to obtain the volume occupancy probability of the three-dimensional points corresponding to the object. Surface reconstruction is performed based on the three-dimensional point volume occupancy probability to generate an initial three-dimensional model corresponding to the object; The texture domain is reconstructed based on the front view image, the rear view image, and the initial 3D model to generate the 3D model.

[0091] In one optional implementation, the step of reconstructing the texture domain based on the front view image, the rear view image, and the initial 3D model to generate the 3D model includes: Determine each grid face of the object; Based on the normal vector corresponding to each of the grid faces and the camera ray vector corresponding to the front view image, the view corresponding to each of the grid faces is determined; Based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, the texture domain of each mesh surface is reconstructed to generate the three-dimensional model.

[0092] In one alternative implementation, the view corresponding to each of the grid surfaces is a front view, a rear view, or a boundary view; The process of reconstructing the texture domain of each mesh surface based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, includes: For each of the aforementioned mesh faces When the corresponding view is the front view, the texture domain of the mesh surface is reconstructed based on the front view image; When the corresponding view is a rear view, the texture domain of the mesh surface is reconstructed based on the rear view image; When the corresponding view is a boundary view, the texture domain is reconstructed for the mesh surface based on the front view image and the rear view image.

[0093] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0094] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0095] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0096] See Figure 6 The computer device in this embodiment includes a processor 601, a memory 602, and a computer program, such as a 3D reconstruction program, stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program, it implements the steps described in the various 3D reconstruction method embodiments above, for example... Figure 1 The steps S101-S105 are shown.

[0097] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 602 and executed by the processor 601 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0098] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0099] The processor 601 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 601 can be any conventional processor. The processor 601 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0100] The memory 602 can be used to store the computer programs and / or modules. The processor 601 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 602 and calling the data stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0101] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by the processor 601, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0102] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, the corresponding rear view image can be determined based on the acquired front view image, i.e., it is not required to acquire images from multiple perspectives, thereby improving the efficiency and applicability of 3D reconstruction. Furthermore, the image features and normal features of each image can be used as prior constraints to constrain the reconstruction, and the implicit depth function corresponding to the object can be determined by combining the random sampling points of the object in 3D space. This implicit depth function can then be used to assist the front view image and the rear view image in 3D reconstruction processing, thereby improving the accuracy of 3D reconstruction.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0104] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: Acquire a front view image, wherein the front view image corresponds to the object to be reconstructed; Determine the rear view image corresponding to the front view image; Determine the image features of the front view image and the rear view image respectively, and determine the normal features of the front view image and the rear view image respectively; Based on the image features, the normal features, and the random sampling points of the object in three-dimensional space, the implicit depth function corresponding to the object is determined; Based on the front view image, the rear view image, and the implicit depth function, a 3D reconstruction process is performed to generate a 3D model corresponding to the object.

2. The method according to claim 1, characterized in that, Determining the rear view image corresponding to the front view image includes: Determine the two-dimensional contour information corresponding to the front view image; The front view image and the two-dimensional contour information are input into an image conversion network to obtain the rear view image output by the image conversion network, wherein the image conversion network is constructed based on a conditional generative adversarial algorithm.

3. The method according to claim 1, characterized in that, The step of determining the implicit depth function corresponding to the object based on the image features, the normal features, and random sampling points of the object in three-dimensional space includes: Gaussian perturbation is added to the random sampling points to obtain the target sampling points; The image features and the normal features are used as prior constraints for the depth latent function, and the depth latent function is constructed based on the target sampling points.

4. The method according to claim 1, characterized in that, The 3D reconstruction process includes surface reconstruction and texture domain reconstruction; The step of performing 3D reconstruction processing based on the front view image, the rear view image, and the implicit depth function to generate a 3D model corresponding to the object includes: Based on the aforementioned depth implicit function, the multilayer perceptron is invoked to obtain the volume occupancy probability of the three-dimensional points corresponding to the object. Surface reconstruction is performed based on the three-dimensional point volume occupancy probability to generate an initial three-dimensional model corresponding to the object; The texture domain is reconstructed based on the front view image, the rear view image, and the initial 3D model to generate the 3D model.

5. The method according to claim 4, characterized in that, The step of reconstructing the texture domain based on the front view image, the rear view image, and the initial 3D model to generate the 3D model includes: Determine each grid face of the object; Based on the normal vector corresponding to each of the grid faces and the camera ray vector corresponding to the front view image, the view corresponding to each of the grid faces is determined; Based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, the texture domain of each mesh surface is reconstructed to generate the three-dimensional model.

6. The method according to claim 5, characterized in that, Each of the grid surfaces corresponds to a front view, a rear view, or a boundary view; The process of reconstructing the texture domain of each mesh surface based on the views corresponding to each mesh surface, as well as the front view image and the rear view image, includes: For each of the aforementioned mesh faces When the corresponding view is the front view, the texture domain of the mesh surface is reconstructed based on the front view image; When the corresponding view is a rear view, the texture domain of the mesh surface is reconstructed based on the rear view image; When the corresponding view is a boundary view, the texture domain is reconstructed for the mesh surface based on the front view image and the rear view image.

7. A three-dimensional reconstruction device, characterized in that, include: A front view acquisition module is used to acquire a front view image, wherein the front view image corresponds to the object to be reconstructed; The rear view determination module is used to determine the rear view image corresponding to the front view image; The feature determination module is used to determine the image features of the front view image and the rear view image respectively, and to determine the normal features of the front view image and the rear view image respectively; The implicit function calculation module is used to determine the depth implicit function corresponding to the object based on the image features, the normal features, and random sampling points of the object in three-dimensional space. The reconstruction module is used to perform three-dimensional reconstruction processing based on the front view image, the rear view image and the depth implicit function to generate a three-dimensional model corresponding to the object.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-6.

10. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-6.

Images