3D reconstruction method and device, and storage medium

Abstract

A three-dimensional (3D) reconstruction method is provided, the method including: obtaining 2D pixel coordinates of a pixel in a 2D image of an object and a direction of a light ray, projecting the 2D pixel coordinates into a 3D space to obtain a plurality of 3D points on the light ray, for each 3D point of the plurality of 3D points, predicting a signed distance value of the 3D point, predicting a validity probability of the 3D point, and predicting an intensity value of the 3D point, obtaining a rendering loss based on the predicted signed distance values, validity probability, and intensity values ​​of the plurality of 3D points, updating a 3D reconstruction network, and extracting a 3D model of the object based on the updated 3D reconstruction network.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Patent Application No. 17 / 947,599, filed September 19, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer vision and computer graphics technology, and in particular to a three-dimensional (3D) reconstruction method, apparatus and storage medium. [Background technology]

[0003] Three-dimensional (3D) modeling and reconstruction are fundamental problems in computer vision and computer graphics. Existing learning-based approaches require a large corpus of 3D data for model training, and data capture and labeling are laborious. Differential Rendering (DR) offers an alternative to learn 3D shapes directly from two-dimensional (2D) images without relying on 3D ground truth. However, existing DR techniques based on signed distance functions (SDFs) are limited to watertight shapes and cannot reconstruct shapes with open boundaries. Summary of the Invention [Problem to be solved by the invention]

[0004] The disclosed method and system is directed to solving one or more of the problems set forth above, as well as other problems. [Means for solving the problem]

[0005] An embodiment of the present disclosure introduces a three-dimensional (3D) reconstruction process. In this process, 2D pixel coordinates and ray directions of pixels in a two-dimensional (2D) image of an object are obtained. The 2D pixel coordinates are projected into 3D space to obtain multiple 3D points on the ray. For each of the multiple 3D points, a signed distance value of the 3D point is predicted, indicating the signed distance from the 3D point to a mesh surface. A validity probability of the 3D point is predicted, indicating the probability that the 3D point has a valid signed distance value. An intensity value of the 3D point is predicted. A rendering loss is then obtained based on the predicted signed distance values, validity probability, and intensity values ​​of the multiple 3D points, and a 3D reconstruction network is updated. A 3D model of the object is extracted based on the updated 3D reconstruction network.

[0006] One aspect of the present disclosure provides a three-dimensional (3D) reconstruction method, including: obtaining two-dimensional (2D) pixel coordinates of pixels in a 2D image of an object and a ray direction; projecting the 2D pixel coordinates into a 3D space to obtain a plurality of 3D points on the ray; predicting, for each of the plurality of 3D points, a signed distance value of the 3D point, where the signed distance value of the 3D point indicates a signed distance from the 3D point to a mesh surface; predicting a validity probability of the 3D point, where the validity probability of the 3D point indicates a probability that the 3D point has a valid signed distance value; predicting an intensity value of the 3D point; obtaining a rendering loss based on the predicted signed distance value, the validity probability, and the intensity values ​​of the plurality of 3D points, and updating a 3D reconstruction network; and extracting a 3D model of the object based on the updated 3D reconstruction network.

[0007] Another aspect of the present disclosure provides an apparatus for three-dimensional (3D) reconstruction, including a memory storing a computer program and a processor. The processor is configured to execute the computer program to: obtain two-dimensional (2D) pixel coordinates of pixels in a 2D image of an object and a ray direction; project the 2D pixel coordinates into 3D space to obtain a plurality of 3D points on the ray; predict, for each of the plurality of 3D points, a signed distance value of the 3D point, where the signed distance value of the 3D point indicates a signed distance from the 3D point to a mesh surface; predict a validity probability of the 3D point, where the validity probability of the 3D point indicates a probability that the 3D point has a valid signed distance value; predict an intensity value of the 3D point; obtain a rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points, and update a 3D reconstruction network; and extract a 3D model of the object based on the updated 3D reconstruction network.

[0008] Another aspect of the present disclosure provides a non-transitory storage medium having stored thereon computer instructions that, when executed by a processor, cause the processor to: obtain two-dimensional (2D) pixel coordinates of pixels in a 2D image of an object and a ray direction; project the 2D pixel coordinates into 3D space to obtain a plurality of 3D points on the ray; for each 3D point of the plurality of 3D points, predict a signed distance value of the 3D point, where the signed distance value of the 3D point indicates a signed distance from the 3D point to a mesh surface; predict a validity probability of the 3D point, where the validity probability of the 3D point indicates a probability that the 3D point has a valid signed distance value; predict an intensity value of the 3D point; obtain a rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points, and update a 3D reconstruction network; and extract a 3D model of the object based on the updated 3D reconstruction network.

[0009] Other aspects of the present disclosure will be apparent to those skilled in the art in light of the description, claims, and drawings of the present disclosure. [Brief explanation of the drawings]

[0010] The patent or application file contains at least one color drawing. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the appropriate fee.

[0011] [Figure 1] FIG. 1 illustrates a computer system consistent with an embodiment of the present disclosure.

[0012] [Figure 2] FIG. 2 illustrates a three-dimensional (3D) reconstruction process consistent with an embodiment of the present disclosure.

[0013] [Figure 3] FIG. 3 illustrates a framework for a 3D reconstruction process consistent with embodiments of the present disclosure.

[0014] [Figure 4] FIG. 4 illustrates a surface representation according to some embodiments of the present disclosure.

[0015] [Figure 5] FIG. 5 shows a visual comparison of 3D reconstructions on watertight surfaces, consistent with an embodiment of the present disclosure.

[0016] [Figure 6] FIG. 6 illustrates a visual comparison of 3D reconstructions on an open surface, consistent with certain embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, technical solutions in embodiments of the present invention will be described with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. Obviously, the described embodiments are only a part, not all, of the embodiments of the present invention. Other embodiments that can be obtained by those skilled in the art based on the embodiments of the present invention without any inventive efforts shall fall within the protection scope of the present disclosure.

[0018] 1 is a block diagram of an exemplary computing system / device capable of implementing the disclosed 3D reconstruction method according to some embodiments of the present disclosure. As shown in FIG. 1, computing system 100 may include a processor 102 and a storage medium 104. According to certain embodiments, computing system 100 may further include a display 106, a communication module 108, peripheral devices 112, and one or more buses 114 for coupling devices. Certain devices may be omitted, and other devices may be included.

[0019] The processor 102 may include any suitable processor(s). In certain embodiments, the processor 102 may include multiple cores for multithreaded or parallel processing and / or a graphics processing unit (GPU). The processor 102 may execute sequences of computer program instructions to perform various processes, such as a 3D reconstruction program. The storage medium 104 may be a non-transitory computer-readable storage medium and may include a memory module, such as a ROM, RAM, a flash memory module, an erasable and rewritable memory, or a mass storage device, such as a CD-ROM, a U-disk, or a hard disk. The storage medium 104 may store computer programs that, when executed by the processor 102, perform various processes. The storage medium 104 may also include one or more databases for storing specific data, such as text scripts, library data, and training data sets, on which specific operations, such as database lookups and data retrievals, can be performed.

[0020] The communications module 108 may include a network device for establishing a connection over a network. The display 106 may include any suitable type of computer display device or electronic device display (e.g., CRT or LCD-based device, touch screen). The peripheral devices 112 may include additional I / O devices such as a keyboard, a mouse, etc.

[0021] In operation, the processor 102 may be configured to execute instructions stored on the storage medium 104 to perform various operations related to the 3D reconstruction method, as detailed in the following description. The 3D reconstruction method for one or more objects may be used in any suitable application requiring a 3D model output based on a 2D image input. As an example, the 3D reconstruction may be applied to games, such as creating and modifying 3D game objects, designing game logic or story plots, or virtual representation of real-world objects in a game environment. The game may be a dress-up game, a makeover game, a home design game, or any other suitable game. In another example, the 3D reconstruction may be used to create / update a character's appearance in a virtual meeting, dress up or makeover in an online shopping scenario, etc.

[0022] As shown in Figure 2, a 3D reconstruction method consistent with an embodiment of the present disclosure includes the following processes. The method can be implemented, for example, by the computing system 100 shown in Figure 1. Figure 3 shows a framework for a 3D reconstruction process consistent with an embodiment of the present disclosure. The method can be implemented according to the framework shown in Figure 3.

[0023] In S202, two-dimensional (2D) pixel coordinates of pixels in a 2D image of an object and a ray direction are obtained. The 2D image may be, for example, the image 302 shown in FIG.

[0024] In some embodiments, multiple 2D pixel coordinates of multiple pixels in the 2D image are acquired. In some embodiments, for 3D image reconstruction, two or more 2D images of the object corresponding to two or more different directions of light rays are acquired, where light rays may be understood to be light rays from the camera.

[0025] TIFF0007778249000001.tif68169

[0026] In S204, the 2D pixel coordinates are projected into a 3D space to obtain a plurality of 3D points on the light rays. In some embodiments, all or some of the plurality of 2D pixels of the 2D image may be projected into the same 3D space according to the light rays corresponding to the 2D pixels.

[0027] TIFF0007778249000002.tif38169

[0028] At S206, for each 3D point of the plurality of 3D points, a signed distance value for the 3D point is predicted. The signed distance value for the 3D point indicates a signed distance from the 3D point to a mesh surface. In some embodiments, the mesh surface may be composed of a collection of polygonal faces, such as triangles, that together form a surface covering of the object.

[0029] TIFF0007778249000003.tif23168

[0030] TIFF0007778249000004.tif71168

[0031] In S208, the validity probability of the 3D point is predicted, which indicates the probability that the 3D point has a valid signed distance value.

[0032] FIG. 4 illustrates a surface representation according to some embodiments of the present disclosure. As shown in FIG. 4, in some embodiments, watertight 3D shapes can be implicitly represented by a signed distance function (SDF). A watertight surface or watertight shape is a closed surface or shape. A closed surface is a surface without edges or boundaries. For example, a closed surface may include a sphere, a torus, or a Klein bottle. Existing techniques based on signed distance functions (SDFs) are limited to watertight shapes. As shown on the left side of FIG. 4, a Marching Cubes algorithm can extract a closed surface from an SDF. The present disclosure provides embodiments that can reconstruct 3D surfaces with arbitrary topologies, including both watertight and open surfaces, according to a combination of a signed distance function and a validity probability function, as shown on the right side of FIG. 4.

[0033] TIFF0007778249000005.tif14168

[0034] In some embodiments, in response to the predicted validity probability of one of the plurality of 3D points being less than a preset threshold, an invalid value is assigned to the predicted signed distance value of one of the plurality of 3D points. The preset threshold may be 0.5, 0.6, 0.7, 0.8, 0.9, etc. The preset threshold may be any suitable value between 0 and 1.

[0035] TIFF0007778249000006.tif38169

[0036] TIFF0007778249000007.tif63169

[0037] In some embodiments, a rendering weight for a 3D point is calculated based on the predicted signed distance value and the predicted validity probability of the 3D point.

[0038] In some embodiments, predicting the rendering weight of the 3D point may include determining a probability density function that maps the predicted signed distance value of the 3D point to a volume density, determining a volume weight function based on the volume density and the validity probability of the 3D point, and determining the rendering weight of the 3D point based on the volume weight function.

[0039] In watertight surface reconstruction, the rendering process ignores 3D points within the range of the camera ray exiting the surface from the inside out. In volume rendering, the volume weight is set to zero. In surface rendering, the points are ignored. This operation does not result in mis-rendering any pixels, since the renderer has already rendered the surface when the ray enters the surface from the outside in.

[0040] For open surfaces, inside and outside may not be clearly defined. A ray can go directly from the "pseudo" inside to the "pseudo" outside without going "pseudo" inside. To render all surfaces, each valid surface point is rendered if a ray enters the surface from the "pseudo" outside towards the "pseudo" inside, and each valid surface point is rendered if a ray exits the surface from the "pseudo" inside towards the "pseudo" outside.

[0041] To apply volume rendering techniques to the reconstruction, a probability density function may be chosen that maps a signed distance field (SDF) to a volume density. In some embodiments, the rendering behaves the same when a ray enters the surface from "pseudo" outside to "pseudo" inside and when a ray exits the surface from "pseudo" inside to "pseudo" outside.

[0042] In some embodiments, determining a probability density function that maps the predicted signed distance values ​​of the 3D points to volume density includes determining the probability density function based on a sigmoid mapping function, the predicted signed distance values, and a sign adjustment function. The sign adjustment function is configured to change the predicted signed distance values ​​of the 3D points from predicted signed distance values ​​in a first direction to predicted signed distance values ​​in a second direction, where the first direction is opposite to the second direction. For example, the predicted signed distance values ​​are negative values, indicating that the first direction of the predicted signed distance is from inside to outside the boundary. The sign adjustment function is used to change the predicted signed distance values ​​to positive values, indicating a direction opposite to the first direction.

[0043] TIFF0007778249000008.tif29169

[0044] TIFF0007778249000009.tif24169

[0045] TIFF0007778249000010.tif25169

[0046] TIFF0007778249000011.tif18169

[0047] TIFF0007778249000012.tif23168

[0048] TIFF0007778249000013.tif14168

[0049] In some embodiments, the weighting function 312, denoted as w(t) on the ray, may be defined based on the signed distance field (SDF) of the scene. In some embodiments, the weighting function may satisfy at least one of an unbiased requirement or an occlusion-aware requirement. In some embodiments, the weighting function may satisfy both an unbiased requirement and an occlusion-aware requirement.

[0050] TIFF0007778249000014.tif20168

[0051] TIFF0007778249000015.tif39168

[0052] In some embodiments, to derive the opacity density ρ(t), one may first consider the simple case of only one plane in the sampling space, and then ρ(t) is generalized to the general case of multiple surface intersections.

[0053] TIFF0007778249000016.tif94168

[0054] TIFF0007778249000017.tif19168

[0055] TIFF0007778249000018.tif18168

[0056] TIFF0007778249000019.tif19168

[0057] TIFF0007778249000020.tif23168

[0058] TIFF0007778249000021.tif109168

[0059] TIFF0007778249000022.tif84168

[0060] In some embodiments, calculating the rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point includes calculating a discrete opacity value for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point and the predicted signed distance value and the predicted validity probability of one of a plurality of 3D points neighboring the 3D point on the ray.

[0061] TIFF0007778249000023.tif19168

[0062] TIFF0007778249000024.tif16168

[0063] JPEG0007778249000025.jpg19169

[0064] JPEG0007778249000026.jpg30169

[0065] In S210, the intensity values ​​of the 3D points are predicted.

[0066] JPEG0007778249000027.jpg29169

[0067] In some embodiments, the intensity value of a 3D point may include an RGB value including an intensity value for a red channel, an intensity value for a green channel, and an intensity value for a blue channel. In some embodiments, the intensity value of a 3D point may include intensity values ​​for one or more of the channels of other color formats, such as CMYK, HEX, or grayscale. In some embodiments, the intensity value of a 3D point may be represented using other suitable methods.

[0068] JPEG0007778249000028.jpg39169

[0069] JPEG0007778249000029.jpg22169

[0070] In S212, a rendering loss is obtained based on the predicted signed distance values, validity probabilities, and intensity values ​​of the multiple 3D points to update the 3D reconstruction network.

[0071] TIFF0007778249000030.tif168169

[0072] In some embodiments, the 3D reconstruction network may be updated with or without an image mask.

[0073] In some embodiments, the 3D reconstruction network may be implemented using either multiple perceptual layers or one-dimensional convolutional layer(s).

[0074] In S214, a 3D model of the object is extracted based on the updated 3D reconstruction network.

[0075] In some embodiments, a 3D model includes a mathematical coordinate-based representation of any surface of an object in three dimensions, including edges, vertices, polygons, etc.

[0076] In some embodiments, the reconstructed surface of the object can be extracted as a zero-level isosurface of the SDF. An isosurface is a surface that represents points of constant value within a volume of space. The isosurface can be extracted directly using the classical Marching Cube (MC) algorithm. The presence of invalid signed distance values ​​can prevent MC from extracting a valid isosurface at locations that do not contain geometry. After the Marching Cubes calculation, all invalid vertices and faces generated by the null cubes can be removed. The remaining vertices and faces are the result of mesh generation.

[0077] To evaluate 3D reconstruction, we compared the 3D model reconstructed by the disclosed 3D reconstruction network with the 3D reconstruction method proposed by Wang et al. in "NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction," NeurIPS (2021) (hereinafter referred to as "Neus").

[0078] FIG. 5 shows a visual comparison of 3D reconstructions for watertight surfaces consistent with embodiments of the present disclosure. As shown in FIG. 5, the first row shows an image of a 3D reconstructed object. The second row of FIG. 5 shows a 3D model of the object reconstructed using the 3D reconstruction method of the present disclosure. The third row of FIG. 5 shows a 3D model of the object reconstructed using Neus's 3D reconstruction method. Table 1 shows a quantitative comparison of watertight surface data using the 3D reconstruction method of the present disclosure and the 3D reconstruction method of Neus. As shown in Table 1, for watertight surfaces, the 3D reconstruction method of the present disclosure generally has smaller numerical errors than the 3D reconstruction method of Neus. [Table 1]

[0079] FIG. 6 shows a visual comparison of 3D reconstructions on an open surface consistent with certain embodiments of the present disclosure, with the front surface highlighted in blue and the back surface highlighted in pink. As shown in FIG. 6, the first row shows an image of the 3D reconstructed object. The second row of FIG. 6 shows the ground truth of the object. The third row of FIG. 6 shows a 3D model of the object reconstructed using the 3D reconstruction method of the present disclosure, and the fourth row of FIG. 6 shows a 3D model of the object reconstructed using the 3D reconstruction method in Neus. As shown in FIG. 6, the 3D reconstruction method of the present disclosure reconstructs a high-fidelity single-layer surface, whereas prior art reconstructs a two-layer surface.

[0080] Table 2 shows a quantitative comparison between open surface data using the disclosed 3D reconstruction method and open surface data using Neus's 3D reconstruction method. As shown in Table 2, for open surfaces, the disclosed 3D reconstruction method has smaller numerical errors than existing techniques. [Table 2]

[0081] Existing techniques based on signed distance functions (SDFs) are limited to watertight shapes. This disclosure provides embodiments that can reconstruct 3D surfaces with arbitrary topologies, including both watertight and open surfaces, according to a combination of signed distance functions and validity probability functions.

[0082] The sequence numbers of the foregoing embodiments of the present disclosure are merely for illustrative purposes and do not indicate the priority of the embodiments.

[0083] When the integrated unit in the above-described embodiments is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in the above-described computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure may be essentially implemented, or a part that contributes to the related art, or all or a part of the technical solution may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for instructing one or more computer devices (which may be personal computers, servers, network devices, etc.) to perform all or a part of the steps of the method in the embodiments of the present disclosure.

[0084] In the above-described embodiments of the present disclosure, the descriptions of the embodiments have different emphasis points. For parts not described in detail in one embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] It should be understood that in some embodiments provided in the present disclosure, the disclosed client may be implemented in other ways. The above-described device embodiments are merely examples. For example, the division of units is merely a division of logical functions, and other division methods may be used in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some functions may be omitted or not performed. Furthermore, couplings, or direct couplings, or communication connections between components shown or discussed may be indirect couplings or communication connections via some interfaces, units, or modules, and may be electrical or other forms.

[0086] The units described as separate parts may or may not be physically separable. The parts shown as units may or may not be physical units, and may be located in one location or distributed across multiple network units. Some or all of the units may be selected according to the actual requirements for achieving the objectives of the solution in the embodiment.

[0087] Furthermore, the functional units in the embodiments of the present disclosure may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or a software functional unit.

[0088] Although the principles and embodiments of the present disclosure are described herein using specific embodiments, the foregoing description of the embodiments is intended only to help understand the core idea of ​​the method and method of the present disclosure. Meanwhile, those skilled in the art may make modifications to the specific embodiments and application scope according to the idea of ​​the present disclosure. In conclusion, the contents of this specification should not be interpreted as limiting the present disclosure.

Claims

1. A three-dimensional (3D) reconstruction method, comprising: Obtaining two-dimensional (2D) pixel coordinates of a pixel in a 2D image of an object and a ray direction; projecting the 2D pixel coordinates into 3D space to obtain a plurality of 3D points on the ray; For each 3D point of the plurality of 3D points, predicting a signed distance value of the 3D point, the predicted signed distance value indicating the signed distance of the 3D point indicating a signed distance from the 3D point to the mesh surface; predicting a validity probability of the 3D point, wherein the predicted validity probability indicative of the validity probability of the 3D point indicates the probability that the 3D point has a valid signed distance value; predicting intensity values ​​of said 3D points; obtaining a rendering loss based on the predicted signed distance value, the predicted validity probability, and intensity values ​​of a plurality of the 3D points and updating a 3D reconstruction network; extracting a 3D model of the object based on the updated 3D reconstruction network; and A method comprising:

2. extracting a 3D model of the object based on the updated 3D reconstruction network, 2. The method of claim 1, comprising: in response to the predicted validity probability of one of the plurality of 3D points being less than a predetermined threshold, assigning an invalid value to the predicted signed distance value of one of the plurality of 3D points; and extracting the 3D model of the object by removing vertices having the predicted signed distance value assigned the invalid value.

3. 2. The method of claim 1, further comprising: for a 3D point of the plurality of 3D points, calculating a rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability for the 3D point.

4. Calculating a rendering weight for a 3D point based on a predicted signed distance value and a predicted validity probability for the 3D point, determining a probability density function that maps predicted signed distance values ​​of the 3D points to volume density; determining a volume weighting function based on volume density and validity probability of the 3D points; and determining a rendering weight for the 3D point based on a volume weighting function.

5. Determining the probability density function that maps the predicted signed distance values ​​of the 3D points to the volume density may include: determining the probability density function based on a sigmoid mapping function, the predicted signed distance value, and a sign adjustment function; 5. The method of claim 4, wherein the sign adjustment function is configured to change the predicted signed distance value of the 3D point from the predicted signed distance value in a first direction to a predicted signed distance value in a second direction, the first direction being opposite to the second direction.

6. Calculating the rendering weight of the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point comprises:

4. The method of claim 3, comprising calculating a discrete opacity value for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point and the predicted signed distance value and the predicted validity probability of one of the plurality of 3D points neighboring the 3D point.

7. predicting a mask based on the rendering weights of the plurality of 3D points; Obtaining the rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points to update the 3D reconstruction network includes: The method of claim 3 , comprising obtaining the rendering loss based on the predicted mask to update the 3D reconstruction network.

8. Predicting the mask based on the rendering weights of the 3D points includes: The method of claim 7 , comprising predicting the mask based on an accumulation of a plurality of the rendering weights of the plurality of 3D points.

9. Obtaining a rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points to update the 3D reconstruction network includes: calculating a rendering probability loss based on the validity probabilities of the plurality of 3D points and a binary cross-entropy of the validity probabilities; and obtaining the rendering loss according to the rendering probability loss.

10. Obtaining the rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points to update the 3D reconstruction network includes: calculating a rendering probability regularization loss based on an average of the validity probabilities of the plurality of 3D points; and obtaining the rendering loss according to the rendering probability regularization loss.

11. An apparatus for three-dimensional (3D) reconstruction, comprising: a memory for storing a computer program; a processor that executes the computer program; and Equipped with The computer program causes the processor to: Obtaining two-dimensional (2D) pixel coordinates of a pixel in a 2D image of an object and a ray direction; Projecting the 2D pixel coordinates into 3D space to obtain a plurality of 3D points on the ray; For each 3D point of the plurality of 3D points, predicting a signed distance value of the 3D point, the predicted signed distance value indicating the signed distance value of the 3D point indicating a signed distance from the 3D point to a mesh surface; predicting a validity probability of the 3D point, wherein a predicted validity probability indicative of the validity probability of the 3D point indicates a probability that the 3D point has a valid signed distance value; predicting intensity values ​​of said 3D points; obtaining a rendering loss based on the predicted signed distance value, the predicted validity probability, and the intensity values ​​of the plurality of 3D points and updating a 3D reconstruction network; and extracting a 3D model of the object based on the updated 3D reconstruction network. Device.

12. The processor:

12. The apparatus of claim 11, further configured: in response to the predicted validity probability of one of the plurality of 3D points being less than a preset threshold, assigning an invalid value to the predicted signed distance value of one of the plurality of 3D points, and extracting the 3D model of the object by removing vertices having the predicted signed distance value assigned the invalid value.

13. The processor 12. The apparatus of claim 11, further configured to: for a 3D point of the plurality of 3D points, calculate a rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability for the 3D point.

14. The processor: determining a probability density function that maps the predicted signed distance values ​​of the 3D points to volume density; determining a volume weighting function based on the volume density and the validity probability of the 3D points; The apparatus of claim 13 , further configured to determine a rendering weight for the 3D point based on the volume weighting function.

15. The processor:

15. The apparatus of claim 14, further configured: to determine the probability density function based on a sigmoid mapping function, the predicted signed distance values, and a sign adjustment function, the sign adjustment function configured to change the predicted signed distance values ​​of the 3D points from the predicted signed distance values ​​in a first direction to predicted signed distance values ​​in a second direction, the first direction being opposite to the second direction.

16. The processor:

14. The apparatus of claim 13, further configured to calculate a discrete opacity of the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point and the predicted signed distance value and the predicted validity probability of one of the plurality of 3D points neighboring the 3D point.

17. The processor: predicting a mask based on the plurality of rendering weights of the plurality of 3D points; The apparatus of claim 15 , further configured to determine the rendering loss based on the predicted mask and update the 3D reconstruction network.

18. The processor: calculating a rendering probability loss based on the validity probabilities of the plurality of 3D points and a binary cross-entropy of the validity probabilities; The apparatus of claim 11 , further configured to obtain the rendering loss according to the rendering probability loss.

19. The processor: calculating a rendering probability regularization loss based on an average of the validity probabilities of the plurality of 3D points; The apparatus of claim 11 , further configured to obtain the rendering loss according to the rendering probability regularization loss.

20. When executed by a processor, the processor: Obtaining two-dimensional (2D) pixel coordinates of a pixel in a 2D image of an object and a ray direction; projecting the 2D pixel coordinates into three-dimensional (3D) space to obtain a plurality of 3D points on the ray; For each 3D point of the plurality of 3D points, predicting a signed distance value of the 3D point, the predicted signed distance value indicating the signed distance value of the 3D point indicating a signed distance from the 3D point to a mesh surface; predicting a validity probability of the 3D point, wherein the predicted validity probability indicative of the validity probability of the 3D point indicates the probability that the 3D has a valid signed distance value; predicting intensity values ​​of said 3D points; obtaining a rendering loss based on the predicted signed distance value, the predicted validity probability, and the intensity values ​​of the plurality of 3D points and updating a 3D reconstruction network; and extracting a 3D model of the object based on the updated 3D reconstruction network.

Images