3D reconstruction method and device, and storage medium

Abstract

A three-dimensional (3D) reconstruction method is provided, the method including: obtaining 2D pixel coordinates of a pixel in a 2D image of an object and a direction of a light ray, projecting the 2D pixel coordinates into a 3D space to obtain a plurality of 3D points on the light ray, for each 3D point of the plurality of 3D points, predicting a signed distance value of the 3D point, predicting a validity probability of the 3D point, and predicting an intensity value of the 3D point, obtaining a rendering loss based on the predicted signed distance values, validity probability, and intensity values ​​of the plurality of 3D points, updating a 3D reconstruction network, and extracting a 3D model of the object based on the updated 3D reconstruction network.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Patent Application No. 17 / 947,599, filed September 19, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer vision and computer graphics technology, and in particular to a three-dimensional (3D) reconstruction method, apparatus and storage medium. [Background technology]

[0003] Three-dimensional (3D) modeling and reconstruction is a fundamental problem in computer vision and computer graphics. Existing learning-based approaches require a large corpus of 3D data for training models, and data capture and labeling are laborious. Differential Rendering (DR) offers an alternative to learn 3D shapes directly from two-dimensional (2D) images without relying on 3D ground truth. However, existing DR techniques based on Signed Distance Functions (SDFs) are limited to watertight shapes and cannot reconstruct shapes with open boundaries. Summary of the Invention [Problem to be solved by the invention]

[0004] The disclosed method and system is directed to solving one or more of the problems set forth above, as well as other problems. [Means for solving the problem]

[0005] An embodiment of the present disclosure introduces a three-dimensional (3D) reconstruction process, in which 2D pixel coordinates and ray directions of pixels in a two-dimensional (2D) image of an object are obtained. The 2D pixel coordinates are projected into a 3D space to obtain a plurality of 3D points on the ray. For each 3D point of the plurality of 3D points, a signed distance value of the 3D point is predicted, indicating a signed distance from the 3D point to a mesh surface. A validity probability of the 3D point is predicted, indicating a probability that the 3D point has a valid signed distance value. An intensity value of the 3D point is predicted. Then, a rendering loss is obtained based on the predicted signed distance values, validity probabilities, and intensity values ​​of the plurality of 3D points, and a 3D reconstruction network is updated. Based on the updated 3D reconstruction network, a 3D model of the object is extracted.

[0006] One aspect of the present disclosure provides a three-dimensional (3D) reconstruction method, the method including: obtaining two-dimensional (2D) pixel coordinates of a pixel in a 2D image of an object and a direction of a light ray; projecting the 2D pixel coordinates into a 3D space to obtain a plurality of 3D points on the light ray; for each 3D point of the plurality of 3D points, predicting a signed distance value of the 3D point, the signed distance value of the 3D point indicating a signed distance from the 3D point to a mesh surface, predicting a validity probability of the 3D point, the validity probability of the 3D point indicating a probability that the 3D point has a valid signed distance value, predicting an intensity value of the 3D point; obtaining a rendering loss and updating a 3D reconstruction network based on the predicted signed distance value, the validity probability, and the intensity values ​​of the plurality of 3D points; and extracting a 3D model of the object based on the updated 3D reconstruction network.

[0007] Another aspect of the present disclosure provides an apparatus for three-dimensional (3D) reconstruction, including a memory storing a computer program and a processor, the processor being configured to execute the computer program to: obtain two-dimensional (2D) pixel coordinates of a pixel in a 2D image of an object and a direction of a ray; project the 2D pixel coordinates into a 3D space to obtain a plurality of 3D points on the ray; predict, for each 3D point of the plurality of 3D points, a signed distance value of the 3D point, the signed distance value of the 3D point indicating a signed distance from the 3D point to a mesh surface; predict a validity probability of the 3D point, the validity probability of the 3D point indicating a probability that the 3D point has a valid signed distance value; predict an intensity value of the 3D point; obtain a rendering loss and update a 3D reconstruction network based on the predicted signed distance value, the validity probability, and the intensity values ​​of the plurality of 3D points; and extract a 3D model of the object based on the updated 3D reconstruction network.

[0008] Another aspect of the disclosure provides a non-transitory storage medium having stored thereon computer instructions that, when executed by a processor, cause the processor to: obtain two-dimensional (2D) pixel coordinates of a pixel in a 2D image of an object and a direction of a light ray; project the 2D pixel coordinates into a 3D space to obtain a plurality of 3D points on the light ray; for each 3D point of the plurality of 3D points, predict a signed distance value of the 3D point, where the signed distance value of the 3D point indicates a signed distance from the 3D point to a mesh surface; predict a validity probability of the 3D point, where the validity probability of the 3D point indicates a probability that the 3D point has a valid signed distance value; predict an intensity value of the 3D point; obtain a rendering loss and update a 3D reconstruction network based on the predicted signed distance value, the validity probability, and the intensity values ​​of the plurality of 3D points; and extract a 3D model of the object based on the updated 3D reconstruction network.

[0009] Other aspects of the present disclosure will be apparent to those skilled in the art in light of the description, claims, and drawings of the present disclosure. [Brief description of the drawings]

[0010] The patent or application file contains at least one color drawing. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the appropriate fee.

[0011] [Figure 1] FIG. 1 illustrates a computer system consistent with an embodiment of the present disclosure.

[0012] [Diagram 2] FIG. 2 illustrates a three-dimensional (3D) reconstruction process consistent with an embodiment of the present disclosure.

[0013] [Diagram 3] FIG. 3 illustrates a framework for a 3D reconstruction process consistent with an embodiment of the present disclosure.

[0014] [Figure 4] FIG. 4 illustrates a surface representation according to some embodiments of the present disclosure.

[0015] [Diagram 5] FIG. 5 shows a visual comparison of 3D reconstructions on watertight surfaces, consistent with an embodiment of the present disclosure.

[0016] [Figure 6] FIG. 6 illustrates a visual comparison of 3D reconstructions on an open surface, consistent with certain embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, the technical solutions in the embodiments of the present invention will be described with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or similar parts. Obviously, the described embodiments are only a part, not all, of the embodiments of the present invention. Other embodiments that can be obtained by those skilled in the art based on the embodiments of the present invention without inventive ideas shall be included in the protection scope of the present disclosure.

[0018] 1 is a block diagram of an exemplary computing system / device capable of implementing the disclosed 3D reconstruction method according to some embodiments of the present disclosure. As shown in FIG. 1, the computing system 100 may include a processor 102 and a storage medium 104. According to certain embodiments, the computing system 100 may further include a display 106, a communication module 108, a peripheral device 112, and one or more buses 114 for coupling the devices. Certain devices may be omitted and other devices may be included.

[0019] The processor 102 may include any suitable processor(s). In certain embodiments, the processor 102 may include multiple cores for multi-threaded or parallel processing, and / or a Graphics Processing Unit (GPU). The processor 102 may execute sequences of computer program instructions to perform various processes, such as a 3D reconstruction program. The storage medium 104 may be a non-transitory computer-readable storage medium and may include memory modules, such as ROM, RAM, flash memory modules, erasable and rewritable memory, mass storage devices, such as CD-ROM, U-disk, hard disk, etc. The storage medium 104 may store computer programs for performing various processes when executed by the processor 102. The storage medium 104 may also include one or more databases for storing certain data, such as text scripts, library data, training data sets, etc., and certain operations, such as database lookup and data lookup, can be performed on the stored data.

[0020] The communications module 108 may include a network device for establishing a connection over a network. The display 106 may include any suitable type of computer display device or electronic device display (e.g., CRT or LCD based device, touch screen). The peripheral devices 112 may include additional I / O devices such as a keyboard, a mouse, etc.

[0021] In operation, the processor 102 may be configured to execute instructions stored in the storage medium 104 to perform various operations related to the 3D reconstruction method as detailed in the following description. The 3D reconstruction method of one or more objects can be used in any suitable application requiring a 3D model output based on a 2D image input. As an example, the 3D reconstruction can be applied to games, such as creating and modifying 3D game objects, designing game logic or story plots, virtual representation of real objects in a game environment, etc. The games can be dress-up games, makeover games, home design games, or other suitable games. In another example, the 3D reconstruction can be used to create / update the appearance of a character in a virtual meeting, dress up or makeover in an online shopping scenario, etc.

[0022] As shown in Fig. 2, a 3D reconstruction method consistent with an embodiment of the present disclosure includes the following processes. The method can be implemented, for example, by the computing system 100 shown in Fig. 1. Fig. 3 shows a framework of a 3D reconstruction process consistent with an embodiment of the present disclosure. The method can be implemented according to the framework shown in Fig. 3.

[0023] In S202, two-dimensional (2D) pixel coordinates and ray directions of pixels in a 2D image of an object are obtained. The 2D image may be, for example, the image 302 shown in FIG.

[0024] In some embodiments, multiple 2D pixel coordinates of multiple pixels in the 2D image are acquired. In some embodiments, for 3D image reconstruction, two or more 2D images of the object are acquired corresponding to two or more different directions of light rays, where the light rays may be understood as light rays from the camera.

[0025] TIFF2025513899000002.tif68169

[0026] In S204, the 2D pixel coordinates are projected into a 3D space to obtain a plurality of 3D points on the light rays. In some embodiments, all or a portion of a plurality of 2D pixels of the 2D image may be projected into the same 3D space according to the light rays corresponding to the 2D pixels.

[0027] TIFF2025513899000003.tif38169

[0028] At S206, for each 3D point of the plurality of 3D points, a signed distance value for the 3D point is predicted. The signed distance value for the 3D point indicates a signed distance from the 3D point to a mesh surface. In some embodiments, the mesh surface may be composed of a collection of polygonal faces, such as triangles, that together form a surface covering of the object.

[0029] TIFF2025513899000004.tif23168

[0030] TIFF2025513899000005.tif71168

[0031] In S208, a validity probability of the 3D point is predicted, which indicates the probability that the 3D point has a valid signed distance value.

[0032] FIG. 4 illustrates a surface representation according to some embodiments of the present disclosure. As shown in FIG. 4, in some embodiments, watertight 3D shapes can be implicitly represented by a signed distance function (SDF). A watertight surface or watertight shape is a closed surface or shape. A closed surface is a surface that has no edges or boundaries. For example, a closed surface may include a sphere, a torus, or a Klein bottle. Existing techniques based on signed distance functions (SDFs) are limited to watertight shapes. As shown on the left side of FIG. 4, a marching cubes algorithm can extract a closed surface from an SDF. The present disclosure provides an embodiment in which a 3D surface with any topology, including both watertight and open surfaces, can be reconstructed according to a combination of a signed distance function and a validity probability function, as shown on the right side of FIG. 4.

[0033] TIFF2025513899000006.tif14168

[0034] In some embodiments, in response to the predicted validity probability of one of the plurality of 3D points being less than a preset threshold, an invalid value is assigned to the predicted signed distance value of one of the plurality of 3D points. The preset threshold may be 0.5, 0.6, 0.7, 0.8, 0.9, etc. The preset threshold may be any suitable value between 0 and 1.

[0035] TIFF2025513899000007.tif38169

[0036] TIFF2025513899000008.tif63169

[0037] In some embodiments, a rendering weight for a 3D point is calculated based on the predicted signed distance value and the predicted validity probability of the 3D point.

[0038] In some embodiments, predicting the rendering weight of the 3D point may include determining a probability density function that maps the predicted signed distance value of the 3D point to a volume density, determining a volume weighting function based on the volume density and the validity probability of the 3D point, and determining a rendering weight for the 3D point based on the volume weighting function.

[0039] In watertight surface reconstruction, the rendering process ignores 3D points that are within the range of the camera ray exiting the surface from the inside out. In volume rendering, the volume weight is set to zero. In surface rendering, the points are ignored. This operation does not result in a pixel being mis-rendered, since the renderer has already rendered the surface when the ray enters the surface from the outside in.

[0040] For open surfaces, inside and outside may not be clearly defined: a ray can go directly from "pseudo" inside to "pseudo" outside without going "pseudo" inside. To render all surfaces, each valid surface point is rendered if a ray enters the surface from "pseudo" outside towards "pseudo" inside, and each valid surface point is rendered if a ray exits the surface from "pseudo" inside towards "pseudo" outside.

[0041] To apply volume rendering methods to the reconstruction, a probability density function may be chosen that maps a signed distance field (SDF) to a volume density. In some embodiments, the rendering behaves the same when a ray enters the surface from "pseudo" outside to "pseudo" inside as when a ray exits the surface from "pseudo" inside to "pseudo" outside.

[0042] In some embodiments, determining a probability density function that maps the predicted signed distance values ​​of the 3D points to a volume density includes determining the probability density function based on a sigmoid mapping function, the predicted signed distance values, and a sign adjustment function. The sign adjustment function is configured to change the predicted signed distance values ​​of the 3D points from predicted signed distance values ​​in a first direction to predicted signed distance values ​​in a second direction, the first direction being opposite to the second direction. For example, the predicted signed distance values ​​are negative values, which indicate that the first direction of the predicted signed distance is from inside to outside the boundary. The sign adjustment function is used to change the predicted signed distance values ​​to positive values, which indicate a direction opposite to the first direction.

[0043] TIFF2025513899000009.tif29169

[0044] TIFF2025513899000010.tif24169

[0045] TIFF2025513899000011.tif25169

[0046] TIFF2025513899000012.tif18169

[0047] TIFF2025513899000013.tif23168

[0048] TIFF2025513899000014.tif14168

[0049] In some embodiments, the weighting function 312, denoted as w(t) on the ray, may be defined based on a signed distance field (SDF) of the scene. In some embodiments, the weighting function may satisfy at least one of an unbiased requirement or an occlusion consideration requirement. In some embodiments, the weighting function may satisfy both the unbiased requirement and the occlusion consideration requirement.

[0050] TIFF2025513899000015.tif20168

[0051] TIFF2025513899000016.tif39168

[0052] In some embodiments, to derive the opacity density ρ(t), a simple case where there is only one plane in the sampling space may first be considered, and then ρ(t) is generalized to the general case of multiple surface intersections.

[0053] TIFF2025513899000017.tif94168

[0054] TIFF2025513899000018.tif19168

[0055] TIFF2025513899000019.tif18168

[0056] TIFF2025513899000020.tif19168

[0057] TIFF2025513899000021.tif23168

[0058] TIFF2025513899000022.tif109168

[0059] TIFF2025513899000023.tif84168

[0060] In some embodiments, calculating a rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point includes calculating a discrete opacity value for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point and the predicted signed distance value and the predicted validity probability of one of a plurality of 3D points neighboring the 3D point on the light ray.

[0061] TIFF2025513899000024.tif19168

[0062] TIFF2025513899000025.tif16168

[0063] JPEG2025513899000026.jpg19169

[0064] JPEG2025513899000027.jpg30169

[0065] In S210, the intensity values ​​of the 3D points are predicted.

[0066] JPEG2025513899000028.jpg29169

[0067] In some embodiments, the intensity value of a 3D point may include an RGB value including an intensity value of a red channel, an intensity value of a green channel, and an intensity value of a blue channel. In some embodiments, the intensity value of a 3D point may include one or more intensity values ​​of the channels of other color formats such as CMYK, HEX, or grayscale. In some embodiments, the intensity value of a 3D point may be represented using other suitable methods.

[0068] JPEG2025513899000029.jpg39169

[0069] JPEG2025513899000030.jpg22169

[0070] In S212, a rendering loss is obtained based on the predicted signed distance values, validity probabilities, and intensity values ​​of the multiple 3D points to update the 3D reconstruction network.

[0071] TIFF2025513899000031.tif168169

[0072] In some embodiments, the 3D reconstruction network may be updated with or without an image mask.

[0073] In some embodiments, the 3D reconstruction network may be implemented using either multiple perceptual layers or one-dimensional convolutional layer(s).

[0074] In S214, a 3D model of the object is extracted based on the updated 3D reconstruction network.

[0075] In some embodiments, a 3D model includes a mathematical coordinate-based representation of any surface of an object in three dimensions, including edges, vertices, polygons, etc.

[0076] In some embodiments, the reconstructed surface of the object can be extracted as a zero-level isosurface of the SDF. An isosurface is a surface that represents a point of constant value within a volume of space. The isosurface can be extracted directly using the classical Marching Cube (MC) algorithm. The presence of invalid signed distance values ​​can prevent MC from extracting a valid isosurface at locations that do not contain features. After the marching cubes computation, all invalid vertices and faces generated by the null cube can be removed. The remaining vertices and faces are the result of the mesh generation.

[0077] To evaluate the 3D reconstruction, we compared the 3D model reconstructed by the 3D reconstruction network disclosed herein with the 3D reconstruction method proposed by Wang et al. in “NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” NeurIPS (2021) (hereinafter, [Neus]).

[0078] FIG. 5 shows a visual comparison of 3D reconstructions on watertight surfaces consistent with embodiments of the present disclosure. As shown in FIG. 5, an image of a 3D reconstruction object is shown in the first row. The second row of FIG. 5 shows a 3D model of the object reconstructed using the 3D reconstruction method of the present disclosure. The third row of FIG. 5 shows a 3D model of the object reconstructed using the 3D reconstruction method of Neus. Table 1 shows a quantitative comparison between the watertight surface data using the 3D reconstruction method of the present disclosure and the watertight surface data using the 3D reconstruction method of Neus, respectively. As shown in Table 1, on watertight surfaces, the 3D reconstruction method of the present disclosure generally has smaller numerical errors than the 3D reconstruction method of Neus. [Table 1]

[0079] FIG. 6 shows a visual comparison of 3D reconstructions on open surfaces, consistent with certain embodiments of the present disclosure, with the front surface highlighted in blue and the back surface highlighted in pink. As shown in FIG. 6, the first row shows an image of the 3D reconstruction object. The second row of FIG. 6 shows the ground truth of the object. The third row of FIG. 6 shows a 3D model of the object reconstructed using the 3D reconstruction method of the present disclosure, and the fourth row of FIG. 6 shows a 3D model of the object reconstructed using the 3D reconstruction method in Neus. As shown in FIG. 6, the 3D reconstruction method of the present disclosure reconstructs a high-fidelity one-layer surface, whereas the prior art reconstructs a two-layer surface.

[0080] A quantitative comparison between open surface data using the disclosed 3D reconstruction method and the Neus 3D reconstruction method is shown in Table 2. As shown in Table 2, for open surfaces, the disclosed 3D reconstruction method has smaller numerical errors compared to existing techniques. [Table 2]

[0081] Existing techniques based on signed distance functions (SDFs) are limited to watertight shapes. This disclosure provides embodiments that can reconstruct 3D surfaces with arbitrary topologies, including both watertight and open surfaces, according to a combination of signed distance functions and validity probability functions.

[0082] The sequence numbers of the above-described embodiments of the present disclosure are merely for illustrative purposes and do not indicate the priority of the embodiments.

[0083] When the integrated unit in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in the above-mentioned computer-readable storage medium. Based on such understanding, the technical solution of the present disclosure may be essentially implemented in the form of a software product, or a part that contributes to the related art, or all or a part of the technical solution. The computer software product is stored in a storage medium and includes a plurality of instructions for instructing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or a part of the steps of the method in the embodiment of the present disclosure.

[0084] In the above-mentioned embodiments of the present disclosure, the description of the embodiments has different emphasis. For parts not described in detail in one embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] In some embodiments provided in the present disclosure, it should be understood that the disclosed client can be implemented in other ways. The above-mentioned device embodiments are merely examples. For example, the division of units is merely a division of logical functions, and other division methods can be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some functions can be omitted or not performed. Furthermore, the coupling or direct coupling or communication connection between components shown or discussed can be an indirect coupling or communication connection via some interfaces, units, or modules, and can be in electrical or other form.

[0086] The units described as separate parts may or may not be physically separable. The parts shown as units may or may not be physical units, and may be located in one location or distributed across multiple network units. Some or all of the units may be selected according to the actual requirements for achieving the purpose of the solution in the embodiment.

[0087] In addition, the functional units in the embodiments of the present disclosure may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or a software functional unit.

[0088] Although the principle and embodiment of the present disclosure are described herein using specific embodiments, the above description of the embodiments is intended only to help understand the method and the core idea of ​​the method of the present disclosure. Meanwhile, those skilled in the art may modify the specific embodiment and application scope according to the idea of ​​the present disclosure. In conclusion, the contents of this specification should not be interpreted as limiting the present disclosure.

Claims

1. A three-dimensional (3D) reconstruction method, comprising: Obtaining a two-dimensional (2D) pixel coordinate and a ray direction for a pixel in a 2D image of an object; projecting the 2D pixel coordinates into 3D space to obtain a number of 3D points on the ray; For each 3D point of the plurality of 3D points, predicting a signed distance value for the 3D point, the signed distance value for the 3D point indicating a signed distance from the 3D point to a mesh surface; predicting a validity probability for the 3D point, the validity probability for the 3D point indicating the probability that the 3D point has a valid signed distance value; predicting intensity values ​​of said 3D points; obtaining a rendering loss based on the predicted signed distance value, the validity probability, and intensity values ​​of a plurality of the 3D points and updating a 3D reconstruction network; extracting a 3D model of the object based on the updated 3D reconstruction network; and The method includes:

2. Extracting a 3D model of the object based on the updated 3D reconstruction network includes:

2. The method of claim 1, comprising: in response to the predicted validity probability of one of the plurality of 3D points being less than a pre-determined threshold, assigning an invalid value to the predicted signed distance value of one of the plurality of 3D points; and extracting the 3D model of the object by removing vertices having the invalid signed distance value.

3. The method of claim 1 , further comprising: for a 3D point of the plurality of 3D points, calculating a rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability for the 3D point.

4. Calculating a rendering weight for a 3D point based on a predicted signed distance value and a predicted validity probability for the 3D point, comprising: determining a probability density function that maps predicted signed distance values ​​of the 3D points to volumetric density; determining a volume weighting function based on volume density and validity probability of the 3D points; and determining a rendering weight for the 3D point based on a volume weighting function.

5. Determining the probability density function that maps the predicted signed distance values ​​of the 3D points to the volume density may include: determining the probability density function based on a sigmoid mapping function, the predicted signed distance value, and a sign adjustment function; 5. The method of claim 4, wherein the sign adjustment function is configured to change the predicted signed distance value of the 3D point from the predicted signed distance value in a first direction to a predicted signed distance value in a second direction, the first direction being opposite to the second direction.

6. Calculating the rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability for the 3D point, 4. The method of claim 3, comprising calculating a discrete opacity value for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point and the predicted signed distance value and the predicted validity probability of one of the plurality of 3D points neighboring the 3D point.

7. predicting a mask based on a plurality of the rendering weights of the plurality of 3D points; Obtaining the rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points to update the 3D reconstruction network includes: The method of claim 3 , comprising obtaining the rendering loss based on a prediction mask to update the 3D reconstruction network.

8. Predicting the mask based on a plurality of the rendering weights of the plurality of 3D points includes: The method of claim 7 , comprising predicting the mask based on an accumulation of a plurality of the rendering weights of the plurality of 3D points.

9. Obtaining a rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points to update the 3D reconstruction network, calculating a rendering probability loss based on the validity probabilities of the plurality of 3D points and a binary cross-entropy of the validity probabilities; and obtaining the rendering loss according to the rendering probability loss.

10. Obtaining the rendering loss based on the predicted signed distance values, the validity probabilities, and the intensity values ​​of the plurality of 3D points to update the 3D reconstruction network includes: computing a rendering probability regularization loss based on an average of the validity probabilities of the plurality of 3D points; and obtaining the rendering loss according to the rendering probability regularization loss.

11. An apparatus for three-dimensional (3D) reconstruction, comprising: A memory for storing a computer program; Obtaining a two-dimensional (2D) pixel coordinate and a ray direction for a pixel in a 2D image of an object; projecting the 2D pixel coordinates into 3D space to obtain a number of 3D points on the ray; For each 3D point of the plurality of 3D points, predicting a signed distance value for the 3D point, the signed distance value for the 3D point indicating a signed distance from the 3D point to a mesh surface; predicting a validity probability for the 3D point, the validity probability for the 3D point indicating the probability that the 3D point has a valid signed distance value; predicting intensity values ​​of said 3D points; obtaining a rendering loss based on the predicted signed distance value, the validity probability, and the intensity values ​​of the plurality of 3D points and updating a 3D reconstruction network; and extracting a 3D model of the object based on the updated 3D reconstruction network. An apparatus comprising: a processor.

12. The processor, 12. The apparatus of claim 11, further configured to, in response to the predicted validity probability of one of the plurality of 3D points being less than a preset threshold, assign an invalid value to the predicted signed distance value of one of the plurality of 3D points and extract the 3D model of the object by removing vertices having the invalid signed distance value.

13. The processor 12. The apparatus of claim 11, further configured to: for a 3D point of the plurality of 3D points, calculate a rendering weight for the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point.

14. The processor, determining a probability density function that maps the predicted signed distance values ​​of the 3D points to volumetric density; determining a volume weighting function based on the volume density and the validity probability of the 3D points; The apparatus of claim 13 , further configured to determine a rendering weight for the 3D point based on the volume weighting function.

15. The processor, 15. The apparatus of claim 14, further configured to determine the probability density function based on a sigmoid mapping function, the predicted signed distance values, and a sign adjustment function, the sign adjustment function configured to change the predicted signed distance values ​​of the 3D points from the predicted signed distance values ​​in a first direction to predicted signed distance values ​​in a second direction, the first direction being opposite to the second direction.

16. The processor, 14. The apparatus of claim 13, further configured to calculate a discrete opacity of the 3D point based on the predicted signed distance value and the predicted validity probability of the 3D point and the predicted signed distance value and the predicted validity probability of one of the plurality of 3D points neighboring the 3D point.

17. The processor, predicting a mask based on a plurality of the rendering weights of the plurality of 3D points; The apparatus of claim 15 , further configured to determine the rendering loss based on the predicted mask and to update the 3D reconstruction network.

18. The processor, calculating a rendering probability loss based on the validity probabilities of the plurality of 3D points and a binary cross-entropy of the validity probabilities; The apparatus of claim 11 , further configured to obtain the rendering loss according to the rendering probability loss.

19. The processor, Calculate a rendering probability regularization loss based on an average of the validity probabilities of the plurality of 3D points; The apparatus of claim 11 , further configured to obtain the rendering loss according to the rendering probability regularization loss.

20. When executed by a processor, the processor: Obtaining a two-dimensional (2D) pixel coordinate and a ray direction for a pixel in a 2D image of an object; projecting the 2D pixel coordinates into a three-dimensional (3D) space to obtain a number of 3D points on the ray; For each 3D point of the plurality of 3D points, predicting a signed distance value for the 3D point, the signed distance value for the 3D point indicating a signed distance from the 3D point to a mesh surface; predicting a validity probability of the 3D point, the validity probability of the 3D point indicating the probability that the 3D point has a valid signed distance value; predicting intensity values ​​of said 3D points; obtaining a rendering loss based on the predicted signed distance value, the validity probability, and the intensity values ​​of the plurality of 3D points and updating a 3D reconstruction network; and extracting a 3D model of the object based on the updated 3D reconstruction network.

Images