Model training method and device, three-dimensional reconstruction method and device, equipment and storage medium

Abstract

The invention provides a model training method and device, a three-dimensional reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: based on a bounding box of a target object, adopting an initial surface reconstruction network to obtain initial three-dimensional surface data of the target object; obtaining a symbol distance parameter and a normal vector of each vertex in a plurality of initial triangular patches in the initial three-dimensional surface data; projecting each vertex of the initial triangular patch according to the symbol distance parameter and the normal vector of each vertex of the initial triangular patch to obtain a projected triangular patch; performing surface rendering according to the projected triangular patch under each view angle to obtain rendering colors of the pixel points under the view angles; and training the initial surface reconstruction network according to the rendering color of the pixel points and the true value color of each pixel point to obtain a target surface reconstruction network. According to the embodiment of the invention, the details of the three-dimensional reconstructed object surface are enriched, and the accuracy of the three-dimensional reconstruction of the complex object curved surface can be ensured.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a model training method, a three-dimensional reconstruction method, and apparatus, equipment, and storage medium thereof. Background Art

[0002] Multi-view 3D reconstruction technology can reconstruct 3D models from 2D images collected from multiple viewpoints. 3D reconstruction technology plays an important role in 3D games, virtual reality, augmented reality and other fields.

[0003] In recent years, implicit surface reconstruction algorithms based on neural rendering have developed rapidly and are playing a vital role in the field of 3D content generation. Currently, this type of technology uses surface reconstruction algorithms to estimate the Signed Distance Function (SDF) value of each sampling point within the object's bounding box. This is then used to extract the object's 3D surface data as an explicit 3D mesh model, which is then rendered using the surface.

[0004] However, since the sampling of surface points in surface rendering methods is mostly approximately linear, this limits the sampling accuracy to a certain extent and cannot accurately estimate the curved surface of complex objects. This makes the surface of the 3D reconstructed object incomplete, with many geometric errors and insufficient surface details. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a model training method, a three-dimensional reconstruction method, and its apparatus, equipment, and storage medium to improve the reconstruction accuracy and the completeness of the model details.

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0007] Based on the bounding box of the target object, a preset initial surface reconstruction network is used to reconstruct the surface of the target object to obtain initial three-dimensional surface data of the target object, wherein the initial three-dimensional surface data includes: a plurality of initial triangular facets;

[0008] Obtaining signed distance parameters and normal vectors of each vertex in the plurality of initial triangles;

[0009] Projecting each vertex of each initial triangle according to the signed distance parameter and the normal vector of each vertex of the initial triangle to obtain a projected triangle;

[0010] Performing surface rendering based on the projected triangles at each viewing angle to obtain the rendering color of each pixel at the viewing angle;

[0011] The initial surface reconstruction network is trained according to the rendered color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle to obtain a target surface reconstruction network.

[0012] In a second aspect, an embodiment of the present application further provides a three-dimensional reconstruction method, the method comprising:

[0013] Based on the bounding box of the target object, a pre-trained surface reconstruction network is used to reconstruct the surface of the target object to extract three-dimensional surface data of the target object; wherein the surface reconstruction network is a target surface reconstruction network obtained by the model training method according to any one of the first aspects above;

[0014] Surface rendering is performed based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

[0015] In a third aspect, an embodiment of the present application further provides a model training device, comprising:

[0016] A first surface reconstruction module is configured to reconstruct the surface of the target object based on a bounding box of the target object and adopt a preset initial surface distance extraction network to obtain initial three-dimensional surface data of the target object, wherein the initial three-dimensional surface data includes: a plurality of initial triangular facets;

[0017] An acquisition module, configured to acquire a signed distance parameter and a normal vector of each vertex in the plurality of initial triangular facets;

[0018] A projection module, configured to project each vertex of each initial triangular facet according to the signed distance parameter and the normal vector of each vertex of the initial triangular facet to obtain a projected triangular facet;

[0019] A first surface rendering module is used to perform surface rendering according to the projected triangle patch at each viewing angle to obtain a rendering color of each pixel at the viewing angle;

[0020] A training module is used to train the initial surface reconstruction network according to the rendering color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle to obtain a target surface reconstruction network.

[0021] In a fourth aspect, an embodiment of the present application further provides a three-dimensional reconstruction device, comprising:

[0022] a second surface reconstruction module, configured to reconstruct the surface of the target object based on the bounding box of the target object using a pre-trained surface reconstruction network to extract three-dimensional surface data of the target object; wherein the surface reconstruction network is a target surface reconstruction network obtained using the model training method described in the first aspect above;

[0023] The second surface rendering module is configured to perform surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

[0024] In the fifth aspect, an embodiment of the present application also provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the model training method as described in any one of the first aspects or the steps of the three-dimensional reconstruction method described in the second aspect.

[0025] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method described in any one of the first aspects or the steps of the three-dimensional reconstruction method described in the second aspect are executed.

[0026] The model training method, 3D reconstruction method, and apparatus, equipment, and storage medium provided in the embodiments of the present application can obtain projected triangles by projecting the signed distance parameters and normal vectors of each vertex in the initial triangle before surface rendering the triangles, thereby achieving fine-tuning of the triangles in the 3D surface data, and realizing nonlinear sampling of the object surface with the help of surface rendering, thereby improving the sampling accuracy and precision in the surface rendering process, and maximizing the training accuracy of the surface reconstruction network, thereby improving the surface integrity of the 3D reconstructed object, avoiding geometric feature errors on the object surface, enriching the surface details of the 3D reconstructed object, and ensuring the accuracy of 3D reconstruction of the curved surface of complex objects.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0030] Figure 2A schematic diagram of an embodiment of the present application providing a method for obtaining initial three-dimensional surface data using an initial surface distance extraction network and an isosurface extraction algorithm;

[0031] Figure 3 A schematic diagram of projecting an initial triangular facet provided in an embodiment of the present application;

[0032] Figure 4 A flow chart of a surface rendering method in a model training method provided in an embodiment of the present application;

[0033] Figure 5 A flow chart of a method for calculating a second target three-dimensional point in a model training method provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of obtaining the rendering color of a pixel through surface rendering during a model training process provided in an embodiment of the present application;

[0035] Figure 7 A flowchart of a method for constructing a network in a model training method provided in an embodiment of the present application;

[0036] Figure 8 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0037] Figure 9 A flowchart of a method for projecting target triangular facets in a three-dimensional reconstruction method provided as an example of this application;

[0038] Figure 10 A flow chart of a method for surface rendering in a three-dimensional reconstruction method provided as an example of this application;

[0039] Figure 11 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0040] Figure 12 A schematic structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0041] Figure 13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0043] The 3D reconstruction method in one embodiment of the present disclosure can be applied to any application scenario involving virtual 3D models, such as 3D gaming scenarios, virtual scenes based on virtual reality, and virtual scenes based on augmented reality. During implementation, the 3D reconstruction method can be run on a local terminal device or a server. When the 3D reconstruction method is run on a server, it can be implemented and executed based on a cloud interaction system, where the cloud interaction system includes a server and a client device.

[0044] To facilitate a better understanding of the three-dimensional reconstruction method provided in the embodiments of the present application, the technical terms involved in the following embodiments of the present application are first explained below.

[0045] Technical term explanation:

[0046] SDF: Signed Distance Function (SDF), which is an implicit representation of the surface of a 3D model. It can determine the distance from a 3D point to the surface of an object in a finite region in 3D space and also defines the sign of the distance. For example, if the point is inside the object, it is negative; if the point is outside the object, it is positive; if the point is on the surface of the object, it is 0.

[0047] MLP: Multilayer Perceptron (MLP), a type of neural network architecture.

[0048] BBox: Bounding Box (BBox), refers to the minimum bounding box surrounding an object in three-dimensional (3D) space.

[0049] SFM: Structure From Motion (SFM) is a technique for estimating three-dimensional structure from a series of two-dimensional images containing visual motion information. SFM can be used to obtain camera parameters and sparse point clouds for each perspective.

[0050] Marching Cubes: Surface extraction algorithms, also known as surface rendering algorithms, are used to extract isosurfaces from volumetric data. The core concept is to define a surface by comparing the vertices to a user-specified threshold, after setting the 3D space and vertex scalar values. This algorithm then creates a triangular patch and connects all cube faces on the isosurface boundary to form a surface.

[0051] Dmtet (Deep Marching Tetrahedra): A differentiable isosurface extraction algorithm. In this algorithm, the surface of an object is represented as a tetrahedral mesh, which can be converted to a mesh using Marching Tetrahedra (similar to Marching Cubes). Each vertex in the tetrahedral mesh stores the SDF value and local deformation vector, and both the SDF value and the local deformation vector are optimized using a neural network.

[0052] Nvdiffrast: An automatic differentiation framework for rasterization of triangular meshes.

[0053] In order to clearly describe the three-dimensional reconstruction method provided in the embodiment of the present application, the model training method provided in the embodiment of the present application is first explained and illustrated by examples through multiple embodiments in combination with multiple drawings, so as to clearly illustrate the pre-trained network model used in the three-dimensional reconstruction method.

[0054] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the model training method may include:

[0055] S101 , based on the bounding box of the target object, a preset initial surface reconstruction network is used to reconstruct the surface of the target object to obtain initial three-dimensional surface data of the target object, where the initial three-dimensional surface data includes: a plurality of initial triangular facets.

[0056] Before performing this step, a bounding box of the target object may be obtained. In one implementation example, the bounding box of the target object can be automatically estimated based on multiple pre-acquired two-dimensional images of the target object from multiple perspectives. For example, the multiple two-dimensional images can be obtained by pre-photographing the target object in a real environment from multiple perspectives using a preset camera. For example, a preset pose estimation algorithm can be used to perform joint pose estimation on the multiple two-dimensional images to obtain three-dimensional point clouds from multiple perspectives and camera intrinsic and extrinsic parameters from multiple perspectives. Then, based on the three-dimensional point clouds and camera intrinsic and extrinsic parameters from multiple perspectives, a bounding box of the target object is estimated to obtain information about the target object's bounding box. The three-dimensional point cloud from each perspective can be, for example, a sparse point cloud. The preset pose estimation algorithm can be, for example, an SFM network. In another implementation example, the bounding box of the target object can be obtained based on bounding box information input by a user. The bounding box information of the target object may include the center position of the target object and the radius of the object.

[0057] The initial surface reconstruction network may include, for example, an initial surface distance extraction network, such as an initial SDF network, and a preset isosurface extraction algorithm, such as the Marching Cubes algorithm. During the surface reconstruction process performed in this step, the initial surface distance extraction network may be used to process the sampling points within the bounding box of the target object to obtain the distance sign parameters and normal vectors of the sampling points. Based on the distance sign parameters and normal vectors of the sampling points, the preset isosurface extraction algorithm may be used to perform a three-dimensional reconstruction of the target object to obtain initial three-dimensional surface data of the target object's three-dimensional surface data. The initial three-dimensional surface data includes a plurality of initial triangular facets. The initial three-dimensional surface data may also be referred to as an explicit three-dimensional mesh model based on triangular facets.

[0058] To better understand the extraction of three-dimensional surface data, the following description is continued with reference to the accompanying drawings. Figure 2 This is a schematic diagram of an embodiment of the present application providing a method of obtaining initial three-dimensional surface data using an initial surface distance extraction network and an isosurface extraction algorithm. Figure 2 As shown in the figure, after the signed distance parameters of the sampling points are obtained using the SDF network, the surface extraction algorithm can be used to extract the surface. Figure 2 The initial three-dimensional surface data of the target object shown may also be called a surface mesh.

[0059] S102: Obtain the signed distance parameter and normal vector of each vertex in the plurality of initial triangular facets.

[0060] For example, when each initial triangular facet is obtained, the coordinates of each vertex in each initial triangular facet can be input into the initial surface distance extraction network to obtain the parameter extraction results of each vertex, which at least include: signed distance parameters and normal vectors. In this way, the signed distance parameters and normal vectors of each vertex are obtained.

[0061] S103 : Projecting each vertex of each initial triangle according to the signed distance parameter and the normal vector of each vertex of each initial triangle to obtain a projected triangle.

[0062] For each vertex of the initial triangle, that is, the triangle mesh, the signed distance parameter and the normal vector of each vertex can be used to perform the closest point projection of the closest signed distance parameter on each vertex of the initial triangle. By projecting each vertex in the initial triangle, the initial triangle is projected to obtain the projected triangle. By projecting the triangle, the triangle can be fine-tuned based on the signed distance parameter.

[0063] In a possible implementation example, each vertex can be translated along the opposite direction of the normal vector of each vertex according to the signed distance parameter of each vertex to obtain the translated vertices; the projected triangular face is: a triangular face formed by the translated vertices.

[0064] To better understand the projection of triangular facets, let's continue with the following illustrations. Figure 3 This is a schematic diagram of projecting the initial triangular facets provided in the embodiment of the present application. Figure 3 As shown, for the vertex x in the initial triangle, the signed distance parameter d, normal vector n and geometric feature vector z of vertex x can be obtained by inputting the initial SDF network, wherein the geometric feature vector z can be used to characterize the surface feature attributes of the object at vertex x. Each vertex x is translated in the opposite direction of the normal vector n, and the translation distance is the corresponding signed distance parameter d to obtain the projected vertex x'. The multiple projected vertices constitute the projected triangle. For example, the first vertex x0 is projected and translated to obtain the first projected vertex x'0, the second vertex x1 is projected and translated to obtain the second projected vertex x'1, and the third vertex x2 is projected and translated to obtain the third projected vertex x'2. Referring to the accompanying drawings, it can be seen that the initial triangle is the neighborhood triangle of the extracted 0 isosurface, and the projected triangle is the 0 isosurface. Based on this, it can be seen that based on the signed distance parameters and normal vectors of each vertex of each initial triangle, the vertices of the initial triangle are projected, so that the projected triangle is closer to the 0 isosurface of the object surface and closer to the real surface of the object.

[0065] S104 , performing surface rendering according to the projected triangles at each viewing angle to obtain the rendering color of each pixel at that viewing angle.

[0066] In a specific implementation, at each viewing angle, a preset surface rendering network may be used to perform surface rendering based on the projected triangular facets to obtain the rendering color of each pixel at that viewing angle.

[0067] S105 , training the initial surface reconstruction network according to the rendered color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle to obtain a target surface reconstruction network.

[0068] For example, the model loss function value can be calculated based on the rendering color of each pixel under each viewing angle and the error of the true color of each pixel in the color image under each viewing angle, and then based on the model loss function value, the sampling gradient descent method is used to adjust the parameters of the initial surface distance extraction network, and the model is trained according to the initial surface reconstruction network after the parameter adjustment until the preset stop iteration condition is met to obtain the target surface reconstruction network. The stop iteration condition can be, for example, reaching a preset maximum number of iterations, or the model loss function value reaches a preset condition. The parameter adjustment of the initial surface reconstruction network can specifically include: adjusting the parameters of the initial surface distance extraction network, i.e., the initial SDF network, so that when the preset stop iteration condition is met, the target surface reconstruction network including: the target surface distance extraction network can be obtained.

[0069] The model training method provided in this embodiment can reconstruct the surface of the target object based on the bounding box of the target object and adopt a preset initial surface reconstruction network to obtain the initial three-dimensional surface data of the target object. The signed distance parameters and normal vectors of each vertex in multiple initial triangular facets in the initial three-dimensional surface data are obtained, and the vertices of the initial triangular facets are projected according to the signed distance parameters and normal vectors of each vertex of each initial triangular facet to obtain projected triangular facets. The surface is then rendered according to the projected triangular facets at each viewing angle to obtain the rendering color of each pixel point at the viewing angle. Therefore, the initial surface reconstruction network can be trained according to the rendering color of each pixel point at the viewing angle and the true value color of each pixel point in the color image at the viewing angle to obtain the target surface reconstruction network. In the model training method provided in this embodiment, during the model training process, before the surface rendering of the triangle, the signed distance parameters and normal vectors of each vertex in the initial triangle are used to project each vertex to obtain the projected triangle, thereby achieving fine-tuning of the triangle in the three-dimensional surface data, and with the help of surface rendering, achieving nonlinear sampling of the object surface, thereby improving the sampling accuracy and precision in the surface rendering process, and improving the training accuracy of the surface reconstruction network as much as possible, thereby improving the surface integrity of the three-dimensional reconstructed object, avoiding geometric feature errors on the object surface, enriching the surface details of the three-dimensional reconstructed object, and ensuring the accuracy of the three-dimensional reconstruction of the complex object surface.

[0070] In order to more clearly understand the model training method provided in the embodiments of the present application, the surface rendering based on the projected triangular facets is described in detail below in combination with multiple embodiments. Figure 4 This is a flow chart of a surface rendering method in a model training method provided in an embodiment of the present application. Figure 4 As shown in S104 above, surface rendering is performed according to the projected triangles at each viewing angle to obtain the rendering color of each pixel at each viewing angle, which may include:

[0071] S401 , performing surface rendering on the projected triangular facets under a viewing angle to obtain intersection information between each pixel point under the viewing angle and the projected triangular facets.

[0072] The intersection information indicates whether a three-dimensional intersection exists between the observation ray corresponding to each pixel point and the projected triangle facet. In this embodiment, the surface rendering network used in the surface rendering of the projected triangle facet may include, for example, a preset surface rendering algorithm and an initial color prediction network.

[0073] For example, at this viewing angle, a preset surface rendering algorithm can be used to perform surface rendering on the projected triangular facets to obtain intersection information between each pixel point and the projected triangular facets at the viewing angle. The preset surface rendering algorithm can be: a preset differentiable surface rendering algorithm, such as a differentiable rasterization rendering algorithm. Among them, each pixel point at the viewing angle can be a pixel point in the two-dimensional image at the viewing angle, and the observation ray corresponding to each pixel point at the viewing angle can be: an observation ray between the equivalent position of each pixel point at the viewing angle and the position of the observation camera at the viewing angle.

[0074] By performing surface rendering on the projected triangular facets, the intersection detection between the observation ray corresponding to each pixel point at each viewing angle and the projected triangular facets is realized, and intersection information is obtained to indicate whether there is a three-dimensional intersection.

[0075] S402 : Determine the coordinates of the target three-dimensional point on the observation ray corresponding to each pixel point according to the intersection information.

[0076] The following examples are provided in conjunction with specific situations of intersection information. In one example, if the intersection information indicates that there is a 3D intersection between the first pixel point corresponding to the first observation ray and the projected triangle facet under the viewing angle, the coordinates of the 3D intersection are the coordinates of the first target 3D point on the first observation ray. The first target 3D point, i.e., the 3D intersection point, can be represented by x p1 .

[0077] In another example, if the intersection information indicates that there is no 3D intersection between the second observation ray and the projected triangle facet corresponding to the second pixel point under the viewing angle, then based on the second observation ray and the coordinates of the viewing camera under the viewing angle, the coordinates of the 3D background point are predicted as the coordinates of the second target 3D point on the second observation ray. The second target 3D point, that is, the predicted 3D background point, can be expressed as

[0078] S403 : Based on the coordinates of the target three-dimensional point, an initial color prediction network is used to perform color prediction to obtain a predicted color of the target three-dimensional point.

[0079] When the coordinates of the target three-dimensional point are obtained, the coordinates of the target three-dimensional point can be input into the initial color prediction network for color prediction to obtain the predicted color of the target three-dimensional point.

[0080] In one possible implementation, the normal vector and geometric feature vector of the target 3D point, as well as the direction vector of the observation ray, can also be obtained. The normal vector and geometric feature vector of the target 3D point can be obtained by processing using a preset initial surface distance extraction network. The direction vector of the first observation ray can be obtained based on the equivalent position of each pixel point at a preset viewing angle in 3D space and the position of the observation camera at that viewing angle.

[0081] Once the coordinates, normal vector, geometric eigenvector, and direction vector of the observation ray are obtained for the target 3D point, they can be input into the initial color prediction network for color prediction, thereby obtaining the predicted color of the target 3D point. Since the coordinates, normal vector, geometric eigenvector, and direction vector of the observation ray represent the characteristics of the target 3D point in multiple dimensions, inputting them into the initial color prediction network improves the accuracy of color prediction, thereby making the predicted color of the target 3D point more accurate and further improving the training accuracy of the model.

[0082] Based on the different examples of the above-mentioned intersection information, in order to achieve color prediction in different example situations, a foreground color prediction network and a background color prediction network can be pre-constructed respectively. That is, the above-mentioned initial color prediction network may include: an initial foreground color prediction network and an initial background color prediction network.

[0083] In one example, if the intersection information indicates that there is a 3D intersection between the first observation ray and the projected triangle, the color prediction can be performed using the initial foreground color prediction network based on the coordinates of the first target 3D point to obtain the predicted color of the first target 3D point. In another example, if the intersection information indicates that there is no 3D intersection between the second observation ray and the projected triangle, the color prediction can be performed using the initial background color prediction network based on the coordinates of the second target 3D point to obtain the predicted color of the second target 3D point. The predicted color of the first target 3D point can be expressed as The predicted color of the second target 3D point can be expressed as

[0084] Of course, the same color prediction network can also be used to predict the color of 3D points. However, in this embodiment, the initial foreground color prediction network and the initial background color prediction network are used to predict the colors of the first and second target 3D points, that is, the foreground 3D points and the predicted background 3D points, respectively. This makes the predicted colors of the first and second target 3D points closer to reality, which in turn makes the rendered colors of pixels at various viewing angles more accurate, thereby improving the accuracy of model training.

[0085] It should be noted that in the process of color prediction of the first target three-dimensional point, the normal vector, geometric feature vector and direction vector of the first observation ray of the first target three-dimensional point can also be obtained, and the coordinates, normal vector, geometric feature vector and direction vector of the first observation ray of the first target three-dimensional point are all input into the initial foreground color prediction network to obtain the predicted color of the first target three-dimensional point.

[0086] Correspondingly, in the process of color prediction of the second target three-dimensional point, the direction vector of the second observation ray can also be obtained, and the coordinates of the second target three-dimensional point and the direction vector of the second observation ray are input into the initial background color prediction network to obtain the predicted color of the second target three-dimensional point.

[0087] S404: Determine the rendering color of each pixel under the viewing angle according to the predicted color of the target three-dimensional point.

[0088] In a specific implementation example, a preset color calculation formula may be used according to the predicted color of the target three-dimensional point to calculate the rendering color of each pixel at the viewing angle.

[0089] The method provided in the embodiment of the present application can obtain the intersection information of each pixel point under the viewing angle and the projected triangular facet during the surface rendering process of the projected triangular facet. Based on the intersection information combined with the initial color prediction network, color prediction is performed, making the rendering color of each pixel point under the viewing angle more accurate, thereby making the predicted color of the target three-dimensional point more accurate, further improving the training accuracy of the model. In addition, in this embodiment, during the model training process, the initial foreground color prediction network and the initial background color prediction network are respectively used to predict the color of the foreground three-dimensional points and the predicted three-dimensional points, further improving the model training accuracy in the unmasked case, so that the trained model can better achieve three-dimensional reconstruction of background objects.

[0090] It should be noted that for the initial color prediction network used in the surface rendering process, in the process of training the initial surface reconstruction network based on the rendering color of each pixel under the viewing angle and the true color of each pixel in the color image under the viewing angle, the initial color prediction network is also trained to obtain the target surface reconstruction network and the target color prediction network.

[0091] Furthermore, when the initial color prediction network includes an initial foreground color prediction network and an initial background color prediction network, the initial foreground color prediction network and the initial background color prediction network may also be trained during the training of the initial surface reconstruction network, thereby obtaining a target surface reconstruction network including a target surface distance extraction network, and a target color prediction network including a target foreground color prediction network and a target background color prediction network. In other words, the above-mentioned model training method may further include:

[0092] According to the rendered color of each pixel under the viewing angle and the true color of each pixel in the color image under the viewing angle, the initial foreground color prediction network and the initial background color prediction network are trained to obtain the target foreground color prediction network and the target background color prediction network to realize the training of the foreground color prediction network and the background color prediction network.

[0093] The following is an explanation of an implementation example of predicting a background 3D point, which is the second 3D target point, in the above method in combination with an embodiment. Figure 5 A flow chart of a method for calculating a second target three-dimensional point in a model training method provided in an embodiment of the present application, such as Figure 5 As shown, in the above method, based on the second observation ray and the coordinates of the observation camera of the viewing angle, predicting the coordinates of the three-dimensional background point as the coordinates of the second target three-dimensional point on the second observation ray may include:

[0094] S501 : Calculate a target translation distance of the observation camera position according to a second observation ray and the observation camera position of the viewing angle.

[0095] For example, the target translation distance of the observation camera position can be calculated using the following formula (1) based on the second observation ray and the observation camera position of the viewing angle.

[0096] F=-x o ·v p +1...Formula (1)

[0097] Among them, F is the target translation distance, x o is the observation camera position of the view, v p is the second observation ray, that is, the observation ray corresponding to the second pixel point under the viewing angle.

[0098] S502 : translating the observation camera position by a target translation distance along a second observation ray to obtain a predicted three-dimensional background point as a second target three-dimensional point.

[0099] For example, along the second observation ray, the following formula (2) can be used to translate the observation camera position by the target translation distance to obtain the predicted three-dimensional background point as the second target three-dimensional point.

[0100]

[0101] in, It can represent the predicted background 3D point, that is, the second target 3D point.

[0102] The calculation method for predicting three-dimensional background points provided in this embodiment can ensure that the coordinates of the predicted three-dimensional background points, that is, the second target three-dimensional points, are accurate, thereby improving the accuracy of the rendering color of pixel points under the viewing angle during the model training process and further improving the model training accuracy.

[0103] Regarding the rendering color of each pixel under the viewing angle, the following embodiments of the present application also provide some possible implementation methods for determining the rendering color of the pixel. In the above S401, after performing surface rendering on the projected triangle under the viewing angle and obtaining the intersection information of each pixel under the viewing angle and the projected triangle, the method may further include:

[0104] If the intersection information indicates that a three-dimensional intersection exists between the first observation ray and the projected triangle facet, an intersection mask corresponding to the first pixel point is determined as a first mask parameter.

[0105] Based on the intersection information, the binary intersection mask corresponding to the pixel point can be determined p If the intersection information indicates that a three-dimensional intersection exists, it is determined that the intersection mask corresponding to the first pixel point is 1, that is, the first mask parameter is 1.

[0106] If the intersection information indicates that there is no three-dimensional intersection point between the second observation ray and the projected triangle facet, the intersection mask corresponding to the second pixel point is determined as the second mask parameter.

[0107] Correspondingly, if the intersection information indicates that there is no three-dimensional intersection point, the intersection mask corresponding to the second pixel point is determined to be 0, that is, the second mask parameter is 0.

[0108] Accordingly, as shown in S404 above, determining the rendering color of each pixel under the viewing angle according to the predicted color of the target 3D point may include:

[0109] Weighting the predicted color of the first target three-dimensional point according to the first mask parameter to obtain a rendering color of the first pixel point under the viewing angle;

[0110] The predicted color of the second target three-dimensional point is weighted according to the second mask parameter to obtain the rendering color of the second pixel point under the viewing angle.

[0111] For example, whether it is the first pixel or the second pixel, the following formula (3) can be used to perform weighted prediction of the color to obtain the rendered color of the pixel.

[0112] c p =mask p c f +(1-mask p )...Formula (3)

[0113] Among them, c p is the rendering color of pixel p, mask p is the intersection mask of pixel p, c f is the predicted point corresponding to the target 3D point of the pixel point.

[0114] In this embodiment, the rendering color of the pixel point is obtained by weighting the intersection mask of the pixel point and the predicted color of the target three-dimensional point on the observation ray corresponding to the pixel point. Since the intersection mask is obtained based on the intersection information, the weighted rendering color is actually based on the color weighting of the foreground and background, which improves the accuracy of the pixel rendering color and the model training accuracy in the unmasked case, so that the trained model can better realize the three-dimensional reconstruction of background objects.

[0115] To better understand the application of the surface rendering algorithm, the foreground color prediction network, and the background color prediction network in the above embodiment, the following is an explanation with reference to the schematic diagram. Figure 6 This is a schematic diagram of obtaining the rendering color of a pixel point by surface rendering during a model training process provided by an embodiment of the present application. Figure 6As shown in the figure, after projecting the initial triangle skin to obtain the projected triangle face, the SDF network can be used to obtain the signed distance parameters, normal vectors and geometric eigenvectors of each vertex after projection, and differentiable raster rendering can be used to obtain the three-dimensional intersection point x between the pixel point p and the projected triangle face. p and the three-dimensional intersection point x p The corresponding binary intersection mask p , and according to the three-dimensional intersection x p The coordinates of the three-dimensional intersection point, the normal vector z p 、The geometric characteristic vector n of the three-dimensional intersection p And the direction vector v of the viewing ray under the viewing angle p , use the foreground color prediction network to get the three-dimensional intersection x p Rendering color Combined with binary intersection mask p Perform weighted addition to obtain the rendering color c of pixel p p If in differentiable raster rendering, there is no three-dimensional intersection x between the pixel point p and the projected triangle facet p , then the above formulas (1) and (2) can be used to predict the background points, and the predicted background points can be obtained Then, based on the predicted background points The coordinates of the ray and the direction vector v of the viewing angle p , using the background color prediction network to get the predicted background point Rendering color Combined with binary intersection mask p Perform weighted addition to obtain the rendering color c of pixel p p .

[0116] For the initial surface reconstruction network, foreground color prediction network, and background color prediction network mentioned in the above embodiments, multiple possible implementations are provided below to illustrate and explain the construction of each network. Figure 7 This is a flow chart of a method for constructing a network in a model training method provided in an embodiment of the present application. Figure 7 As shown, the method may further include:

[0117] S701: Obtain information about the bounding box of the target object.

[0118] For example, information about the bounding box of the target object may be obtained by referring to the description of the bounding box in S101 in the above embodiment, which will not be repeated here.

[0119] S702: Based on the bounding box information, construct an initial surface reconstruction network, a foreground color prediction network, and a background color prediction network.

[0120] For example, based on the information of the bounding box, an initial surface reconstruction network, such as an initial surface distance extraction network and an initial foreground color prediction network, can be constructed in the three-dimensional space within the bounding box, and an initial background color prediction network can be constructed in the three-dimensional space outside the bounding box.

[0121] Specifically, when constructing an initial surface reconstruction network, such as an initial surface distance network, an initial surface reconstruction network initialized as a unit sphere can be constructed within the bounding box based on the bounding box information. For example, the initial surface distance extraction network initialized as a unit sphere can have a network structure such as an MLP encoding network structure.

[0122] The input of the initial surface distance extraction network may include: three-dimensional coordinates x, and the output of the initial surface distance extraction network is: signed distance parameter d, normal vector n and geometric feature vector z.

[0123] For the initial foreground color prediction network and the initial background color prediction network, the preset three-dimensional space can be first divided into a foreground space within the bounding box and a background space outside the bounding box based on the information of the bounding box. Then, based on the foreground space within the bounding box, a color prediction network corresponding to the foreground space is constructed as the initial foreground color prediction network to perform color prediction on the three-dimensional points in the foreground space; and a color prediction network corresponding to the background space is constructed as the initial background color prediction network to perform color prediction on the three-dimensional points in the background space.

[0124] The input of the foreground color prediction network is: the foreground three-dimensional coordinate x in the foreground space f , normal vector n, geometric feature vector z and direction vector v of the observation ray, the output is: the color vector c of the foreground 3D point in the foreground space f ; The input of the background color prediction network is: background 3D point x in the background space b The direction vector v of the observation ray is output as: the color vector c of the background 3D point in the background space b .

[0125] The method of the embodiment of the present application can be constructed based on the information of the bounding box during the network construction process, so that the coordinates of the three-dimensional points of the input of each constructed network can be restricted, thereby realizing network processing of the corresponding three-dimensional points, further improving the processing accuracy of each network, improving the three-dimensional surface reconstruction of surface reconstruction networks such as surface distance extraction networks, and also improving the foreground color prediction network and the background color prediction network for the color prediction of three-dimensional points in their respective spaces, further realizing the reconstruction accuracy of objects in the three-dimensional reconstruction process.

[0126] In the case where the model training method provided in any of the above embodiments obtains a target surface reconstruction network, the embodiment of the present application may also provide an implementation example of three-dimensional reconstruction of the target object based on the target surface reconstruction network. The device body for executing the three-dimensional reconstruction method may be the same hardware body as the device body of the above-mentioned model training method, or may be a different hardware body, and the embodiment of the present application is not limited to this. Before executing the three-dimensional reconstruction method, the pre-trained model parameters are first imported, and the three-dimensional reconstruction method provided by the embodiment of the present application can be executed in combination with the model. Among them, the pre-trained model parameters may at least include: the model parameters of the target surface reconstruction network mentioned above, such as the target surface distance extraction network, the target foreground color prediction network, and the target background color prediction network.

[0127] The specific implementation process of 3D reconstruction is explained and illustrated in detail below with reference to the accompanying drawings. Figure 8 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the method may include:

[0128] S801 : Based on the bounding box of the target object, a pre-trained surface reconstruction network is used to reconstruct the surface of the target object and extract three-dimensional surface data of the target object.

[0129] The surface reconstruction network is a target surface reconstruction network obtained by the model training method described in any of the above embodiments. For example, the surface reconstruction network at least includes: a target surface distance extraction network and an isosurface extraction algorithm obtained by the above training method. In a possible implementation, the sampling points within the bounding box of the target object can be processed according to the target surface distance extraction network to obtain the distance sign parameters and normal vectors of the sampling points; then, based on the distance sign parameters and normal vectors of the sampling points, the isosurface extraction algorithm is used to perform three-dimensional reconstruction of the target object to obtain three-dimensional surface data of the target object. The three-dimensional surface data can also be called an explicit three-dimensional mesh model based on triangular facets.

[0130] S802: Perform surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

[0131] Surface rendering can be performed based on the three-dimensional surface data at a preset viewing angle to obtain the rendering color of each three-dimensional point in the preset three-dimensional space where the target object is located at the preset viewing angle. The three-dimensional rendering data at least includes: the rendering color of each three-dimensional point in the preset three-dimensional space. The specific implementation of the surface rendering in this step can be the same as the above Figures 4 to 6 The surface rendering during the model training process shown in is similar to the rendering color of the target three-dimensional point under the viewing angle. The specific description is given above and will not be repeated here.

[0132] The target surface reconstruction network used in the 3D reconstruction method provided in the embodiment of the present application, because during the network training process, before the surface rendering of the triangle facets, the signed distance parameters and normal vectors of each vertex in the initial triangle facets are projected on each vertex to obtain the projected triangle facets, thereby achieving fine-tuning of the triangle facets in the 3D surface data, and with the help of surface rendering, achieving nonlinear sampling of the object surface, improving the sampling accuracy and precision in the surface rendering process, and improving the training accuracy of the surface reconstruction network as much as possible, thereby improving the surface integrity of the 3D reconstructed object, avoiding geometric feature errors on the object surface, enriching the surface details of the 3D reconstructed object, and ensuring the accuracy of the 3D reconstruction of the curved surface of complex objects.

[0133] In the process of 3D reconstruction based on the model obtained by the above training, when the 3D surface data of the target object is obtained, the surface rendering can be performed directly based on each target triangle in the 3D surface data, or the target triangle can be projected and then the surface rendering can be performed based on the projected target triangle, thereby further ensuring the nonlinear sampling in the surface rendering process, improving the sampling accuracy and precision in the surface rendering process, and improving the reconstruction accuracy of the object surface in the 3D reconstruction process. The following method flow chart is used to further explain: Figure 9 This is a flowchart of a method for projecting target triangular facets in a 3D reconstruction method provided as an example in this application. The 3D surface data shown above includes: multiple target triangular facets; accordingly, before performing surface rendering based on the 3D surface data in S802 to obtain 3D rendering data of the target object, the 3D reconstruction method may also include:

[0134] S901: Obtain the signed distance parameters and normal vectors of each vertex in multiple target triangles.

[0135] For the specific implementation of this step, please refer to the implementation of S102 in the above model training method, which will not be repeated here.

[0136] S902 : Project each vertex of each target triangle according to the signed distance parameter and the normal vector of each vertex of each target triangle to obtain a projected target triangle.

[0137] For the specific implementation of this step, please refer to the implementation of S103 in the above model training method, which will not be repeated here.

[0138] As shown in S802 above, performing surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object may include:

[0139] S903 , performing surface rendering according to the projected target triangular facets to obtain the rendering color of each 3D point in the preset 3D space where the target object is located, wherein the 3D rendering data includes the rendering color of each 3D point.

[0140] The specific implementation of this step can be found in the above model training method. Figure 4 The implementation of obtaining the rendering color of the target three-dimensional point in the method embodiment shown is not repeated here.

[0141] A 3D reconstruction method provided in an embodiment of the present application can project target triangular facets in the reconstructed 3D surface data before surface rendering in the 3D reconstruction method, thereby achieving fine-tuning of the target triangular facets in the 3D surface data, and with the help of surface rendering, achieving nonlinear sampling of the object surface in the 3D reconstruction, thereby improving the surface integrity of the 3D reconstructed object, avoiding errors in the geometric features of the object surface, enriching the surface details of the 3D reconstructed object, and ensuring the accuracy of the 3D reconstruction of the curved surface of complex objects.

[0142] In one possible implementation, the model training method not only obtains the target surface reconstruction network, but also obtains the target foreground prediction network and the target background prediction network. Then, during the 3D reconstruction process, for the surface rendering of the 3D surface data, color prediction can be performed separately for the 3D foreground points and 3D background points, ensuring the accuracy of the color prediction of the 3D points. The following method flow chart is used to further explain: Figure 10 This is a flow chart of a surface rendering method in a three-dimensional reconstruction method provided as an example in this application. Figure 10 As shown, performing surface rendering based on the three-dimensional surface data in S802 as shown above to obtain three-dimensional rendering data of the target object may include:

[0143] S1001. Based on the three-dimensional surface data, the coordinates of the three-dimensional foreground point and the coordinates of the three-dimensional background point of the target object in the preset three-dimensional space are obtained.

[0144] For the specific implementation of obtaining the coordinates of the three-dimensional foreground point and the three-dimensional background point based on the three-dimensional surface data, please refer to the specific implementation of obtaining the coordinates of the first target three-dimensional point and the second target three-dimensional point in the above embodiment, which will not be repeated here.

[0145] S1002: Based on the coordinates of the three-dimensional foreground point, a foreground color prediction network is used to perform color prediction to obtain a rendering color of the three-dimensional foreground point.

[0146] The specific implementation of this step is similar to the implementation of obtaining the rendering color of the first target three-dimensional point in the above-mentioned model training method. The specific description is given above and will not be repeated here.

[0147] S1003 , using a background color prediction network to perform color prediction based on the coordinates of the three-dimensional background point to obtain a rendering color of the three-dimensional background point.

[0148] The three-dimensional rendering data includes the rendering colors of the three-dimensional foreground points and the rendering colors of the three-dimensional background points.

[0149] The specific implementation of this step is similar to the implementation of obtaining the rendering color of the second target three-dimensional point in the above-mentioned model training method. The specific description is given above and will not be repeated here.

[0150] The three-dimensional reconstruction method provided in this embodiment can improve the three-dimensional reconstruction accuracy of background objects by using a foreground color prediction network and a background color prediction network to predict the rendering color of three-dimensional foreground points and the rendering color of three-dimensional background points respectively during the surface rendering process of the three-dimensional reconstruction method.

[0151] The following describes the model training device, three-dimensional reconstruction device, equipment and storage medium provided by this application for execution. The specific implementation process and technical effects are described above and will not be repeated below.

[0152] Figure 11 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 11 As shown, the model training device 1100 may include:

[0153] The first surface reconstruction module 1101 is used to reconstruct the surface of the target object based on the bounding box of the target object and adopt a preset initial surface distance extraction network to obtain initial three-dimensional surface data of the target object. The initial three-dimensional surface data includes: multiple initial triangular facets.

[0154] The acquisition module 1102 is configured to acquire the signed distance parameters and normal vectors of each vertex in a plurality of initial triangular facets.

[0155] The projection module 1103 is configured to project each vertex of the initial triangle according to the signed distance parameter and the normal vector of each vertex of each initial triangle to obtain a projected triangle.

[0156] The first surface rendering module 1104 is configured to perform surface rendering based on the projected triangles at each viewing angle to obtain a rendering color for each pixel at that viewing angle.

[0157] The training module 1105 is used to train the initial surface reconstruction network according to the rendering color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle to obtain a target surface reconstruction network.

[0158] The model training device provided in this embodiment can, during the model training process, project each vertex by using the signed distance parameters and normal vectors of each vertex in the initial triangle before surface rendering of the triangle, thereby obtaining a projected triangle. This achieves fine-tuning of the triangles in the three-dimensional surface data, and with the help of surface rendering, achieves nonlinear sampling of the object surface, improves the sampling accuracy and precision in the surface rendering process, and maximizes the training accuracy of the surface reconstruction network, thereby improving the surface integrity of the three-dimensional reconstructed object, avoiding geometric feature errors on the object surface, enriching the surface details of the three-dimensional reconstructed object, and ensuring the accuracy of the three-dimensional reconstruction of the curved surface of complex objects.

[0159] In a feasible implementation scheme, the projection module 1103 is specifically used to: translate each vertex according to the signed distance parameter of each vertex along the opposite direction of the normal vector of each vertex, thereby obtaining the translated vertices; the projected triangular face is: a triangular face formed by the translated vertices.

[0160] In a feasible implementation scheme, the first surface rendering module 1104 is specifically used to: perform surface rendering on the projected triangular facets under the viewing angle to obtain the intersection information of each pixel point under the viewing angle and the projected triangular facets, and the intersection information is used to indicate whether there is a three-dimensional intersection between the observation ray corresponding to each pixel point under the viewing angle and the projected triangular facets; determine the coordinates of the target three-dimensional point on the observation ray corresponding to each pixel point based on the intersection information; perform color prediction using the initial color prediction network based on the coordinates of the target three-dimensional point to obtain the predicted color of the target three-dimensional point; determine the rendering color of each pixel point under the viewing angle based on the predicted color of the target three-dimensional point.

[0161] In a feasible implementation scheme, the first surface rendering module 1104 is further specifically used to: if the intersection information indicates that there is a three-dimensional intersection between the first observation ray and the projected triangular facet corresponding to the first pixel point under the viewing angle, then the coordinates of the three-dimensional intersection are determined as the coordinates of the first target three-dimensional point on the first observation ray; if the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangular facet corresponding to the second pixel point under the viewing angle, then based on the second observation ray and the coordinates of the observation camera of the viewing angle, the coordinates of the three-dimensional background point are predicted as the coordinates of the second target three-dimensional point on the second observation ray; if the intersection information indicates that there is a three-dimensional intersection between the first observation ray and the projected triangular facet, the initial foreground color prediction network is used to perform color prediction based on the coordinates of the first target three-dimensional point to obtain the predicted color of the first target three-dimensional point; if the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangular facet, the initial background color prediction network is used to perform color prediction based on the coordinates of the second target three-dimensional point to obtain the predicted color of the second target three-dimensional point.

[0162] In a feasible implementation scheme, the training module 1105 is also used to: train the initial foreground color prediction network and the initial background color prediction network according to the rendered color of each pixel under the viewing angle and the true color of each pixel in the color image under the viewing angle to obtain the target foreground color prediction network and the target background color prediction network.

[0163] In a feasible implementation scheme, the first surface rendering module 1104 is further specifically used to: calculate the target translation distance of the observation camera position based on the second observation ray and the observation camera position of the viewing angle; translate the observation camera position by the target translation distance along the second observation ray to obtain the coordinates of the predicted three-dimensional background point as the coordinates of the second target three-dimensional point.

[0164] In a feasible implementation scheme, the first surface rendering module 1104 is also used to: if the intersection information indicates that there is a three-dimensional intersection between the first observation ray and the projected triangle, determine the intersection mask corresponding to the first pixel point as the first mask parameter; if the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangle, determine the intersection mask corresponding to the second pixel point as the second mask parameter; weight the predicted color of the first target three-dimensional point according to the first mask parameter to obtain the rendering color of the first pixel point under the viewing angle, and weight the predicted color of the second target three-dimensional point according to the second mask parameter to obtain the rendering color of the second pixel point under the viewing angle.

[0165] In an implementation example, the first surface rendering module 1104 is specifically used to: use a preset differentiable rasterization rendering algorithm under the viewing angle to perform surface rendering on the projected triangular facets, and obtain intersection information between each pixel point and the projected triangular facets under the viewing angle.

[0166] In a feasible embodiment, the model training device 1100 further includes:

[0167] The construction module is used to obtain the bounding box information of the target object; based on the bounding box information, an initial surface reconstruction network, an initial foreground color prediction network, and an initial background color prediction network are constructed.

[0168] In a feasible implementation scheme, the construction module is specifically used to: construct an initial surface reconstruction network initialized to a unit sphere within the bounding box based on the information of the bounding box; divide the preset three-dimensional space into a foreground space within the bounding box and a background space outside the bounding box based on the information of the bounding box; construct a color prediction network corresponding to the foreground space as an initial foreground color prediction network to perform color prediction on the three-dimensional points in the foreground space; construct a color prediction network corresponding to the background space as an initial background color prediction network to perform color prediction on the three-dimensional points in the background space.

[0169] In a feasible implementation scheme, the acquisition module 1102 is further configured to acquire the normal vector, geometric feature vector, and direction vector of the first target 3D point; and acquire the direction vector of the second observation ray.

[0170] In a feasible implementation scheme, the first surface rendering module 1104 is specifically used to: perform color prediction using an initial foreground color prediction network based on the coordinates, normal vector, geometric feature vector of the first target three-dimensional point and the direction vector of the second observation ray to obtain the predicted color of the first target three-dimensional point; perform color prediction using an initial background color prediction network based on the coordinates of the second target three-dimensional point and the direction vector of the second observation ray to obtain the predicted color of the second target three-dimensional point.

[0171] Figure 12 A schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application is shown in FIG. Figure 12 As shown, the 3D reconstruction device 1200 may include:

[0172] The first surface reconstruction module 1201 is used to reconstruct the surface of the target object based on the bounding box of the target object using a pre-trained surface reconstruction network to extract the three-dimensional surface data of the target object; wherein the surface reconstruction network is a target surface reconstruction network obtained by using the model training method described in any of the above embodiments.

[0173] The second surface rendering module 1202 is configured to perform surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

[0174] The target surface reconstruction network used in the 3D reconstruction method provided in this embodiment is that before the surface rendering of the triangle facets, the signed distance parameters and normal vectors of each vertex in the initial triangle facets are projected on each vertex to obtain the projected triangle facets, thereby achieving fine-tuning of the triangle facets in the 3D surface data, and with the help of surface rendering, achieving nonlinear sampling of the object surface, improving the sampling accuracy and precision in the surface rendering process, and improving the training accuracy of the surface reconstruction network as much as possible, thereby improving the surface integrity of the 3D reconstructed object, avoiding geometric feature errors on the object surface, enriching the surface details of the 3D reconstructed object, and ensuring the accuracy of the 3D reconstruction of the curved surface of complex objects.

[0175] In a feasible embodiment, the three-dimensional surface data includes: a plurality of target triangular facets; the three-dimensional reconstruction device 1200 further includes:

[0176] An acquisition module is used to obtain the signed distance parameters and normal vectors of each vertex in multiple target triangles;

[0177] The projection module is used to project each vertex of the target triangle according to the signed distance parameter and the normal vector of each vertex of each target triangle to obtain a projected target triangle.

[0178] The second surface rendering module 1202 is specifically configured to perform surface rendering based on the projected target triangles to obtain the rendering color of each 3D point in the preset 3D space where the target object is located; the 3D rendering data includes the rendering color of each 3D point.

[0179] In one feasible embodiment, the second surface rendering module 1202 is specifically configured to obtain the coordinates of a 3D foreground point and a 3D background point of a target object in a preset 3D space based on the 3D surface data; perform color prediction using a foreground color prediction network based on the coordinates of the 3D foreground point to obtain the rendering color of the 3D foreground point; and perform color prediction using a background color prediction network based on the coordinates of the 3D background point to obtain the rendering color of the 3D background point. The 3D rendering data includes the rendering colors of the 3D foreground point and the rendering colors of the 3D background point.

[0180] Figure 13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 13 As shown, the computer device includes: a processor 1301, a storage medium 1302 and a bus 1303. The storage medium 1302 stores machine-readable instructions executable by the processor 1301. When the computer device runs a model training method as in the embodiment, the processor 1301 communicates with the storage medium 1302 via the bus 1303, and the processor 1201 executes the machine-readable instructions. The processor 1301 performs the preamble of the method item to perform the following steps:

[0181] Based on the bounding box of the target object, a preset initial surface distance extraction network is used to reconstruct the surface of the target object to obtain initial three-dimensional surface data of the target object, where the initial three-dimensional surface data includes: multiple initial triangular facets;

[0182] Get the signed distance parameters and normal vectors of each vertex in multiple initial triangles;

[0183] Projecting each vertex of the initial triangle according to the signed distance parameter and normal vector of each vertex of the initial triangle to obtain a projected triangle;

[0184] At each viewing angle, the surface is rendered based on the projected triangles to obtain the rendering color of each pixel at that viewing angle.

[0185] According to the rendering color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle, the initial surface reconstruction network is trained to obtain the target surface reconstruction network.

[0186] In a feasible implementation, the processor 1301, when performing projection on each vertex of each initial triangle according to the signed distance parameter and the normal vector of each vertex of each initial triangle to obtain a projected triangle, is specifically configured to:

[0187] According to the signed distance parameter of each vertex, each vertex is translated along the opposite direction of the normal vector of each vertex by the corresponding signed distance parameter to obtain the translated vertices; the projected triangle patch is: the triangle formed by the translated vertices.

[0188] In a feasible embodiment, when the processor 1301 performs surface rendering based on the projected triangles at each viewing angle to obtain the rendering color of each pixel at the viewing angle, it is specifically configured to:

[0189] Under the viewing angle, the surface of the projected triangle facet is rendered to obtain the intersection information of each pixel point under the viewing angle and the projected triangle facet. The intersection information is used to indicate whether there is a three-dimensional intersection between the observation ray corresponding to each pixel point under the viewing angle and the projected triangle facet;

[0190] According to the intersection information, the coordinates of the target three-dimensional point on the observation ray corresponding to each pixel point are determined;

[0191] According to the coordinates of the target 3D point, the initial color prediction network is used to predict the color and obtain the predicted color of the target 3D point;

[0192] According to the predicted color of the target 3D point, the rendering color of each pixel under the viewing angle is determined.

[0193] In a feasible implementation manner, when determining the coordinates of the target three-dimensional point on the observation ray corresponding to each pixel point based on the intersection information, the processor 1301 specifically performs the following steps:

[0194] If the intersection information indicates that there is a three-dimensional intersection between the first observation ray and the projected triangle facet corresponding to the first pixel point under the viewing angle, the coordinates of the three-dimensional intersection are determined to be the coordinates of the first target three-dimensional point on the first observation ray;

[0195] If the intersection information indicates that there is no three-dimensional intersection between the second observation ray corresponding to the second pixel point under the viewing angle and the projected triangle facet, then based on the second observation ray and the coordinates of the observation camera under the viewing angle, the coordinates of the three-dimensional background point are predicted as the coordinates of the second target three-dimensional point on the second observation ray;

[0196] In a feasible implementation, when the processor 1301 performs color prediction based on the coordinates of the target 3D point using the initial color prediction network to obtain the predicted color of the target 3D point, the processor 1301 specifically performs:

[0197] If the intersection information indicates that there is a 3D intersection between the first observation ray and the projected triangle, the color prediction is performed using the initial foreground color prediction network according to the coordinates of the first target 3D point to obtain the predicted color of the first target 3D point;

[0198] If the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangle, the initial background color prediction network is used to perform color prediction based on the coordinates of the second target three-dimensional point to obtain the predicted color of the second target three-dimensional point.

[0199] In a feasible implementation manner, the processor 1301 is further configured to execute:

[0200] According to the rendered color of each pixel under the viewing angle and the true color of each pixel in the color image under the viewing angle, the initial foreground color prediction network and the initial background color prediction network are trained to obtain the target foreground color prediction network and the target background color prediction network.

[0201] In one feasible implementation, the processor 1301, when executing the prediction of the coordinates of the three-dimensional background point as the coordinates of the second target three-dimensional point on the observation ray corresponding to the second pixel point based on the second observation ray and the position of the observation camera of the viewing angle, is specifically configured to:

[0202] Calculate the target translation distance of the observation camera position according to the second observation ray and the observation camera position of the viewing angle;

[0203] The observation camera position is translated by the target translation distance along the second observation ray to obtain the coordinates of the predicted three-dimensional background point as the coordinates of the second target three-dimensional point.

[0204] In a feasible implementation manner, the processor 1301 is further configured to execute:

[0205] If the intersection information indicates that there is a three-dimensional intersection between the first observation ray and the projected triangle, determine the intersection mask corresponding to the first pixel point as the first mask parameter;

[0206] If the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangle facet, determining the intersection mask corresponding to the second pixel point as the second mask parameter;

[0207] In a feasible implementation, the processor 1301, when determining the rendering color of each pixel under a viewing angle based on the predicted color of the target 3D point, is specifically configured to:

[0208] The predicted color of the first target 3D point is weighted according to the first mask parameter to obtain the rendered color of the first pixel under the viewing angle; the predicted color of the second target 3D point is weighted according to the second mask parameter to obtain the rendered color of the second pixel under the viewing angle.

[0209] In a feasible implementation, when the processor 1301 performs surface rendering on the projected triangular facet under the viewing angle and obtains the intersection information between each pixel point and the projected triangular facet under the viewing angle, it is specifically configured to:

[0210] Under the viewing angle, the preset differentiable rasterization rendering algorithm is used to perform surface rendering on the projected triangular facets to obtain the intersection information between each pixel point and the projected triangular facets under the viewing angle.

[0211] In a feasible embodiment, before performing surface reconstruction of the target object based on the bounding box of the target object using a pre-trained surface reconstruction network to extract the three-dimensional surface data of the target object, the processor 1301 is further configured to perform:

[0212] Get the bounding box information of the target object;

[0213] Based on the bounding box information, an initial surface reconstruction network, an initial foreground color prediction network, and an initial background color prediction network are constructed.

[0214] In a feasible implementation manner, the processor 1301 is specifically configured to:

[0215] Based on the information of the bounding box, an initial surface reconstruction network initialized as a unit sphere is constructed within the bounding box;

[0216] Based on the information of the bounding box, the preset three-dimensional space is divided into a foreground space within the bounding box and a background space outside the bounding box;

[0217] Constructing a color prediction network corresponding to the foreground space as an initial foreground color prediction network to perform color prediction on three-dimensional points in the foreground space;

[0218] A color prediction network corresponding to the background space is constructed as the initial background color prediction network to perform color prediction on three-dimensional points in the background space.

[0219] In one feasible implementation, the processor 1301 is further configured to execute:

[0220] Obtain the normal vector, geometric characteristic vector and direction vector of the first target 3D point, and obtain the direction vector of the second observation ray;

[0221] In a feasible implementation, when the processor 1301 performs color prediction based on the coordinates of the first target 3D point using the initial foreground color prediction network to obtain the predicted color of the first target 3D point, the processor 1301 is specifically configured to:

[0222] According to the coordinates, normal vector, geometric feature vector of the first target 3D point and the direction vector of the second observation ray, the initial foreground color prediction network is used to perform color prediction to obtain the predicted color of the first target 3D point;

[0223] In a feasible implementation, when the processor 1301 performs color prediction based on the coordinates of the second target 3D point using the initial background color prediction network to obtain the predicted color of the second target 3D point, the processor 1301 is specifically configured to:

[0224] According to the coordinates of the second target 3D point and the direction vector of the second observation ray, the initial background color prediction network is used to perform color prediction to obtain the predicted color of the second target 3D point.

[0225] When a computer device executes a three-dimensional reconstruction method as described in the embodiment, the processor 1301 communicates with the storage medium 1302 via the bus 1303, and the processor 1301 executes the machine-readable instructions, including the preamble of the method item, to perform the following steps:

[0226] Based on the bounding box of the target object, a pre-trained surface reconstruction network is used to reconstruct the surface of the target object and extract the three-dimensional surface data of the target object; wherein the surface reconstruction network is a target surface reconstruction network obtained by the model training method described in any of the above embodiments;

[0227] Surface rendering is performed based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

[0228] In a feasible embodiment, the three-dimensional surface data includes: a plurality of target triangular facets; before performing surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object, the processor 1301 is further configured to perform:

[0229] Get the signed distance parameters and normal vectors of each vertex in multiple target triangles;

[0230] According to the signed distance parameters and normal vectors of each vertex of each target triangle, each vertex of the target triangle is projected to obtain a projected target triangle.

[0231] In a feasible implementation, when the processor 1301 performs surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object, it is specifically configured to execute:

[0232] Surface rendering is performed based on the projected target triangular facets to obtain the rendering color of each 3D point in the preset 3D space where the target object is located; the 3D rendering data includes: the rendering color of each 3D point.

[0233] In a feasible implementation, when the processor 1301 performs surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object, it is specifically configured to execute:

[0234] Based on the three-dimensional surface data, coordinates of three-dimensional foreground points and three-dimensional background points of the target object in a preset three-dimensional space are obtained;

[0235] According to the coordinates of the 3D foreground point, the foreground color prediction network is used to predict the color and obtain the rendering color of the 3D foreground point;

[0236] According to the coordinates of the three-dimensional background points, a background color prediction network is used to perform color prediction to obtain the rendering color of the three-dimensional background points; the three-dimensional rendering data includes: the rendering color of the three-dimensional foreground point and the rendering color of the three-dimensional background point.

[0237] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor when it is running, and the processor executes the steps of any of the above-mentioned model training methods or three-dimensional reconstruction methods.

[0238] An embodiment of the present application also provides a computer program product, which includes program instructions. When the program instructions are executed by a processor, the steps of any of the above-mentioned model training methods or three-dimensional reconstruction methods are executed.

[0239] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0240] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0241] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0242] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0243] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0244] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0245] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: include: Based on the bounding box of the target object, a preset initial surface reconstruction network is used to reconstruct the surface of the target object to obtain initial three-dimensional surface data of the target object, wherein the initial three-dimensional surface data includes: a plurality of initial triangular facets; Obtaining signed distance parameters and normal vectors of each vertex in the plurality of initial triangles; Projecting each vertex of each initial triangle according to the signed distance parameter and the normal vector of each vertex of the initial triangle to obtain a projected triangle; Performing surface rendering based on the projected triangles at each viewing angle to obtain the rendering color of each pixel at the viewing angle; The initial surface reconstruction network is trained according to the rendered color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle to obtain a target surface reconstruction network.

2. The method according to claim 1, characterized in that The step of projecting each vertex of each initial triangular facet according to the signed distance parameter and the normal vector of each vertex of the initial triangular facet to obtain a projected triangular facet comprises: According to the signed distance parameters of the vertices, the vertices are translated along the opposite direction of the normal vectors of the vertices by the corresponding signed distance parameters to obtain the translated vertices; the projected triangular facets are: the triangular face formed by the translated vertices.

3. The method according to claim 1, characterized in that The surface rendering is performed according to the projected triangular facets at each viewing angle to obtain the rendering color of each pixel at the viewing angle, including: Performing surface rendering on the projected triangular facet at the viewing angle to obtain intersection information between each pixel point and the projected triangular facet at the viewing angle, the intersection information being used to indicate whether a three-dimensional intersection exists between an observation ray corresponding to each pixel point at the viewing angle and the projected triangular facet; Determining the coordinates of the target three-dimensional point on the observation ray corresponding to each pixel point according to the intersection information; Performing color prediction using an initial color prediction network according to the coordinates of the target three-dimensional point to obtain a predicted color of the target three-dimensional point; The rendering color of each pixel point at the viewing angle is determined according to the predicted color of the target three-dimensional point.

4. The method according to claim 3, characterized in that Determining the coordinates of the target three-dimensional point on the observation ray corresponding to each pixel point according to the intersection information includes: If the intersection information indicates that a three-dimensional intersection exists between the first observation ray corresponding to the first pixel point at the viewing angle and the projected triangular facet, determining the coordinates of the three-dimensional intersection as the coordinates of the first target three-dimensional point on the first observation ray; If the intersection information indicates that there is no three-dimensional intersection between the second observation ray corresponding to the second pixel point at the viewing angle and the projected triangle facet, predicting the coordinates of the three-dimensional background point as the coordinates of the second target three-dimensional point on the second observation ray based on the second observation ray and the coordinates of the observation camera at the viewing angle; The method of performing color prediction using an initial color prediction network according to the coordinates of the target three-dimensional point to obtain a predicted color of the target three-dimensional point includes: If the intersection information indicates that the first observation ray and the projected triangle facet have a three-dimensional intersection, performing color prediction using an initial foreground color prediction network based on the coordinates of the first target three-dimensional point to obtain a predicted color of the first target three-dimensional point; If the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangle facet, color prediction is performed using the initial background color prediction network according to the coordinates of the second target three-dimensional point to obtain the predicted color of the second target three-dimensional point.

5. The method according to claim 4, characterized in that The method further comprises: The initial foreground color prediction network and the initial background color prediction network are trained according to the rendered color of each pixel under the viewing angle and the true color of each pixel in the color image under the viewing angle to obtain a target foreground color prediction network and a target background color prediction network.

6. The method according to claim 4, characterized in that The step of predicting the coordinates of the three-dimensional background point as the coordinates of the second target three-dimensional point on the observation ray corresponding to the second pixel point based on the second observation ray and the position of the observation camera of the viewing angle includes: Calculating a target translation distance of the observation camera position according to the second observation ray and the observation camera position of the viewing angle; The observation camera position is translated along the second observation ray by the target translation distance to obtain the coordinates of the predicted three-dimensional background point as the coordinates of the second target three-dimensional point.

7. The method according to claim 4, characterized in that The method further comprises: If the intersection information indicates that there is a three-dimensional intersection between the first observation ray and the projected triangle, determine the intersection mask corresponding to the first pixel point as a first mask parameter; If the intersection information indicates that there is no three-dimensional intersection between the second observation ray and the projected triangle facet, determining the intersection mask corresponding to the second pixel point as a second mask parameter; The determining, based on the predicted color of the target three-dimensional point, the rendering color of each pixel at the viewing angle includes: weighting the predicted color of the first target three-dimensional point according to the first mask parameter to obtain a rendering color of the first pixel point at the viewing angle; The predicted color of the second target three-dimensional point is weighted according to the second mask parameter to obtain a rendering color of the second pixel point under the viewing angle.

8. The method according to claim 3, characterized in that The step of performing surface rendering on the projected triangular facet at the viewing angle to obtain intersection information between each pixel point and the projected triangular facet at the viewing angle includes: Under the viewing angle, a preset differentiable rasterization rendering algorithm is used to perform surface rendering on the projected triangular facets to obtain intersection information between each pixel point and the projected triangular facets under the viewing angle.

9. The method according to claim 4, characterized in that Before performing surface reconstruction on the target object based on the bounding box of the target object using a pre-trained surface reconstruction network and extracting three-dimensional surface data of the target object, the method further includes: Obtaining information about a bounding box of the target object; Based on the information of the bounding box, the initial surface reconstruction network, the initial foreground color prediction network and the initial background color prediction network are constructed.

10. The method according to claim 9, characterized in that The constructing of the initial surface reconstruction network, the initial foreground color prediction network, and the initial background color prediction network based on the information of the bounding box includes: Based on the information of the bounding box, constructing the initial surface reconstruction network initialized as a unit sphere within the bounding box; Based on the information of the bounding box, the preset three-dimensional space is divided into a foreground space within the bounding box and a background space outside the bounding box; Constructing a color prediction network corresponding to the foreground space as the initial foreground color prediction network to perform color prediction on three-dimensional points in the foreground space; A color prediction network corresponding to the background space is constructed as the initial background color prediction network to perform color prediction on three-dimensional points in the background space.

11. The method according to claim 4, characterized in that The method further comprises: Obtaining a normal vector, a geometric characteristic vector, and a direction vector of the first observation ray of the first target three-dimensional point; Obtaining a direction vector of the second observation ray; The method of performing color prediction using an initial foreground color prediction network according to the coordinates of the first target 3D point to obtain a predicted color of the first target 3D point includes: performing color prediction using the initial foreground color prediction network based on the coordinates, normal vector, geometric feature vector of the first target 3D point and the direction vector of the second observation ray to obtain a predicted color of the first target 3D point; The method of performing color prediction using an initial background color prediction network according to the coordinates of the second target three-dimensional point to obtain a predicted color of the second target three-dimensional point includes: According to the coordinates of the second target three-dimensional point and the direction vector of the second observation ray, the initial background color prediction network is used to perform color prediction to obtain a predicted color of the second target three-dimensional point.

12. A three-dimensional reconstruction method, characterized in that: include: Based on the bounding box of the target object, a pre-trained surface reconstruction network is used to reconstruct the surface of the target object to extract three-dimensional surface data of the target object; wherein the surface reconstruction network is a target surface reconstruction network obtained by the model training method according to any one of claims 1 to 11; Surface rendering is performed based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

13. The method according to claim 12, characterized in that The three-dimensional surface data includes: a plurality of target triangular facets; before performing surface rendering based on the three-dimensional surface data to obtain the three-dimensional rendering data of the target object, the method further includes: Obtaining the signed distance parameters and normal vectors of each vertex in the plurality of target triangles; Projecting each vertex of each target triangle according to the signed distance parameter and the normal vector of each vertex of the target triangle to obtain a projected target triangle; The performing surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object includes: Surface rendering is performed according to the projected target triangular facets to obtain the rendering color of each three-dimensional point in the preset three-dimensional space where the target object is located; the three-dimensional rendering data includes: the rendering color of each three-dimensional point.

14. The method according to claim 12, characterized in that The performing surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object includes: Based on the three-dimensional surface data, coordinates of three-dimensional foreground points and three-dimensional background points of the target object in a preset three-dimensional space are obtained; According to the coordinates of the three-dimensional foreground point, a foreground color prediction network is used to perform color prediction to obtain a rendering color of the three-dimensional foreground point; According to the coordinates of the three-dimensional background point, a background color prediction network is used to perform color prediction to obtain the rendering color of the three-dimensional background point; the three-dimensional rendering data includes: the rendering color of the three-dimensional foreground point and the rendering color of the three-dimensional background point.

15. A model training device, characterized in that: include: A first surface reconstruction module is configured to reconstruct the surface of the target object based on a bounding box of the target object and adopt a preset initial surface distance extraction network to obtain initial three-dimensional surface data of the target object, wherein the initial three-dimensional surface data includes: a plurality of initial triangular facets; An acquisition module, configured to acquire a signed distance parameter and a normal vector of each vertex in the plurality of initial triangular facets; A projection module, configured to project each vertex of each initial triangular facet according to the signed distance parameter and the normal vector of each vertex of the initial triangular facet to obtain a projected triangular facet; A first surface rendering module is used to perform surface rendering according to the projected triangle patch at each viewing angle to obtain a rendering color of each pixel at the viewing angle; A training module is used to train the initial surface reconstruction network according to the rendering color of each pixel under the viewing angle and the true value color of each pixel in the color image under the viewing angle to obtain a target surface reconstruction network.

16. A three-dimensional reconstruction device, characterized in that: include: a second surface reconstruction module, configured to reconstruct the surface of the target object based on the bounding box of the target object using a pre-trained surface reconstruction network to extract three-dimensional surface data of the target object; wherein the surface reconstruction network is a target surface reconstruction network obtained using the model training method according to any one of claims 1 to 11; The second surface rendering module is configured to perform surface rendering based on the three-dimensional surface data to obtain three-dimensional rendering data of the target object.

17. A computer device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the model training method according to any one of claims 1 to 11, or the steps of the three-dimensional reconstruction method according to any one of claims 12 to 14.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the model training method according to any one of claims 1 to 11 or the three-dimensional reconstruction method according to any one of claims 12 to 14.

Images