Model training method and device, three-dimensional reconstruction method and device, equipment and storage medium

Abstract

The invention provides a model training method and device, a three-dimensional reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the steps of inputting coordinates of sampling points in a bounding box of a target object into a pre-training surface distance extraction network to obtain first symbol distance parameters, normal vectors and geometric feature vectors of the sampling points; according to the coordinates, the normal vectors and the geometric feature vectors of the sampling points, determining hollowed-out mask parameters of the observation rays corresponding to the sampling points, and according to the first symbol distance parameters of the sampling points, determining the depth value of the vertex of the upper surface of the observation rays; calculating a model loss value according to the hollowed-out mask parameter of the observation ray and the depth value of the vertex of the upper surface of the observation ray; and according to the model loss value, performing parameter adjustment on the pre-trained surface distance extraction network to obtain a target surface distance extraction network. According to the embodiment of the invention, the three-dimensional reconstruction precision of the hollow object is improved, and the workload of subsequent manual model repair is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a model training method, a three-dimensional reconstruction method, and apparatus, equipment, and storage medium thereof. Background Art

[0002] Multi-view 3D reconstruction technology reconstructs 3D models from captured 2D images from multiple perspectives. This technology plays a crucial role in 3D gaming, virtual reality, augmented reality, and other fields. It is a crucial tool for producing 3D digital assets in virtual scene applications, such as games. It enhances the realism of 3D content, creating a sense of immersion for users while reducing the workload and costs of modeling.

[0003] In recent years, implicit surface reconstruction algorithms based on neural rendering have developed rapidly and play a vital role in the field of 3D content generation. Currently, this type of technology implicitly stores the 3D information within an object's bounding box in a signed distance function (SDF) network encoded by a neural network. Through training with differentiable neural rendering, the 3D surface data of the object is extracted and used as an explicit 3D mesh model.

[0004] While implicit surface reconstruction algorithms using neural rendering can reconstruct complete and continuous 3D mesh models, they are unable to reconstruct hollowed-out parts of objects in practical applications. Specifically, for objects with hollowed-out surfaces, these parts often appear black in captured photos. Implicit surface reconstruction algorithms tend to interpret these hollowed-out parts as smooth black surfaces, failing to recover the shape of the holes. This limits the accuracy of 3D reconstruction of hollowed-out objects and increases the workload of subsequent manual restoration. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a model training method, a three-dimensional reconstruction method, and its apparatus, equipment and storage medium to improve the three-dimensional reconstruction accuracy of hollow objects and reduce the workload of subsequent manual repair of the model.

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0007] Get multiple sampling points within the bounding box of the target object;

[0008] Inputting the coordinates of the plurality of sampling points into a pre-trained surface distance extraction network to obtain first signed distance parameters of the plurality of sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points;

[0009] determining, according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point, a hollowing mask parameter of the observation ray corresponding to the sampling point, wherein the hollowing mask parameter of the observation ray is used to indicate whether the sampling point is in a hollowed portion on the observation ray;

[0010] Determining, according to the first signed distance parameter of the sampling point, a hollowing mask parameter of the observation ray corresponding to the sampling point and a depth value of a vertex on the surface of the observation ray;

[0011] Calculating a model loss value according to a hollowing mask parameter of the observation ray and a depth value of a vertex on the surface of the observation ray;

[0012] According to the model loss value, the parameters of the pre-trained surface distance extraction network are adjusted to obtain a target surface distance extraction network.

[0013] In a second aspect, an embodiment of the present application further provides a three-dimensional reconstruction method, the method comprising:

[0014] Get the sampling points within the bounding box of the target object;

[0015] Inputting the coordinates of a plurality of sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the plurality of sampling points; wherein the surface distance extraction network is a target surface distance extraction network obtained by the model training method described in the first aspect;

[0016] According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction on the target object to obtain a three-dimensional grid of the target surface of the target object.

[0017] In a third aspect, an embodiment of the present application further provides a model training device, comprising:

[0018] A first acquisition module is used to acquire multiple sampling points within a bounding box of the target object;

[0019] a first extraction module, configured to input the coordinates of the plurality of sampling points into a pre-trained surface distance extraction network to obtain first signed distance parameters of the plurality of sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points;

[0020] a determination module, configured to determine, based on the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point, a hollowing mask parameter of the observation ray corresponding to the sampling point; and determine, based on the first signed distance parameter of the sampling point, a depth value of a vertex on the surface of the observation ray, wherein the hollowing mask parameter of the observation ray is used to indicate whether the sampling point on the observation ray of the bounding box is in a hollowed portion;

[0021] A calculation module, configured to calculate a model loss value based on a hollowing mask parameter of the observation ray and a depth value of a vertex on a surface of the observation ray;

[0022] A training module is used to adjust the parameters of the pre-trained surface distance extraction network according to the model loss value to obtain a target surface distance extraction network.

[0023] In a fourth aspect, an embodiment of the present application further provides a three-dimensional reconstruction device, comprising:

[0024] A second acquisition module is used to obtain sampling points within the bounding box of the target object;

[0025] A second extraction module is configured to input the coordinates of multiple sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the multiple sampling points; wherein the surface distance extraction network is a target surface distance extraction network obtained using the model training method described in the first aspect;

[0026] The reconstruction module is used to perform three-dimensional reconstruction on the target object using a preset surface extraction algorithm according to the target symbol distance parameter to obtain a three-dimensional grid of the target surface of the target object.

[0027] In the fifth aspect, an embodiment of the present application also provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the model training method as described in any one of the first aspects or the steps of the three-dimensional reconstruction method described in the second aspect.

[0028] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method described in any one of the first aspects or the steps of the three-dimensional reconstruction method described in the second aspect are executed.

[0029] The model training method, three-dimensional reconstruction method, and apparatus, device, and storage medium provided in the embodiments of the present application can, during the model training process, determine the hollowing mask parameters of the observation ray corresponding to the sampling point based on the coordinates, normal vectors, and geometric feature vectors of the sampling point, thereby realizing the identification of hollowed-out parts within the bounding box of the target object. The depth values of the surface vertices on the observation ray can be determined based on the first symbol distance parameter of the sampling point, thereby realizing the surface depth detection of the target object. Then, based on the hollowing mask parameters of the observation ray and the depth values of the surface vertices, a model loss value is calculated, and then based on the model loss value, the parameters of the pre-trained surface distance extraction network are adjusted. This can achieve the targetedness of the hollowing parts within the bounding box of the target object and the depth values of the surface vertices on the observation ray during the training of the surface distance extraction network. This can improve the three-dimensional reconstruction accuracy of hollow objects with hollowed-out parts on the surface during the three-dimensional reconstruction based on the target surface distance extraction network, as well as the recovery of the hollow shape of the hollow object surface, thereby minimizing the workload of subsequent manual model repair.

[0030] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0033] Figure 2 A flow chart of a method for obtaining hollow mask parameters and depth values of observation rays in a model training method provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram showing the relationship between the reflection vector, normal vector, and direction vector of an observation ray of a sampling point provided in an embodiment of the present application;

[0035] Figure 4 Schematic diagram of the effects of the original image, diffuse reflection color image, and hollow mask image of the target object in the embodiment of the present application;

[0036] Figure 5 A flow chart of a method for calculating a model loss value in a model training method provided in an embodiment of the present application;

[0037] Figure 6 A schematic diagram showing a comparison of volume rendering weights corresponding to a pre-trained surface distance extraction network and a target surface distance extraction network provided in an embodiment of the present application;

[0038] Figure 7 Flowchart of the training method for the pre-trained surface distance extraction network and the pre-trained color decomposition network provided in the embodiments of the present application;

[0039] Figure 8 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0040] Figure 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0041] Figure 10 A schematic structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0042] Figure 11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0044] The 3D reconstruction method in one embodiment of the present disclosure can be applied to any application scenario involving virtual 3D models, such as 3D gaming scenarios, virtual scenes based on virtual reality, and virtual scenes based on augmented reality. During implementation, the 3D reconstruction method can be run on a local terminal device or a server. When the 3D reconstruction method is run on a server, it can be implemented and executed based on a cloud interaction system, where the cloud interaction system includes a server and a client device.

[0045] To facilitate a better understanding of the three-dimensional reconstruction method provided in the embodiments of the present application, the technical terms involved in the following embodiments of the present application are first explained below.

[0046] Technical term explanation:

[0047] SDF: Signed Distance Function (SDF), which is an implicit representation of the surface of a 3D model. It can determine the distance from a 3D point to the surface of an object in a finite region in 3D space and also defines the sign of the distance. For example, if the point is inside the object, it is positive; if the point is outside the object, it is negative; if the point is on the surface of the object, it is 0.

[0048] MLP: Multilayer Perceptron (MLP), a type of neural network architecture.

[0049] BBox: Bounding Box (BBox), refers to the minimum bounding box surrounding an object in three-dimensional (3D) space.

[0050] SFM: Structure From Motion (SFM) is a technique for estimating three-dimensional structure from a series of two-dimensional images containing visual motion information. SFM can be used to obtain camera parameters and sparse point clouds for each perspective.

[0051] Marching Cubes: Surface extraction algorithms, also known as surface rendering algorithms, are used to extract isosurfaces from volumetric data. The core concept is to define a surface by comparing the vertices to a user-specified threshold, after setting the 3D space and vertex scalar values. This algorithm then creates a triangular patch and connects all cube faces on the isosurface boundary to form a surface.

[0052] In order to clearly describe the three-dimensional reconstruction method provided in the embodiment of the present application, the model training method provided in the embodiment of the present application is first explained and illustrated by examples through multiple embodiments in combination with multiple drawings, so as to clearly illustrate the pre-trained network model used in the three-dimensional reconstruction method.

[0053] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the model training method may include:

[0054] S101: Acquire multiple sampling points within a bounding box of a target object.

[0055] Before performing this step, a bounding box of the target object may be obtained. In one implementation example, the bounding box of the target object can be automatically estimated based on multiple pre-acquired two-dimensional images of the target object from multiple perspectives. For example, the multiple two-dimensional images can be obtained by pre-capturing the target object in a real environment using a camera from multiple perspectives. For example, a preset pose estimation algorithm can be used to perform joint pose estimation on the multiple two-dimensional images to obtain three-dimensional point clouds from multiple perspectives and camera intrinsic and extrinsic parameters from multiple perspectives. Then, based on the three-dimensional point clouds and camera intrinsic and extrinsic parameters from multiple perspectives, a bounding box of the target object is estimated to obtain information about the target object's bounding box. The three-dimensional point cloud from each perspective can be, for example, a sparse point cloud. The preset pose estimation algorithm can be, for example, an SFM network. In another implementation example, the bounding box of the target object can be obtained based on bounding box information input by a user. The bounding box information of the target object may include the center position of the target object and the radius of the object.

[0056] When the bounding box of the target object is obtained, three-dimensional points within the bounding box may be sampled to obtain a plurality of sampling points.

[0057] S102: Input the coordinates of the plurality of sampling points into a pre-trained surface distance extraction network to obtain first signed distance parameters of the plurality of sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points.

[0058] The pre-trained surface distance extraction network can be, for example, a pre-trained SDF network. When S101 is executed, the coordinates of multiple sampling points are obtained, where the coordinates of each sampling point can be the three-dimensional coordinate x of the sampling point. Inputting the coordinates of the sampling point into the pre-trained surface distance extraction network yields the first signed distance parameter d1, the normal vector n, and the geometric feature vector z of the sampling point. The pre-trained SDF network can be a neural network with an MLP architecture, that is, an SDF network pre-defined and trained using an MLP architecture.

[0059] The geometric feature vector z can be used to characterize the surface feature properties of the object at the sampling point x. In other words, the input parameters of the pre-trained surface distance extraction network include the coordinates x of the sampling point within the bounding box, and the output parameters include the SDF value of the sampling point, the normal vector n, and the geometric feature vector z.

[0060] It should be noted that in this example, the pre-trained surface distance extraction network can be a surface distance extraction network pre-trained based on a real image of the target object, i.e., a real color image, combined with a volume rendering framework. The volume rendering framework can also be called a volume rendering framework.

[0061] S103 : Determine hollowing mask parameters of the observation ray corresponding to the sampling point according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point.

[0062] The hollowing mask parameter of the observation ray indicates whether the sampling point on the observation ray is located in a hollowed area. For example, the color of the sampling point can be predicted based on its coordinates, normal vector, and geometric eigenvector to obtain the color parameter of the observation ray corresponding to the sampling point. Based on the color parameter, the hollowing mask parameter of the observation ray corresponding to the sampling point is then determined to identify the hollowed area within the bounding box.

[0063] S104 : Determine the depth value of the surface vertex on the observation ray according to the first signed distance parameter of the sampling point.

[0064] For example, the surface of the target object can be extracted based on the first signed distance parameter of the sampling point to obtain the vertex on the surface of the target object corresponding to the observation ray of the sampling point, that is, the coordinates of the surface vertex on the observation ray. Once the coordinates of the surface vertex are obtained, the depth value of the surface vertex on the observation ray can be obtained by depth rendering the surface vertex. The depth value of the surface vertex can be used to represent the depth value of the surface vertex on the z-axis of the camera coordinate system corresponding to the observation ray, that is, the depth value between the position of the surface vertex in the camera coordinate system and the virtual camera position of the observation ray.

[0065] S105 , calculating a model loss value according to the hollowing mask parameters of the observation ray and the depth values of the surface vertices on the observation ray.

[0066] S106. Adjust the parameters of the pre-trained surface distance extraction network according to the model loss value to obtain the target surface distance extraction network.

[0067] The hollow mask parameter of the observation ray is used to indicate whether the sampling point on the observation ray is in a hollow area. The depth value of the surface vertex on the observation ray can be used to represent the volume rendering weight of the sampling point on the observation ray. The volume rendering weights of the hollow area and the non-hollow area are different, and the rendering color of the corresponding observation ray is also different. Therefore, in this embodiment, the hollow mask parameter of the observation ray and the depth value of the surface vertex on the observation ray are used to calculate the model loss value, and then the parameter adjustment process with the pre-trained surface distance extraction network is actually: model supervised training of the hollow mask parameter of the observation ray and the depth value of the surface vertex on the observation ray. In this way, the target surface distance extraction network can ensure the 3D reconstruction accuracy of hollow objects with hollow areas on the surface during the 3D reconstruction process. The depth value of the surface vertex on the observation ray used in the training process can to a certain extent assist the target surface distance extraction network in recovering the shape of the hole on the surface of the hollow object during the 3D reconstruction process.

[0068] For example, based on the model loss function value, a sampled gradient descent method is used to adjust the parameters of the pre-trained surface distance extraction network. Model training is then performed based on the pre-trained surface reconstruction network after the parameter adjustment until a preset stopping condition is met, thereby obtaining a target surface distance extraction network. The stopping condition may be, for example, reaching a preset maximum number of iterations or reaching a preset condition for the model loss function value.

[0069] The model training method provided in this embodiment can input the coordinates of multiple sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain the first signed distance parameters, normal vectors and geometric feature vectors of the multiple sampling points. Then, based on the coordinates, normal vectors and geometric feature vectors of the sampling points, the hollowing mask parameters of the observation rays corresponding to the sampling points are determined. Based on the first signed distance parameters of the sampling points, the depth values of the surface vertices on the observation rays are determined, thereby realizing the recognition of hollowed-out parts within the bounding box of the target object and the surface depth detection of the target object. Then, based on the hollowing mask parameters of the observation rays and the depth values of the surface vertices on the observation rays, the model loss value is calculated. Based on the model loss value, the parameters of the pre-trained surface distance extraction network are adjusted to obtain the target surface distance extraction network. In the model training method provided in this embodiment, during the model training process, the hollow mask parameters of the observation ray corresponding to the sampling point and the depth values of the surface vertices on the observation ray can be determined according to the first symbol distance parameter of the sampling point, thereby realizing the recognition of the hollow parts within the bounding box of the target object and the surface depth detection of the target object. Then, based on the hollow mask parameters of the observation ray and the depth values of the surface vertices, the model loss value is calculated, and then the parameters of the pre-trained surface distance extraction network are adjusted based on the model loss value. In this way, during the training process of the surface distance extraction network, the hollow parts within the bounding box of the target object and the depth values of the surface vertices on the observation ray can be targeted. This can improve the three-dimensional reconstruction accuracy of the hollow object with hollow parts on the surface during the three-dimensional reconstruction based on the target surface distance extraction network, and the recovery of the hollow shape of the hollow object surface, thereby minimizing and avoiding the workload of subsequent manual model repair.

[0070] In order to more clearly understand the model training method provided in the embodiments of the present application, the following further describes in detail the hollowing mask parameters and depth values for obtaining the observation rays in combination with multiple embodiments. Figure 2 A flow chart of a method for obtaining hollow mask parameters and depth values of observation rays in a model training method provided in an embodiment of the present application. Figure 2 As shown, in S103 shown above, determining the hollowing mask parameters of the observation ray corresponding to the sampling point according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point may include:

[0071] S201 , using a pre-trained color decomposition network, and obtaining multiple color components of an observation ray according to the coordinates of a sampling point, a normal vector of the sampling point, and a geometric feature vector of the sampling point.

[0072] The multiple color components include at least: diffuse color C d In one example, to achieve color decomposition of the observation ray, in a specific implementation example, the coordinate x of the sampling point, the normal vector n of the sampling point, and the geometric feature vector z of the sampling point can be input into the pre-trained color decomposition network, and multiple color components of the observation ray including the diffuse color can be obtained. For example, the pre-trained color decomposition network may include at least: a first decomposition network to achieve decomposition of the diffuse color. The pre-trained color decomposition network C can be a network defined by the following formula: C = C d (x, n, z), where C d (x, n, z) can be used to characterize the first decomposition network. The pre-trained color decomposition network can also be a neural network with an MLP architecture, that is, the pre-trained color decomposition network can be: a color decomposition network pre-defined and trained using an MLP architecture.

[0073] In the solution of this example, a pre-trained color decomposition network can be used based on the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point to decompose the diffuse color of the observation ray from the color of the observation ray, and then the hollow recognition of the observation ray can be achieved based on the diffuse color.

[0074] In another example, in order to avoid the problem that the diffuse color may be interfered with by other color components such as specular reflection related components, which makes the recognition of the diffuse color less accurate, when using a pre-trained color decomposition network to decompose the observation ray, in addition to inputting the coordinate x of the sampling point, the normal vector n of the sampling point, and the geometric feature vector z of the sampling point, the direction vector v of the observation ray and the reflection vector r of the observation ray can also be input. That is to say, in this example, according to the coordinate x of the sampling point, the normal vector n of the sampling point, the geometric feature vector z of the sampling point, the direction vector v of the observation ray, and the reflection vector r of the observation ray, the diffuse color C can be obtained, including: d , Specular color C s and the specular reflection coefficient C t The reflection vector represents the reflection direction of the observation ray relative to the normal vector of the sampling point.

[0075] In this example, multiple color components of the observation ray can be obtained according to the coordinate x of the sampling point, the normal vector n of the sampling point, the geometric characteristic vector z of the sampling point, the direction vector v of the observation ray, and the reflection vector r of the observation ray, including the diffuse reflection color C d , Specular color C s and the specular reflection coefficient C t That is, in this example, while realizing the decomposition of the diffuse reflection color on the observation ray, the decomposition of the specular reflection color and the specular reflection coefficient is also realized, thereby avoiding the interference of the specular reflection color and the specular reflection coefficient on the diffuse reflection color, making the decomposition of the diffuse reflection color more accurate.

[0076] Accordingly, the pre-trained color decomposition network may further include: a second decomposition network and a third decomposition network to respectively realize the specular reflection color C s and the specular reflection coefficient C t In this example, the pre-trained color decomposition network can be a network defined by the following formula: C = C d (x, n, z) + C s (x, n, z, v, r) + C t (x, v, r), where C d (x, n, z) can be used to characterize the first decomposition network, C s (x, n, z, v, r) and C t (x, v, r) are used to represent the second decomposition network and the third decomposition network respectively. In a specific implementation example, the first decomposition network in the pre-trained color decomposition network can obtain the diffuse reflection color C of the observation ray based on the coordinate x of the sampling point, the normal vector n of the sampling point, and the geometric feature vector z of the sampling point. d The second decomposition network in the pre-trained color decomposition network obtains the specular reflection color C of the observation ray based on the coordinate x of the sampling point, the normal vector n of the sampling point, the geometric feature vector z of the sampling point, the direction vector v of the observation ray, and the reflection vector r of the observation ray. s The third decomposition network in the pre-trained color decomposition network obtains the specular reflection coefficient C of the observation ray based on the coordinate x of the sampling point, the direction vector v of the observation ray, and the reflection vector r of the observation ray. t .

[0077] The reflection vector r of the observation ray can be obtained, for example, based on the normal vector n of the sampling point and the direction vector v of the observation ray using r = 2(-v·n)n + v. To more clearly understand the relationship between the reflection vector r of the observation ray, the normal vector n of the sampling point, and the direction vector v of the observation ray, the following diagram is used to illustrate the relationship. Figure 3Schematic diagram of the relationship between the reflection vector, normal vector and direction vector of the observation ray of the sampling point provided in the embodiment of the present application. Figure 3 As shown, taking the coordinate x of the sampling point on the surface of the object as an example, the reflection vector r of the observation ray is the reflection direction of the direction vector v of the observation ray with respect to the normal vector n.

[0078] S202: Determine the hollowing mask parameters of the observation ray according to the diffuse reflection color.

[0079] If the diffuse reflection color is black, the hollow mask parameter of the observation ray is determined to be the first mask parameter, which is used to indicate that the sampling point on the observation ray is a hollow area. If the diffuse reflection color is not black, the hollow mask parameter of the observation ray is determined to be the second mask parameter, which is used to indicate that the sampling point on the observation ray is not a hollow area.

[0080] The hollowing mask parameters can be, for example, binary mask parameters. For example, the first mask parameter can be 1, indicating that the sampling point of the observation ray is a hollowed portion, and the hollowed portion within the bounding box is marked. The second mask parameter can be 0, indicating that the sampling point of the observation ray is a non-hollowed portion, and the non-hollowed portion within the bounding box is marked. In other words, in this embodiment, the hollowing mask parameters of the observation ray can be determined based on the color component of the observation ray, thereby achieving separate marking of the hollowed and non-hollowed portions within the bounding box.

[0081] In order to more clearly understand the implementation of the hollow mask in the embodiment of the present application, the following is an example explanation combined with the effect diagram information. Figure 4 The figure is a schematic diagram of the effect of the original image, diffuse reflection color map and hollow mask map of the target object in the embodiment of the present application. When the pre-trained color decomposition network is used to obtain multiple color components of each observation ray, multiple color component maps of the target object can be generated based on the multiple color components of each observation ray. For example, the specular reflection color C of each observation ray can be used to generate multiple color component maps of the target object. s ,generate Figure 4 The specular reflection color map of the target object shown in FIG; the diffuse reflection color C based on each observation ray d ,generate Figure 4 Diffuse color map of the target object shown in .

[0082] When the hollowing mask parameters of each observation ray are obtained in this embodiment, the hollowing mask parameters of each observation ray can also be used to generate Figure 4 The hollowed-out mask map of the target object shown in . Figure 4 The hollow mask diagram in FIG. 1 clearly shows the hollowed-out parts and non-hollowed-out parts of the target object marked in this embodiment.

[0083] The solution of the embodiment of the present application can use a pre-trained color decomposition network to obtain the diffuse reflection color of the observation ray based on the coordinates of the sampling point, the normal vector of the sampling point and the geometric feature vector of the sampling point, and then determine the hollow mask parameters of the observation ray to achieve accurate recognition and detection of the hollow part; on the basis of achieving the diffuse reflection color, the direction vector of the observation ray and the reflection vector of the observation ray are also input into the pre-trained color decomposition network to achieve the decomposition of the specular reflection color and the specular reflection coefficient on the observation ray, thereby avoiding the interference of the specular reflection color and the specular reflection coefficient on the diffuse reflection color, making the decomposition of the diffuse reflection color more accurate, and achieving further accurate identification of the hollow part.

[0084] Regarding the depth detection of surface vertices on the observation ray in the above embodiment, the present application also provides the following possible implementation example. In this example, based on the first signed distance parameter of the sampling points, a preset surface extraction algorithm is used to obtain a three-dimensional surface mesh of the target object, and depth rendering is performed on the three-dimensional surface mesh to obtain the depth values of the surface vertices on the observation ray.

[0085] The preset surface extraction algorithm may be an isosurface extraction algorithm, such as a Marching Cubes algorithm. The three-dimensional surface mesh of the target object may also be referred to as three-dimensional surface data, or surface mesh data, etc., which may include at least the coordinates of a plurality of surface vertices of the target object.

[0086] In a possible implementation, depth rendering may be performed on each surface vertex on the surface three-dimensional mesh according to information of the observation ray, such as the direction vector of the observation ray, to obtain the depth value of the surface vertex on the observation ray.

[0087] The solution of the embodiment of the present application can obtain the surface three-dimensional mesh of the target object and perform depth rendering on the surface three-dimensional mesh based on the first symbol distance parameter of the sampling point in combination with a preset surface extraction algorithm, and obtain the depth value of the surface vertex on the observation ray, thereby realizing the acquisition of the depth value of the observation ray. Then, the target surface distance extraction network trained based on the model loss function value obtained in this way can effectively ensure the three-dimensional reconstruction accuracy of the hollow object and the restoration of the hollow shape of the hollow object surface.

[0088] In another possible implementation, during the process of adjusting the parameters of the pre-trained surface distance extraction network, the parameters of the pre-trained color decomposition network may also be adjusted based on the model loss function value, so that during the training of the pre-trained surface distance extraction network, the pre-trained color decomposition network is also trained. Therefore, during the process of retraining the model based on the pre-trained surface distance extraction network after parameter adjustment, model training can be performed based on the pre-trained surface distance extraction network after parameter adjustment and the pre-trained color network after parameter adjustment until a preset stopping iteration condition is met, thereby obtaining a target surface distance extraction network and a target color decomposition network. The target color decomposition network can be used to perform three-dimensional reconstruction of a target object using the target surface distance extraction network to obtain a three-dimensional surface mesh of the target object, that is, three-dimensional surface data, and also to obtain the rendering color of each surface vertex on the target object.

[0089] The following continues to explain the specific implementation of calculating the model loss value based on the hollow mask parameters of the observation ray and the depth value of the surface vertex during the model training process with the help of the accompanying drawings, so as to clearly explain how to realize the supervised training of the surface distance extraction network based on the hollow mask parameters of the observation ray and the depth value of the surface vertex. Figure 5 This is a flow chart of a method for calculating model loss value in a model training method provided in an embodiment of the present application. Figure 5 As shown, in S104 shown in the above embodiment, according to the hollowing mask parameters of the observation ray and the depth values of the surface vertices on the observation ray, calculating the model loss value may include:

[0090] S501: Obtain the initial volume rendering weight of each sampling point on the observation ray.

[0091] For example, the initial volume rendering weight of each sampling point on the observation ray may be determined according to the first signed distance parameter of the sampling point on the observation ray.

[0092] S502: Determine a hollowing ray from a plurality of observation rays according to a hollowing mask parameter of the observation ray.

[0093] In a possible implementation example, based on the hollowing mask parameters of multiple observation rays, the observation ray corresponding to the sampling point in the hollowed part can be determined as a hollowed ray from the multiple observation rays, and the other observation rays other than the hollowed rays in the multiple observation rays can be determined as non-hollowed rays.

[0094] S503 : Determine a first loss value according to the depth value of the surface vertex on the hollowed-out ray and the initial volume rendering weight of each sampling point on the hollowed-out ray and using a preset volume rendering weight constraint condition.

[0095] The preset volume rendering weight constraint condition is: the sum of the initial volume rendering weights of each sampling point whose depth value on the hollow ray is less than the depth value of the surface vertex is 0. In a specific implementation example, the sum of the initial volume rendering weights of each sampling point whose depth value on the hollow ray is less than the depth value of the surface vertex can be first determined based on the depth value of the surface vertex on the hollow ray, and the first loss value is calculated based on the sum of the initial volume rendering weights and the preset reference weight of 0. The larger the first loss value, the farther the sum of the initial volume rendering weights is from the healing reference weight; the smaller the first loss value, the closer the sum of the initial volume rendering weights is to the preset reference weight of 0.

[0096] S504: Calculate a model loss value based on the first loss value.

[0097] The first loss value may be referred to as a weighted loss value, also known as a weighted loss value. In this embodiment, the model loss value may be directly calculated based on the first loss value, or the model loss value may be calculated based on the first loss value and the loss values of other dimensions.

[0098] To better understand the supervised training of the surface distance extraction network in this scheme, the depth values of each sampling point obtained by the pre-trained surface distance extraction network and the target surface distance extraction network are compared and analyzed in combination with the schematic diagram below. Figure 6 A schematic diagram of the comparison of volume rendering weights corresponding to a pre-trained surface distance extraction network and a target surface distance extraction network provided in an embodiment of the present application. Figure 6 The corresponding relationship between the depth value and volume rendering weight of the sampling point in stage 1 and stage 2 is shown in the figure. For a hollow ray sampling 3D points from near to far, the horizontal axis is the depth value t from near to far, and the vertical axis is the volume rendering weight w(t) corresponding to different depth values.

[0099] Among them, stage 1 is the correspondence between the initial volume rendering weights of each sampling point on the observation ray obtained by the pre-trained surface distance extraction network and the depth value of each sampling point, and stage 2 is the correspondence between the initial volume rendering weights of each sampling point on the observation ray obtained by the target surface distance extraction network and the depth value of each sampling point.

[0100] In stage 1, the rendered depth value is at the cutout position t1, while the true depth value t2 should be after the cutout position. The cutout position t1 can be used to represent the rendered depth value at the cutout position, i.e., the sampling point on the cutout ray. The true depth value t2 can be used to represent the true depth value at the cutout position, i.e., the sampling point on the cutout ray.

[0101] In stage 2, since the preset volume rendering weight constraint condition is used when calculating the first loss value during the training of the target surface distance extraction network, the preset condition that the sum of the initial volume rendering weights of each sampling point whose depth value on the hollow ray is less than the depth value of the surface vertex is 0 is set, that is, the sum of the rendering volume weights w before the hollow position t1 is constrained to be 0, then the rendering depth value of the hollow position obtained based on the target surface distance extraction network after training in stage 2 is the true depth value t2.

[0102] The solution of this embodiment can determine the first loss value based on the depth value of the surface vertices on the hollow ray and the initial volume rendering weights of each sampling point on the hollow ray, and then calculate the model loss value by using the preset volume rendering weight constraint condition. In this way, the pre-trained surface distance extraction network is trained based on the model loss value, thereby realizing the supervised training of the model based on the hollow mask parameters and depth values, improving the accuracy of the supervised training of the surface distance extraction network, and thus ensuring the three-dimensional reconstruction accuracy of the hollow object by the trained target surface distance extraction network as much as possible.

[0103] In some other possible implementation examples of this embodiment, volume rendering may be performed based on the initial volume rendering weight on the observation ray to obtain a rendering color of the observation ray; and a second loss value may be calculated based on the rendering color of the observation ray and a true color corresponding to the observation ray; wherein the true color of the observation ray is the color of the pixel position corresponding to the observation ray in the two-dimensional image of the target object;

[0104] Calculate a model loss value based on the first loss value and the second loss value.

[0105] That is to say, the solution of this embodiment, while calculating the first loss value, i.e., the weight loss value, can also calculate the second loss value, i.e., the color loss value, based on the rendered color of the observation ray and the true color corresponding to the observation ray, and then the first loss value and the second loss value can be accumulated to obtain the model loss value.

[0106] In the process of obtaining the rendered color of the observation ray, for example, the following method can be used: volume rendering is performed based on the initial volume rendering weight of the observation ray and the initial rendering color of the observation ray to obtain the rendered color of the observation ray. The initial rendering color of the observation ray can be obtained, for example, based on multiple color components of the observation ray, and the multiple color components of the observation ray can be obtained based on the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point using the above-mentioned pre-trained color decomposition network. For details, please refer to the above and will not be repeated here.

[0107] In this implementation example of the present embodiment, since the second loss value can be calculated based on the rendered color of the observation ray and the true color of the observation ray, and then the model loss value can be calculated in combination with the first loss value, the pre-trained surface distance extraction network is trained based on the model loss value. In addition to the supervised training of the model based on the hollow mask parameters and the depth value, the supervised training can also be combined with the two-dimensional image of the target object, that is, the color in the real image, thereby improving the accuracy of the supervised training of the surface distance extraction network, thereby ensuring as much as possible the three-dimensional reconstruction accuracy of the hollow object by the trained target surface distance extraction network.

[0108] Before executing the aforementioned model training method, the surface distance extraction network and color decomposition network used in the embodiments of the present application can also be pre-trained by pre-training the initial surface distance extraction network and color decomposition network, i.e., a first stage of model training is also included. This is explained below with reference to the flowchart. Figure 7 This is a flow chart of the training method for the pre-trained surface distance extraction network and the pre-trained color decomposition network provided in the embodiment of the present application. Figure 7 As shown, the training method of the pre-trained surface distance extraction network and the pre-trained color decomposition network may include, for example:

[0109] S701: Using an initial surface distance extraction network, extract parameters according to the coordinates of multiple sampling points to obtain second signed distance parameters of the multiple sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points.

[0110] In a specific implementation process, the coordinates of the sampling points can be input into the initial surface distance extraction network to obtain the second signed distance parameter of the sampling points, the normal vector of the sampling points, and the geometric feature vector of the sampling points.

[0111] S702 : Using an initial color decomposition network, obtain multiple initial color components of the observation ray according to the coordinates of the sampling points, the normal vectors of the sampling points, and the geometric characteristic vectors of the sampling points.

[0112] Correspondingly, the coordinates of the sampling points, the normal vectors of the sampling points, and the geometric feature vectors of the sampling points can all be input into the initial color decomposition network to obtain multiple initial color components of the observation ray.

[0113] S703 : Perform volume rendering according to the second signed distance parameter of the sampling point to obtain a volume rendering weight of the sampling point.

[0114] S704 : Obtain an initial rendering color of the observation ray according to multiple initial color components of the observation ray and the volume rendering weight of the sampling point.

[0115] For example, multiple initial color components of the observation ray may be weighted according to the volume rendering weight of the sampling point to obtain the initial rendering color of the observation ray.

[0116] S705 , adjusting parameters of the initial surface distance extraction network and the initial color decomposition network according to the initial rendering color of the observation ray and the true color corresponding to the observation ray, to obtain a pre-trained surface distance network and a pre-trained color decomposition network.

[0117] For example, the initial model loss value can be calculated based on the initial rendered color of the observation ray and the true color corresponding to the observation ray. The parameters of the initial surface distance extraction network and the initial color decomposition network are then adjusted based on the initial model loss value until the preset stop iteration condition is met. This results in a pre-trained surface distance network and a pre-trained color decomposition network. The true color corresponding to the observation ray is the color in the two-dimensional image of the target object at the pixel position corresponding to the observation ray, that is, the color in the real image.

[0118] The method provided in the embodiment of the present application can, under the volume rendering framework, pre-train color supervision training using real images of the target object to obtain a pre-trained surface distance extraction network and a pre-trained color decomposition network, and then perform model training based on the pre-trained surface distance extraction network and the pre-trained color decomposition network, thereby realizing phased training of the surface distance extraction network and the color decomposition network, thereby improving the training accuracy of the target surface distance extraction network and the target color decomposition network, and further improving the three-dimensional reconstruction accuracy of the target surface distance extraction network and the target color decomposition network for hollow objects.

[0119] The following is a detailed explanation and description of the specific implementation process of 3D reconstruction with examples. Figure 8 The flowchart of the three-dimensional reconstruction method provided in the embodiment of the present application is as follows: Figure 8 As shown, the method may include:

[0120] S801: Acquire sampling points within the bounding box of the target object.

[0121] In this embodiment, the specific implementation of the bounding box of the target object can be estimated based on multiple two-dimensional images under multiple perspectives as described in the above embodiment S101 to obtain the bounding box of the target object. The bounding box of the target object can also be stored during the training process of the target surface distance extraction network, and the bounding box of the target object obtained during the network training process can be directly obtained here.

[0122] Secondly, for the multiple sampling points within the bounding box, a similar method to the above S101 can be used to obtain multiple sampling points. Alternatively, during the training process of the target surface distance extraction network, the multiple sampling points obtained based on the bounding box extraction of the target object can be stored, and the multiple sampling points within the bounding box obtained during the network training process can be directly obtained here.

[0123] S802: Input the coordinates of multiple sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the multiple sampling points.

[0124] The surface reconstruction network is a target surface reconstruction network obtained by using the model training method provided by any of the above embodiments.

[0125] S803 : Perform three-dimensional reconstruction of the target object using a preset surface extraction algorithm according to the target symbol distance parameter to obtain a three-dimensional mesh of the target surface of the target object.

[0126] In a possible implementation, for example, a preset surface extraction algorithm can be used to extract an isosurface based on the target signed distance parameters of multiple sampling points. The resulting 3D points with a signed distance parameter of 0 serve as the target surface 3D mesh of the target object. The surface extraction algorithm can be, for example, the aforementioned Marching cubes algorithm. In this case, the target surface 3D mesh can include coordinate information for each surface vertex, which can be used to render and display a colorless 3D virtual model of the target object.

[0127] The target surface reconstruction network used in the three-dimensional reconstruction method provided in the embodiment of the present application can improve the three-dimensional reconstruction accuracy of hollow objects with hollow parts on the surface during the three-dimensional reconstruction process based on the target surface distance extraction network, and restore the shape of the holes on the surface of the hollow objects, due to its targeting of the depth values of the hollow parts within the target object's boundary box and the surface vertices on the observation ray during the model training process, thereby minimizing and avoiding the workload of subsequent manual repair of the model.

[0128] In the case of using the target surface distance extraction network to perform three-dimensional reconstruction and obtain a three-dimensional mesh of the target surface of the target object provided in this embodiment, the embodiment of the present application can also provide a possible implementation example of color rendering of the three-dimensional surface data to determine the rendering color of each surface vertex. It should be noted that when the rendering color of each surface vertex is obtained, a colored three-dimensional virtual model of the hollow object can be rendered and displayed based on the three-dimensional surface data and the rendering color of each surface vertex.

[0129] In the specific application implementation process, a target surface distance extraction network can also be used to obtain the target signed distance parameters of the surface vertices, the normal vectors of the surface vertices, and the geometric feature vectors of the surface vertices. According to the coordinates of the surface vertices, the normal vectors of the surface vertices, and the geometric feature vectors of the surface vertices, a target color decomposition network is used to obtain multiple target color components of the surface vertices. Volume rendering is performed according to the target signed distance parameters of the surface vertices to obtain the volume rendering weights of the sampling points. Then, according to the multiple target color components of the surface vertices and the volume rendering weights of the surface vertices, the target rendering colors of the surface vertices are obtained.

[0130] The method provided in this embodiment can obtain the rendering color of each surface vertex of the target object when the three-dimensional mesh of the target surface of the target object is obtained through three-dimensional reconstruction. It not only realizes the reconstruction of the three-dimensional model of the target object, but also realizes the rendering of the surface color of the target object. The final three-dimensional reconstruction data can reflect the vertex information of the object surface and the color information of each vertex on the object surface, thereby improving the three-dimensional reconstruction accuracy and information completeness of the hollow object.

[0131] The following describes the model training device, three-dimensional reconstruction device, equipment and storage medium provided by this application for execution. The specific implementation process and technical effects are described above and will not be repeated below.

[0132] Figure 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the model training device 900 may include:

[0133] The first acquisition module 901 is configured to acquire a plurality of sampling points within a bounding box of a target object.

[0134] The first extraction module 902 is used to input the coordinates of multiple sampling points into a pre-trained surface distance extraction network to obtain first signed distance parameters of the multiple sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points.

[0135] Determination module 903 is used to determine the hollowing mask parameters of the observation ray corresponding to the sampling point based on the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point; and determine the depth value of the surface vertex on the observation ray based on the first signed distance parameter of the sampling point, wherein the hollowing mask parameters of the observation ray are used to indicate whether the sampling point on the observation ray is in a hollowed area.

[0136] The calculation module 904 is used to calculate the model loss value according to the hollowing mask parameters of the observation ray and the depth value of the surface vertex on the observation ray.

[0137] The training module 905 is used to adjust the parameters of the pre-trained surface distance extraction network according to the model loss value to obtain the target surface distance extraction network.

[0138] The model training device provided in this embodiment can, during the model training process, determine the hollowing mask parameters of the observation ray corresponding to the sampling point based on the coordinates, normal vectors, and geometric feature vectors of the sampling point, and determine the depth values of the surface vertices on the observation ray based on the first signed distance parameters of the sampling point, thereby realizing the identification of hollowed-out parts within the bounding box of the target object and the detection of the surface depth of the target object. Then, based on the hollowing mask parameters of the observation ray and the depth values of the surface vertices, a model loss value is calculated, and then the parameters of the pre-trained surface distance extraction network are adjusted based on the model loss value. This can achieve the targetedness of the hollowing parts within the bounding box of the target object and the depth values of the surface vertices on the observation ray during the training of the surface distance extraction network. This can improve the three-dimensional reconstruction accuracy of hollow objects with hollowed-out parts on the surface during the three-dimensional reconstruction based on the target surface distance extraction network, as well as the recovery of the shape of the holes on the surface of the hollow objects, thereby minimizing and avoiding the workload of subsequent manual model repair.

[0139] In one feasible embodiment, the determination module 903 is specifically configured to:

[0140] Using a pre-trained color decomposition network, multiple color components of the observation ray are obtained according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point; the multiple color components include at least: diffuse reflection color;

[0141] Determines the hollowing mask parameters for the viewing ray based on the diffuse color.

[0142] In one feasible embodiment, the determination module 903 is specifically configured to:

[0143] According to the first signed distance parameter of the sampling points, a preset surface extraction algorithm is used to obtain a three-dimensional surface mesh of the target object;

[0144] Perform depth rendering on the surface 3D mesh to obtain the depth value of the surface vertex on the observation ray.

[0145] In a feasible implementation scheme, the determination module 903 is specifically used to: if the diffuse reflection color is black, determine the hollow mask parameter of the observation ray as the first mask parameter, and the first mask parameter is used to indicate that the sampling point on the observation ray is a hollow part; if the diffuse reflection color is not black, determine the hollow mask parameter of the observation ray as the second mask parameter; the second mask parameter is used to indicate that the sampling point on the observation ray is a non-hollow part.

[0146] In a feasible implementation scheme, the calculation module 904 is specifically used to: obtain the initial volume rendering weight of each sampling point on the observation ray; determine the hollow ray from multiple observation rays based on the hollow mask parameters of the observation ray; determine the first loss value using a preset volume rendering weight constraint condition based on the depth value of the surface vertex on the hollow ray and the initial volume rendering weight of each sampling point on the hollow ray; wherein the preset volume rendering weight constraint condition is: the sum of the initial volume rendering weights of each sampling point on the hollow ray whose depth value is less than the depth value of the surface vertex is 0; and calculate the model loss value based on the first loss value.

[0147] In a feasible implementation scheme, the calculation module 904 is further specifically used to: perform volume rendering according to the initial volume rendering weight on the observation ray to obtain the rendering color of the observation ray; calculate the second loss value according to the rendering color of the observation ray and the true color corresponding to the observation ray; wherein the true color of the observation ray is the color of the pixel position corresponding to the observation ray in the two-dimensional image of the target object; and calculate the model loss value according to the first loss value and the second loss value.

[0148] In a feasible implementation scheme, the training module 905 is further used to: adjust the parameters of the pre-trained color decomposition network according to the model loss value to obtain the target color decomposition network.

[0149] In one feasible embodiment, determination module 903 is specifically configured to utilize a pretrained color decomposition network to obtain multiple color components of the observation ray based on the coordinates of the sampling point, the normal vector of the sampling point, the geometric feature vector of the sampling point, the direction vector of the observation ray, and the reflection vector of the observation ray. The reflection vector represents the reflection direction of the observation ray relative to the normal vector of the sampling point. The multiple color components also include specular reflection color and specular reflection coefficient.

[0150] In a feasible implementation scheme, the pre-trained surface distance extraction network and the pre-trained color decomposition network are networks trained in the following manner:

[0151] An initial surface distance extraction network is used to extract parameters according to the coordinates of multiple sampling points to obtain the second signed distance parameters, normal vectors and geometric feature vectors of the sampling points.

[0152] An initial color decomposition network is used to obtain multiple initial color components of the observation ray according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point;

[0153] Perform volume rendering according to the second signed distance parameter of the sampling point to obtain a volume rendering weight of the sampling point;

[0154] Obtaining the initial rendering color of the observation ray according to multiple initial color components of the observation ray and the volume rendering weight of the sampling point;

[0155] According to the initial rendering color of the observation ray and the true color corresponding to the observation ray, the parameters of the initial surface distance extraction network and the initial color decomposition network are adjusted to obtain the pre-trained surface distance extraction network and the pre-trained color decomposition network.

[0156] Figure 10 A schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application is shown in FIG. Figure 10 As shown, the 3D reconstruction apparatus 1000 may include:

[0157] The second acquisition module 1001 is used to acquire sampling points within the bounding box of the target object;

[0158] The second extraction module 1002 is used to input the coordinates of multiple sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the multiple sampling points; wherein the surface distance extraction network is a target surface distance extraction network obtained by using the model training method described in any of the above embodiments.

[0159] The reconstruction module 1003 is used to perform three-dimensional reconstruction of the target object using a preset surface extraction algorithm according to the target symbol distance parameter to obtain a three-dimensional mesh of the target surface of the target object.

[0160] The target surface reconstruction network used in the three-dimensional reconstruction method provided in this embodiment can improve the three-dimensional reconstruction accuracy of hollow objects with hollow parts on the surface during the three-dimensional reconstruction process based on the target surface distance extraction network, and restore the shape of the holes on the surface of the hollow objects, because it is targeted at the hollow parts within the target object's boundary box and the depth values of the surface vertices on the observation ray during the model training process, thereby minimizing and avoiding the workload of subsequent manual repair of the model.

[0161] Figure 11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is as follows: Figure 11 As shown, the computer device includes: a processor 1101, a storage medium 1102 and a bus 1103. The storage medium 1102 stores machine-readable instructions executable by the processor 1101. When the computer device runs a model training method as in the embodiment, the processor 1101 communicates with the storage medium 1102 via the bus 1103. The processor 1101 executes the machine-readable instructions. The processor 1101 performs the preamble of the method item to perform the following steps:

[0162] Get multiple sampling points within the bounding box of the target object;

[0163] Input the coordinates of the plurality of sampling points into a pre-trained surface distance extraction network to obtain parameter extraction results of the plurality of sampling points, wherein the parameter extraction results of the sampling points include: a first signed distance parameter of the sampling points;

[0164] Determine, based on the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point, a hollowing mask parameter of the observation ray corresponding to the sampling point, wherein the hollowing mask parameter of the observation ray is used to indicate whether the sampling point is in a hollowed portion on the observation ray;

[0165] Determine the depth value of the surface vertex on the observation ray according to the first signed distance parameter of the sampling point;

[0166] Calculate the model loss value according to the hollowing mask parameters of the observation ray and the depth value of the surface vertex on the observation ray;

[0167] According to the model loss value, the parameters of the pre-trained surface distance extraction network are adjusted to obtain the target surface distance extraction network.

[0168] In a feasible implementation manner, when the processor 1101 determines the hollowing mask parameters of the observation ray corresponding to the sampling point based on the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point, it is specifically configured to:

[0169] Using a pre-trained color decomposition network, multiple color components of the observation ray are obtained according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point; the multiple color components include at least: diffuse reflection color;

[0170] Determines the hollowing mask parameters for the viewing ray based on the diffuse color.

[0171] In a feasible implementation manner, when the processor 1101 determines the depth value of the surface vertex on the observation ray according to the first signed distance parameter of the sampling point, it is specifically configured to:

[0172] According to the first signed distance parameter of the sampling points, a preset surface extraction algorithm is used to obtain a three-dimensional surface mesh of the target object;

[0173] Perform depth rendering on the surface 3D mesh to obtain the depth value of the surface vertex on the observation ray.

[0174] In a feasible implementation manner, when determining the hollowing mask parameters of the observation ray according to the diffuse reflection color, the processor 1101 is specifically configured to:

[0175] If the diffuse reflection color is black, the hollow mask parameter of the observation ray is determined to be the first mask parameter, and the first mask parameter is used to indicate that the sampling point on the observation ray is a hollow part;

[0176] If the diffuse reflection color is not black, the hollow mask parameter of the observation ray is determined to be the second mask parameter; the second mask parameter is used to indicate that the sampling point on the observation ray is a non-hollow part.

[0177] In a feasible implementation, when the processor 1101 calculates the model loss value based on the hollowing mask parameter of the observation ray and the depth value of the surface vertex on the observation ray, it is specifically configured to:

[0178] Get the initial volume rendering weight of each sampling point on the observation ray;

[0179] Determining a hollowing ray from a plurality of observation rays according to a hollowing mask parameter of the observation ray;

[0180] Determining a first loss value based on the depth values of the surface vertices on the hollowed-out ray and the initial volume rendering weights of each sampling point on the hollowed-out ray using a preset volume rendering weight constraint condition; wherein the preset volume rendering weight constraint condition is that the sum of the initial volume rendering weights of each sampling point on the hollowed-out ray whose depth value is less than the depth value of the surface vertex is 0;

[0181] Calculate the model loss value based on the first loss value.

[0182] In a feasible embodiment, before executing the calculation of the model loss value based on the first loss value, the processor 1101 is also used to: perform volume rendering based on the initial volume rendering weight on the observation ray to obtain the rendering color of the observation ray; calculate the second loss value based on the rendering color of the observation ray and the true value color corresponding to the observation ray; wherein the true value color of the observation ray is the color of the pixel position corresponding to the observation ray in the two-dimensional image of the target object.

[0183] When the processor 1101 calculates the model loss value based on the first loss value, it is specifically used to: calculate the model loss value based on the first loss value and the second loss value.

[0184] In a feasible implementation scheme, the processor 1101 is further configured to adjust parameters of the pre-trained color decomposition network according to the model loss value to obtain a target color decomposition network.

[0185] In one feasible embodiment, when the processor 1101 executes the pre-trained color decomposition network to obtain multiple color components of the observation ray according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point, it is specifically configured to:

[0186] Using a pre-trained color decomposition network, multiple color components of the observation ray are obtained based on the coordinates of the sampling point, the normal vector of the sampling point, the geometric feature vector of the sampling point, the direction vector of the observation ray, and the reflection vector of the observation ray. The reflection vector represents the reflection direction of the observation ray relative to the normal vector of the sampling point. The multiple color components also include specular reflection color and specular reflection coefficient.

[0187] In a feasible embodiment, the training surface distance extraction network and the pre-trained color decomposition network are networks trained in the following manner:

[0188] An initial surface distance extraction network is used to extract parameters according to the coordinates of multiple sampling points to obtain the second signed distance parameters, normal vectors and geometric feature vectors of the sampling points.

[0189] An initial color decomposition network is used to obtain multiple initial color components of the observation ray according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point;

[0190] Perform volume rendering according to the second signed distance parameter of the sampling point to obtain a volume rendering weight of the sampling point;

[0191] Obtaining the initial rendering color of the observation ray according to multiple initial color components of the observation ray and the volume rendering weight of the sampling point;

[0192] According to the initial rendering color of the observation ray and the true color corresponding to the observation ray, the parameters of the initial surface distance extraction network and the initial color decomposition network are adjusted to obtain the pre-trained surface distance extraction network and the trained color decomposition network.

[0193] When a computer device executes a three-dimensional reconstruction method as described in the embodiment, the processor 1101 communicates with the storage medium 1102 via the bus 1103, and the processor 1101 executes the machine-readable instructions, including the preamble of the method item, to perform the following steps:

[0194] Get the sampling points within the bounding box of the target object;

[0195] Input the coordinates of multiple sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the multiple sampling points; wherein the surface distance extraction network is a target surface distance extraction network obtained by the model training method described in any of the above embodiments;

[0196] According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction of the target object to obtain a three-dimensional grid of the target surface of the target object.

[0197] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor when it is running, and the processor executes the steps of any of the above-mentioned model training methods or three-dimensional reconstruction methods.

[0198] An embodiment of the present application also provides a computer program product, which includes program instructions. When the program instructions are executed by a processor, the steps of any of the above-mentioned model training methods or three-dimensional reconstruction methods are executed.

[0199] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0200] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0202] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0204] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0205] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: include: Get multiple sampling points within the bounding box of the target object; Inputting the coordinates of the plurality of sampling points into a pre-trained surface distance extraction network to obtain first signed distance parameters of the plurality of sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points; Determining, according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point, a hollowing mask parameter of the observation ray corresponding to the sampling point, wherein the hollowing mask parameter of the observation ray is used to indicate whether the sampling point is in a hollowed portion on the observation ray; determining a depth value of a vertex on the surface of the observation ray according to a first signed distance parameter of the sampling point; Calculating a model loss value according to a hollowing mask parameter of the observation ray and a depth value of a vertex on the surface of the observation ray; According to the model loss value, the parameters of the pre-trained surface distance extraction network are adjusted to obtain a target surface distance extraction network.

2. The method according to claim 1, characterized in that The determining, according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point, of the hollowing mask parameters corresponding to the observation ray of the sampling point includes: Using a pre-trained color decomposition network, a plurality of color components of the observation ray are obtained according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point; the plurality of color components include at least diffuse reflection color; Determine a hollowing mask parameter of the observation ray according to the diffuse reflection color.

3. The method according to claim 1, characterized in that The determining, according to the first signed distance parameter of the sampling point, the depth value of the surface vertex on the observation ray comprises: Obtaining a three-dimensional surface mesh of the target object using a preset surface extraction algorithm according to the first signed distance parameter of the sampling points; Depth rendering is performed on the three-dimensional surface mesh to obtain depth values of surface vertices on the observation ray.

4. The method according to claim 2, characterized in that The step of determining the hollowing mask parameter of the observation ray according to the diffuse reflection color includes: If the diffuse reflection color is black, determining that the hollowing mask parameter of the observation ray is a first mask parameter, where the first mask parameter is used to indicate that the sampling point on the observation ray is a hollowed portion; If the diffuse reflection color is not black, the hollow mask parameter of the observation ray is determined to be a second mask parameter; the second mask parameter is used to indicate that the sampling point on the observation ray is a non-hollow portion.

5. The method according to claim 1, characterized in that The calculating the model loss value according to the hollowing mask parameter of the observation ray and the depth value of the vertex on the surface of the observation ray includes: Obtaining initial volume rendering weights for each sampling point on the observation ray; determining a hollowing ray from a plurality of the observation rays according to a hollowing mask parameter of the observation ray; Determining a first loss value based on the depth value of the surface vertex on the hollowed-out ray and the initial volume rendering weight of each sampling point on the hollowed-out ray using a preset volume rendering weight constraint condition; wherein the preset volume rendering weight constraint condition is: the sum of the initial volume rendering weights of each sampling point on the hollowed-out ray whose depth value is less than the depth value of the surface vertex is 0; Calculate the model loss value based on the first loss value.

6. The method according to claim 5, characterized in that Before calculating the model loss value according to the first loss value, the method further includes: Performing volume rendering according to the initial volume rendering weight on the observation ray to obtain a rendering color of the observation ray; Calculating a second loss value according to the rendered color of the observation ray and a true color corresponding to the observation ray; wherein the true color of the observation ray is the color of the pixel position corresponding to the observation ray in the two-dimensional image of the target object; Calculating the model loss value according to the first loss value includes: Calculate the model loss value based on the first loss value and the second loss value.

7. The method according to claim 2, characterized in that The method further comprises: According to the model loss value, the parameters of the pre-trained color decomposition network are adjusted to obtain a target color decomposition network.

8. The method according to claim 2, characterized in that The method of using a pre-trained color decomposition network to obtain multiple color components of the observation ray according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric feature vector of the sampling point includes: The pre-trained color decomposition network is used to obtain multiple color components of the observation ray based on the coordinates of the sampling point, the normal vector of the sampling point, the geometric feature vector of the sampling point, the direction vector of the observation ray, and the reflection vector of the observation ray, wherein the reflection vector is used to represent: the reflection direction of the observation ray relative to the normal vector of the sampling point, and the multiple color components also include: specular reflection color and specular reflection coefficient.

9. The method according to claim 2, characterized in that The pre-trained surface distance extraction network and the pre-trained color decomposition network are networks trained in the following manner: Using an initial surface distance extraction network, performing parameter extraction based on the coordinates of the plurality of sampling points, to obtain second signed distance parameters of the plurality of sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points; Using an initial color decomposition network, a plurality of initial color components of the observation ray are obtained according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point; Performing volume rendering according to the second signed distance parameter of the sampling point to obtain a volume rendering weight of the sampling point; Obtaining an initial rendering color of the observation ray according to a plurality of initial color components of the observation ray and a volume rendering weight of the sampling point; According to the initial rendering color of the observation ray and the true color corresponding to the observation ray, parameters of the initial surface distance extraction network and the initial color decomposition network are adjusted to obtain the pre-trained surface distance extraction network and the pre-trained color decomposition network.

10. A three-dimensional reconstruction method, characterized in that: include: Get the sampling points within the bounding box of the target object; Inputting the coordinates of a plurality of sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the plurality of sampling points; wherein the surface distance extraction network is a target surface distance extraction network obtained by the model training method according to any one of claims 1 to 9; According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction on the target object to obtain a three-dimensional grid of the target surface of the target object.

11. A model training device, characterized in that: include: A first acquisition module is used to acquire multiple sampling points within a bounding box of the target object; a first extraction module, configured to input the coordinates of the plurality of sampling points into a pre-trained surface distance extraction network to obtain first signed distance parameters of the plurality of sampling points, normal vectors of the sampling points, and geometric feature vectors of the sampling points; a determination module, configured to determine a hollowing mask parameter of an observation ray corresponding to the sampling point according to the coordinates of the sampling point, the normal vector of the sampling point, and the geometric characteristic vector of the sampling point; determining a depth value of a vertex on the surface of the observation ray according to a first signed distance parameter of the sampling point, wherein a hollowing mask parameter of the observation ray is used to indicate whether the sampling point on the observation ray of the bounding box is in a hollowed portion; A calculation module, configured to calculate a model loss value based on a hollowing mask parameter of the observation ray and a depth value of a vertex on a surface of the observation ray; A training module is used to adjust the parameters of the pre-trained surface distance extraction network according to the model loss value to obtain a target surface distance extraction network.

12. A three-dimensional reconstruction device, characterized in that: include: A second acquisition module is used to obtain sampling points within the bounding box of the target object; a second extraction module, configured to input the coordinates of a plurality of sampling points within the bounding box of the target object into a pre-trained surface distance extraction network to obtain target symbol distance parameters of the plurality of sampling points; wherein the surface distance extraction network is a target surface distance extraction network obtained by the model training method according to any one of claims 1 to 9; The reconstruction module is used to perform three-dimensional reconstruction on the target object using a preset surface extraction algorithm according to the target symbol distance parameter to obtain a three-dimensional grid of the target surface of the target object.

13. A computer device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the model training method according to any one of claims 1 to 9 or the three-dimensional reconstruction method according to claim 10.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the model training method according to any one of claims 1 to 9 or the three-dimensional reconstruction method according to claim 10.

Images