Three-dimensional generation method, device, electronic device and storage medium

Abstract

The disclosed embodiments disclose a 3D generation method, apparatus, electronic device, and storage medium, wherein the method comprises: generating a multi-view image based on an image to be reconstructed; obtaining a signed distance value corresponding to each grid point in a preset 3D space based on a signed distance prediction network, wherein the grid points are vertices of a cube in the preset 3D space; determining a first 3D mesh representation corresponding to the image to be reconstructed based on the signed distance value corresponding to each grid point; and determining a textured target 3D mesh representation corresponding to the image to be reconstructed based on the first 3D mesh representation, using the multi-view image as a supervisory image. The disclosed embodiments can both express objects with complex topologies and avoid reconstruction artifacts caused by post-processing operations, thereby helping to improve the quality of 3D reconstruction.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology, and in particular to a three-dimensional generation method, device, electronic device and storage medium. Background Art

[0002] In single vision Figure 3 In the field of 3D reconstruction, diffusion models are often used to generate images from multiple perspectives. Sparse perspective reconstruction is then performed based on these images to obtain a 3D result. Related technologies primarily include explicit and implicit 3D reconstruction methods. However, explicit reconstruction methods struggle to reconstruct objects with complex topologies. Post-processing operations in implicit reconstruction methods are prone to producing reconstruction artifacts, resulting in low-quality 3D reconstruction results. Summary of the Invention

[0003] The embodiments of the present disclosure provide a three-dimensional generation method, device, electronic device, and storage medium, which can not only reconstruct objects with complex topologies but also improve the quality of the reconstructed objects.

[0004] One aspect of the embodiments of the present disclosure provides a three-dimensional generation method, comprising:

[0005] Generate multi-view images based on the image to be reconstructed;

[0006] Based on a signed distance prediction network, a signed distance value corresponding to each grid point in a preset three-dimensional space is obtained; the grid points are vertices of a cube in the preset three-dimensional space;

[0007] Determining a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points;

[0008] The multi-view image is used as a supervisory image, and based on the first three-dimensional grid representation, a textured target three-dimensional grid representation corresponding to the image to be reconstructed is determined.

[0009] Another aspect of the present disclosure provides a three-dimensional generation device, including:

[0010] A first processing module, configured to generate a multi-view image based on the image to be reconstructed;

[0011] The second processing module is configured to predict the signed distance values corresponding to the respective grid points in a preset three-dimensional space based on a signed distance prediction network; the grid points are vertices of a cube in the preset three-dimensional space.

[0012] a third processing module, configured to determine a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points;

[0013] The fourth processing module is configured to use the multi-view image as a supervisory image and determine a textured target three-dimensional grid representation corresponding to the image to be reconstructed based on the first three-dimensional grid representation.

[0014] In another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the three-dimensional generation method described in any of the above embodiments of the present disclosure is implemented.

[0015] Another aspect of the present disclosure provides an electronic device, comprising:

[0016] Memory for storing computer programs;

[0017] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the three-dimensional generation method described in any of the above embodiments of the present disclosure is implemented.

[0018] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions, when executed by a processor, implement the three-dimensional generation method described in any of the above embodiments of the present disclosure.

[0019] The three-dimensional generation method, device, electronic device and storage medium provided by the present disclosure can generate a multi-view image based on the image to be reconstructed; then, based on the signed distance prediction network, the signed distance value corresponding to each grid point in the preset three-dimensional space can be obtained; then, based on the signed distance value corresponding to each grid point, a first three-dimensional grid representation corresponding to the image to be reconstructed can be determined; then, the multi-view image can be used as a supervision image, and based on the first three-dimensional grid representation, a textured target three-dimensional grid representation corresponding to the image to be reconstructed can be determined. The method of the embodiment of the present disclosure introduces an implicit expression of SDF (Signed Distance Function) by predicting the signed distance value (i.e., SDF value), thereby being able to express objects with complex topology, and then determining the first three-dimensional grid representation (i.e., mesh) corresponding to the image to be reconstructed based on the signed distance value corresponding to each grid point, obtaining an explicit grid expression, which can avoid reconstruction artifacts caused by post-processing operations and help improve the quality of three-dimensional reconstruction. The multi-view image can be used as a supervision image, and the textured target three-dimensional grid representation corresponding to the image to be reconstructed can be obtained by rendering according to the display grid expression. The rendering speed is fast, so that the textured target three-dimensional grid representation can be quickly obtained.

[0020] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0023] Figure 1 is a flowchart of a three-dimensional generation method provided by an exemplary embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of a three-dimensional generation method provided by another exemplary embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of a three-dimensional generation method provided by another exemplary embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of a three-dimensional generation method provided by another exemplary embodiment of the present disclosure;

[0027] Figure 5 It is a schematic diagram of the overall framework of a three-dimensional generation algorithm provided by an exemplary embodiment of the present disclosure;

[0028] Figure 6 is a schematic structural diagram of a three-dimensional generation device provided by an exemplary embodiment of the present disclosure;

[0029] Figure 7 is a schematic structural diagram of a three-dimensional generation device provided by another exemplary embodiment of the present disclosure;

[0030] Figure 8 It is a structural diagram of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0031] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0032] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0033] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0034] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0035] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0036] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0040] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0042] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0043] Overview of the Disclosure

[0044] In the process of realizing the present disclosure, the inventors found that in the single-view Figure 3 In the field of three-dimensional generation, diffusion models are usually used to generate images from multiple perspectives, and then sparse perspective reconstruction is performed based on the images from multiple perspectives to obtain three-dimensional generation results. In related technologies, three-dimensional reconstruction is mainly achieved through two methods, one is the explicit reconstruction method, and the other is the implicit reconstruction method. Among them, the explicit reconstruction method is to perform explicit operations such as vertex movement and edge splitting on the initialized simple topology to gradually approach the target topology. However, these explicit operations are difficult to perform in a small range, so it is difficult for the explicit reconstruction method to reconstruct complex topological objects with rich surface details. The implicit reconstruction method obtains an implicit expression of the object based on a trained deep neural network, such as NeRF (Neural Radiance Field). NeRF implicitly represents the 3D object by learning the mapping relationship from input (the coordinates of the sampling points in space) to output (such as surface color or density), and then extracts the three-dimensional object surface, that is, the three-dimensional mesh, from the implicit expression through post-processing. Although this implicit reconstruction method can perform three-dimensional reconstruction of objects with complex geometric structures, it requires first obtaining an implicit textured representation of the object and then obtaining an explicit representation of the object through post-processing operations. This post-processing operation is prone to reconstruction artifacts, resulting in low-quality three-dimensional reconstruction results. In addition, training a deep neural network from the coordinates of sampling points in the input space to the output 3D object surface color or density requires a lot of time and resources, resulting in a long 3D generation time.

[0045] Exemplary Methods

[0046] Figure 1 FIG. 1 is a flow chart of a three-dimensional generation method provided by an exemplary embodiment of the present disclosure. The method may include the following steps:

[0047] Step 202: Generate multi-view images based on the image to be reconstructed.

[0048] In some optional embodiments, the image to be reconstructed may be one or more single-view images, and the image to be reconstructed may include an object to be reconstructed.

[0049] In some optional embodiments, the image to be reconstructed can be diffused based on any trained multi-view diffusion model to generate a multi-view image. The number of multi-view images can be a preset number, which can be set according to actual needs, for example, 3, 4, 5, 6, etc. The multi-view image can include color images (i.e., RGB images) from multiple perspectives and a normal map corresponding to each color image.

[0050] Step 204 : Based on the signed distance prediction network, obtain the signed distance value corresponding to each grid point in the preset three-dimensional space.

[0051] The grid points are all vertices contained in the preset three-dimensional space. The signed distance value can represent the distance of a point relative to the surface. If the point is inside the surface, the distance is negative; if the point is outside the surface, the distance is positive; if the point is on the surface, the distance is zero.

[0052] In some optional embodiments, the signed distance prediction network can be implemented using any feasible neural network. Preferably, the signed distance prediction network can be implemented based on a coding network and a multi-layer perceptron. The signed distance prediction network can be used to predict the signed distance field (SDF) of the object to be reconstructed in a preset three-dimensional space, i.e., the signed distance values (or SDF values) corresponding to each grid point, thereby obtaining an implicit signed distance field representation of the object to be reconstructed. The signed distance values represent the distance between the grid points and the surface of the object to be reconstructed.

[0053] In some optional embodiments, the preset three-dimensional space may be a cubic space with a preset side length. The preset three-dimensional space may be divided into a preset number of small cubes. The grid points are the vertices of the small cubes. For example, the preset three-dimensional space may be a cubic space with a side length of 2. A three-dimensional space coordinate system may be established with the center point of the preset three-dimensional space as the origin. The coordinates of the cube vertices range between [-1, 1]. The position of each grid point in the three-dimensional space coordinate system may be determined. For example, the position X of any grid point i may be i It can be expressed as (x i ,y i ,z i ).

[0054] Step 206 : Determine a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points.

[0055] The first three-dimensional grid representation (the three-dimensional grid representation may also be referred to as mesh representation) may refer to a three-dimensional grid that describes the surface of the object in the image to be reconstructed.

[0056] In some optional embodiments, based on the signed distance values corresponding to the grid points, a surface extraction algorithm with a signed distance value of 0 may be used to extract the first three-dimensional grid representation corresponding to the image to be reconstructed.

[0057] Step 208 : Using the multi-view images as supervisory images, and based on the first three-dimensional mesh representation, determine a textured target three-dimensional mesh representation corresponding to the image to be reconstructed.

[0058] In some optional embodiments, a multi-view image is used as a supervisory image to supervise the iterative optimization of the 3D reconstruction algorithm (or simply the algorithm) of the disclosed embodiment. After the iterative optimization is completed, a textured target 3D mesh representation corresponding to the image to be reconstructed is determined based on the first 3D mesh representation obtained by the optimized 3D reconstruction algorithm. The processing of the 3D reconstruction algorithm includes steps 202 to 208 described above, wherein steps 204 to 208 are iteratively executed based on the updated parameters under the supervision of the multi-view image, and then the first 3D mesh representation is obtained according to the above steps based on the iteratively optimized algorithm, and the textured target 3D mesh representation is rendered based on the first 3D mesh representation.

[0059] The three-dimensional generation method provided by the embodiment of the present disclosure can express objects with complex topologies by introducing the implicit expression of SDF (Signed Distance Function), and can determine the explicit expression of the first three-dimensional grid representation (i.e., mesh) corresponding to the image to be reconstructed based on the signed distance values corresponding to each grid point. The textured target three-dimensional grid representation corresponding to the image to be reconstructed is directly determined based on the first three-dimensional grid representation. This can avoid the reconstruction artifact problem caused by the explicit expression of the object obtained from the implicit expression of the texture, thereby helping to improve the quality of three-dimensional reconstruction. In addition, using multi-view images as supervision images, rendering is performed according to the display grid expression to obtain the textured target three-dimensional grid representation corresponding to the image to be reconstructed. That is, rendering according to the display grid expression has a fast rendering speed, so that the textured target three-dimensional grid representation can be quickly obtained.

[0060] In some optional embodiments, Figure 2 FIG. 1 is a flow chart of a three-dimensional generation method provided by another exemplary embodiment of the present disclosure. Figure 2 As shown, based on any of the above three-dimensional generation method embodiments, predicting the signed distance values corresponding to each grid point in the preset three-dimensional space based on the signed distance prediction network may include the following steps:

[0061] Step 302 : Perform multi-resolution hash coding on each grid point in the preset three-dimensional space to obtain multi-resolution coding features.

[0062] Among them, multi-resolution hash coding can also be called multi-resolution hash grid position coding. The coding principle is: for a given grid position X∈R 3 , R 3 Representing a three-dimensional space, X is mapped to corresponding positions at multiple hash resolutions. This allows X to be mapped from a three-dimensional space to a multi-dimensional space to obtain multi-resolution encoded features. Specifically, the encoded features for each hash resolution can be obtained through trilinear interpolation. The encoded features of all resolutions are then fused together to obtain a multi-resolution encoded feature, which is used as a position code to predict the signed distance value corresponding to the grid point.

[0063] Step 304: predict the symbol distance value corresponding to each grid point based on the multi-resolution coding feature.

[0064] In some optional embodiments, the multi-resolution coding features can be predicted by a prediction network to obtain the signed distance values corresponding to each grid point. The prediction network can be, for example, a multi-layer perceptron (MLP) or other networks.

[0065] This embodiment encodes the grid point positions through multi-resolution hash coding, mapping the grid point positions from three-dimensional space to n-dimensional space, thereby improving the coding capability. It can be combined with a lighter-weight MLP, thereby accelerating the convergence of the signed distance prediction network and capturing more geometric details, thereby improving the perspective generalization capability.

[0066] In some optional embodiments, Figure 3 FIG. 1 is a flow chart of a three-dimensional generation method provided by another exemplary embodiment of the present disclosure. Figure 3 As shown, based on any of the above three-dimensional generation method embodiments, determining a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to each grid point may include the following steps:

[0067] Step 402 : Based on the signed distance values corresponding to the grid points, a point set whose edge signed distances between any two adjacent grid points are preset values is calculated.

[0068] In some optional embodiments, since the grid points are vertices of small cubes in a preset three-dimensional space, and the edges of the small cubes connect two adjacent grid points, a set of points on each edge of the small cube whose signed distance is a preset value can be calculated based on the signed distance values corresponding to each grid point. The preset value can be 0, that is, the set of points on each edge of the small cube in the preset three-dimensional space whose SDF value is 0 is calculated.

[0069] In some optional embodiments, the point set with an SDF value of 0 on each edge of the small cube may be calculated based on the following formula:

[0070]

[0071] Among them, u e Represents the adjacent grid point X i and X j The position of the point on the edge where the signed distance value is 0. s(X i ) represents the grid point X i The signed distance value, s(X j ) represents the grid point X j The signed distance value, s(X i ) and s(X j ) have opposite signs. i and α j Represents the grid point X i and grid point X j The corresponding learnable scaling factor parameter. i and α j Continuously learn and update during the 3D reconstruction algorithm optimization process.

[0072] Step 404 : Determine a first three-dimensional mesh vertex set based on a point set whose signed distance is a preset value.

[0073] In some optional embodiments, based on a point set with a preset signed distance, a three-dimensional mesh vertex calculation formula can be used to determine three-dimensional mesh vertices (or mesh vertices), and a first three-dimensional mesh vertex set can be obtained based on the calculated three-dimensional mesh vertices.

[0074] In some optional embodiments, the points on the edge of the small cube with a signed distance value of 0 can form dual faces of different shapes, and the mesh vertices can be calculated and extracted based on the shapes of the dual faces. For example, the mesh vertices can be obtained by the following calculation method:

[0075]

[0076] Among them, v d Indicates the coordinates of the extracted mesh vertices. E Represents the edge set of dual face. e is a learnable algorithm parameter. The parameter α corresponding to each grid point i and parameter β e By connecting each grid point and each edge separately, it can be used to flexibly locate the position with a signed distance value of 0 and the mesh vertex position, thereby achieving the effect of flexible extraction of the three-dimensional mesh surface.

[0077] Step 406: Determine a first three-dimensional mesh representation based on the first three-dimensional mesh vertex set.

[0078] After obtaining the first 3D mesh vertex set, a first 3D mesh representation can be obtained based on the first 3D mesh vertex set. For example, connections can be established between the 3D mesh vertices based on the positional relationships between the 3D mesh vertices in the first 3D mesh vertex set to form a 3D mesh and obtain the first 3D mesh representation.

[0079] This embodiment combines explicit three-dimensional mesh representation with implicit signed distance fields, which not only enables end-to-end reconstruction of the three-dimensional mesh, but also avoids problems such as reconstruction artifacts and slow convergence caused by post-processing operations. It also achieves the efficient encoding capability of implicit representation for complex geometric surfaces, thereby improving the three-dimensional reconstruction quality of complex topologies.

[0080] In some optional embodiments, Figure 4 FIG. 1 is a flow chart of a three-dimensional generation method provided by another exemplary embodiment of the present disclosure. Figure 4 As shown, based on any of the above-mentioned 3D generation method embodiments, the supervision image can include a supervision normal map and a supervision color image corresponding to each viewpoint. The supervision normal map and the supervision color image can be generated by diffusion based on the image to be reconstructed using a multi-view diffusion model. The supervision color image can be an RGB image.

[0081] Using the multi-view images as supervisory images and based on the first three-dimensional mesh representation, determining a textured target three-dimensional mesh representation corresponding to the image to be reconstructed may include the following steps:

[0082] Step 502 : Based on the vertices represented by the first 3D mesh, their corresponding vertex normal vectors and observation viewing angle parameters, determine the predicted color image corresponding to the target viewing angle in each viewing angle and the color corresponding to each 3D mesh vertex.

[0083] The observation perspective parameter includes the viewing direction corresponding to each perspective of the multi-view image. During each iteration, a target perspective can be randomly determined from each perspective. The predicted color image can be a predicted RGB image.

[0084] In some optional embodiments, in order to reduce the amount of computation, a perspective may be randomly selected as a target perspective in each iteration for updating algorithm parameters.

[0085] In some optional embodiments, for any 3D mesh vertex, a vertex normal vector corresponding to the 3D mesh vertex can be calculated based on the normal vector of the mesh surface on which the 3D mesh vertex resides. For example, the normal vectors of all mesh surfaces on which the 3D mesh vertex resides can be weighted averaged, and the average value can be used as the vertex normal vector corresponding to the 3D mesh vertex.

[0086] In some optional embodiments, the viewing direction corresponding to each viewing angle may be expressed as a direction vector of the viewing direction corresponding to each viewing angle.

[0087] In some optional embodiments, the predicted normal maps corresponding to each viewpoint can be obtained by rasterizing the first 3D mesh representation. The color image corresponding to the target viewpoint and the colors corresponding to each 3D mesh vertex can be predicted using a learnable rendering algorithm. The rendering algorithm can be, for example, a texture prediction network (or color prediction network).

[0088] Step 504: supervise the predicted normal map and predicted color image of the target perspective based on the supervision image. When the supervision result meets the preset conditions, determine the textured target three-dimensional mesh representation corresponding to the image to be reconstructed based on the first three-dimensional mesh representation and the colors corresponding to each three-dimensional mesh vertex.

[0089] Among them, supervising the predicted normal map and predicted color image of the target perspective based on the supervisory image can refer to guiding the update of the parameters of the 3D reconstruction algorithm based on the error between the predicted normal map and the predicted color image and the supervisory image until the supervision result meets the preset conditions. Then, based on the first 3D mesh representation obtained by the optimized parameters and the colors corresponding to each 3D mesh vertex, a textured target 3D mesh representation corresponding to the image to be reconstructed can be obtained. The preset conditions can be set according to specific needs. For example, the preset conditions may include at least one of the number of iterations reaching a preset number threshold, the algorithm convergence, etc.

[0090] This embodiment can predict the color of each 3D mesh vertex through the first 3D mesh representation, the vertex normal vectors of each 3D mesh vertex and the viewing direction corresponding to each viewing angle, and can extract the predicted color image of the target viewing angle. The predicted color image and the predicted normal map are combined with the supervision image to calculate the loss, and guide the update of the parameters until the supervision result meets the preset conditions. When the supervision result meets the preset conditions, the obtained first 3D mesh representation and the predicted 3D mesh vertex color have reached the optimal level. The color of the 3D mesh vertex can represent the texture of the 3D mesh representation. Therefore, a textured target 3D mesh representation can be obtained based on the first 3D mesh representation and the color of the 3D mesh vertex. Since the predicted vertex color is under the supervision of the supervision image, compared with the volume rendering of the post-processing operation in implicit reconstruction, the disclosed embodiment can effectively avoid complex post-processing operations, thereby improving the rendering speed, reducing the computational complexity, and improving the algorithm convergence speed.

[0091] In some optional embodiments, based on any of the above three-dimensional generation method embodiments, determining a predicted normal map corresponding to each viewing angle, a predicted color image corresponding to a target viewing angle in each viewing angle, and a color corresponding to each three-dimensional mesh vertex based on the first three-dimensional mesh representation, vertex normal vectors corresponding to each three-dimensional mesh vertex of the first three-dimensional mesh representation, and viewing angle parameters may include:

[0092] Based on the first three-dimensional grid representation and the viewing directions corresponding to each viewing angle, the predicted normal maps corresponding to each viewing angle are extracted; based on each three-dimensional grid vertex represented by the first three-dimensional grid, the predicted normal maps corresponding to each viewing angle and the viewing directions, the texture prediction network is used to predict the color corresponding to each three-dimensional grid vertex; based on the color corresponding to each three-dimensional grid vertex, the predicted color image corresponding to the target viewing angle is determined.

[0093] In some optional embodiments, the viewing direction can be used to determine all three-dimensional mesh vertices and mesh faces represented by the first three-dimensional mesh that can be seen from the corresponding viewing angle, and then based on the three-dimensional mesh vertices and mesh faces that can be seen from each viewing angle, the normal map corresponding to each viewing angle can be extracted as the predicted normal map for each viewing angle.

[0094] In some optional embodiments, the texture prediction network can be implemented using any feasible neural network, for example, a deep learning neural network.

[0095] In some optional embodiments, the texture prediction network may include a position encoding network and a prediction network. For example, the position encoding network may be implemented using a network structure based on hash coding, multi-resolution hash coding, or other position coding. The prediction network may be implemented using a multi-layer perceptron (MLP) or other possible network structures.

[0096] In some optional embodiments, during the iterative optimization process, a texture prediction network can be used to directly obtain a predicted color image for the target viewpoint based on each 3D mesh vertex in the first 3D mesh representation, the predicted normal map for each viewpoint, and the viewing direction of the target viewpoint. This color image is then used to update the parameters. After the optimization is complete, the color of each 3D mesh vertex is obtained using the first 3D mesh representation, the predicted normal map for each viewpoint, and the viewing direction of each viewpoint obtained using the optimized parameters.

[0097] This embodiment predicts the color of the 3D mesh vertices through the texture prediction network, performs back propagation through supervised image supervision, guides the training of the signed distance prediction network and the texture prediction network, improves the network prediction effect, and thus obtains a high-quality textured target 3D mesh representation.

[0098] In some optional embodiments, based on any of the above three-dimensional generation method embodiments, predicting the color corresponding to each three-dimensional mesh vertex based on each three-dimensional mesh vertex represented by the first three-dimensional mesh and the predicted normal map and viewing direction corresponding to each viewing angle using a texture prediction network may include:

[0099] Each 3D mesh vertex is hashed to obtain the encoding feature; based on the predicted normal map and viewing direction corresponding to each 3D mesh vertex and each viewing angle, a multi-layer perceptron is used to predict the color corresponding to each 3D mesh vertex.

[0100] In some optional embodiments, the hash coding may be single-resolution hash coding or multi-resolution hash coding. The principle of multi-resolution hash coding may be described in the foregoing content.

[0101] This embodiment implements a lightweight texture prediction network based on hash coding and a multi-layer perceptron, which helps to accelerate network convergence.

[0102] In some optional embodiments, based on any of the above three-dimensional generation method embodiments, supervising the predicted normal map and the predicted color image of the target perspective based on the supervision image may include:

[0103] Based on the predicted color image corresponding to the target perspective, a predicted mask map corresponding to the target perspective is generated; based on the supervised color image corresponding to the target perspective, a supervised mask map corresponding to the target perspective is generated; based on the supervised color image, supervised normal map and supervised mask map corresponding to the target perspective, the predicted color image, predicted normal map and predicted mask map corresponding to the target perspective are supervised.

[0104] The predicted mask image may be a binary mask image including two pixel values, 0 and 1. 0 may indicate that the pixel belongs to the background of the image, and 1 may indicate that the pixel belongs to the object in the image. The supervised mask image may be obtained by performing object recognition on the supervised color image.

[0105] In some optional embodiments, the mean square error can be calculated based on the supervised color image, supervised normal map, supervised mask map corresponding to the target perspective, and the predicted color image, predicted normal map and predicted mask map corresponding to the target perspective to guide the update of parameters.

[0106] This embodiment combines RGB image, normal map and mask Figure 1 It guides the iterative optimization of the algorithm, which helps to further improve the reconstruction effect of the algorithm.

[0107] In some optional embodiments, based on any of the above three-dimensional generation method embodiments, supervision of the predicted color image, predicted normal map, and predicted mask map corresponding to the target perspective based on the supervised color image, supervised normal map, and supervised mask map corresponding to the target perspective may include:

[0108] Based on the predicted color image and the supervised color image corresponding to the target perspective, a first loss is determined; based on the predicted normal map and the supervised normal map corresponding to the target perspective, a second loss is determined; based on the predicted mask map and the supervised mask map corresponding to the target perspective, a third loss is determined; based on the first loss, the second loss and the third loss, a comprehensive loss is determined; based on the comprehensive loss, a supervision result is determined; when the supervision result does not meet the preset conditions, the parameters of the 3D reconstruction algorithm are updated based on the comprehensive loss, and based on the updated parameters, the steps of predicting the signed distance values corresponding to each grid point in the preset 3D space based on the signed distance prediction network and subsequent steps are repeatedly executed to achieve iterative update of the parameters until the supervision result meets the preset conditions, and an optimized 3D reconstruction algorithm is obtained, and then the color of each 3D mesh vertex is predicted based on the optimized 3D reconstruction algorithm, thereby obtaining a textured target 3D mesh representation.

[0109] The 3D reconstruction algorithm can include the aforementioned multi-view diffusion model, signed distance prediction network, mesh generation algorithm, rasterization operations, and texture prediction network. The multi-view diffusion model is a pre-trained model. During the iteration process, the network parameters of the multi-view diffusion model are treated as hyperparameters and are not updated. Therefore, based on the 3D reconstruction algorithm, end-to-end 3D reconstruction can be achieved without complex post-processing.

[0110] In some optional embodiments, the first loss, second loss, and third loss can be calculated using any applicable loss function. For example, the loss function may include a mean square error (MSE) loss function (also known as an L2 loss function), a mean absolute error (MAE) loss function (also known as an L1 loss function), and the like.

[0111] In some optional embodiments, the first loss, the second loss, and the third loss may be weighted to obtain a comprehensive loss, wherein the weight of each loss may be set according to actual conditions.

[0112] In some optional embodiments, based on any of the above-mentioned three-dimensional generation method embodiments, determining a comprehensive loss based on the first loss, the second loss, and the third loss may include: calculating at least one of the following losses: calculating a signed distance gradient of two adjacent grid points based on the signed distance values corresponding to each grid point; determining a fourth loss based on the signed distance gradient and a gradient threshold of two adjacent grid points; calculating a Laplace smoothness value based on the first three-dimensional grid representation as a fifth loss; calculating the negative cosine similarity of normal vectors of two adjacent surfaces in the first three-dimensional grid representation; and determining a sixth loss based on the negative cosine similarity and a similarity threshold of the normal vectors of two adjacent surfaces. The comprehensive loss is determined based on the first loss, the second loss, the third loss, and at least one of the fourth loss, the fifth loss, and the sixth loss.

[0113] Among them, the signed distance gradient of any adjacent grid points can be expressed as the absolute value of the ratio of the difference between the signed distance values of the adjacent grid points and the distance between the two adjacent grid points. The gradient threshold can be set according to actual needs. For example, the gradient threshold can be set to 1. The fourth loss can be expressed as the sum of the errors between each signed distance gradient and the gradient threshold (the absolute value of the difference between the signed distance gradient and the gradient threshold). The fourth loss can guide the learning of a reasonable signed distance value that meets the constraints of the eikonal equation. That is, the SDF gradient norm of any position in the preset three-dimensional space is 1. The Laplace smoothness value, as the fifth loss, can be used to smooth the extracted mesh surface so that the distance from each three-dimensional mesh vertex to the average position of the adjacent vertex is minimized, thereby avoiding rough surfaces as a whole, which helps to further improve the quality of three-dimensional reconstruction. For any two adjacent surfaces, the cosine similarity of the normal vectors of the two surfaces represents the angle between the two surfaces. The cosine similarity ranges from -1 to 1. If the angle between the two surfaces is 0, the cosine similarity is 1. If the angle between the two surfaces is 90 degrees, the cosine similarity is 0. If the angle between the two surfaces is 180 degrees, the cosine similarity is -1. Negative cosine similarity refers to the negative value of cosine similarity. The similarity threshold can be set to 1. The error between the negative cosine similarity and the similarity threshold is used as the sixth loss to participate in the parameter update, which can minimize the negative cosine similarity, thereby further smoothing the reconstructed 3D mesh representation and further improving the 3D reconstruction quality and authenticity.

[0114] In some optional embodiments, any one, two, or three of the fourth, fifth, and sixth losses can be weighted averaged with the first, second, and third losses to obtain a comprehensive loss. The weights of the various losses can be set according to actual needs. Preferably, the first, second, third, fourth, fifth, and sixth losses can be weighted averaged (or added together) to obtain a comprehensive loss.

[0115] In some optional embodiments, Figure 5 FIG. 1 is a schematic diagram of the overall framework of a three-dimensional reconstruction algorithm provided by an exemplary embodiment of the present disclosure. Figure 5 As shown in FIG, a multi-view diffusion model can be used to generate a multi-view image corresponding to the image to be reconstructed. The multi-view image can include RGB images of a preset number of viewpoints and normal maps corresponding to each RGB image. The RGB image and normal map generated by diffusion are used as supervision images. A mask map can also be generated based on the RGB image. The RGB image, normal map and mask map are the same. Figure 1It serves as a supervisory image. The preset three-dimensional space takes a 96*96*96 three-dimensional space composed of cubes with vertex coordinates ranging from [-1,1] as an example, that is, the preset three-dimensional space can include 96*96*96 small cubes. A three-dimensional space coordinate system is established with the center point of the preset three-dimensional space as the origin, so that the coordinate components of each grid point in the preset three-dimensional space in three directions are all in the range of -1 to 1. The signed distance prediction network includes hash encoding and MLP. The signed distance prediction network can predict the SDF value corresponding to each grid point in the preset three-dimensional space. The mesh generation algorithm can extract mesh vertices and faces to generate a first three-dimensional mesh representation. Specifically, the points on the edge of the cube with an SDF value of 0 can be calculated based on the predicted SDF value, and then the mesh vertices can be calculated based on the points with an SDF value of 0, and the first three-dimensional mesh representation can be obtained based on the obtained mesh vertices. Then, the normal map of each perspective can be extracted from the first three-dimensional mesh representation through rasterization operation. The 3D mesh vertices (i.e., mesh vertex sets), the normal maps for each view (i.e., the predicted normal maps described above), and the view direction of the first 3D mesh representation are input into a texture prediction network. The texture prediction network includes hash encoding and MLP. The texture prediction network can predict the color of each mesh vertex. During this process, a view can be randomly selected from each view as a target view, and a color image of the target view can be extracted as a predicted color image. The mean squared error (the combined loss of the first, second, and third losses) is calculated based on the predicted color image of the target view, the predicted normal map, and the supervision image obtained by diffusion. The fourth, fifth, and sixth losses described above can also be calculated. The parameters of each part of the algorithm are updated through backpropagation. Based on the updated parameters, the above process is iterated again according to the above process until the iteration ends, resulting in optimized parameters. The target 3D mesh representation with texture is predicted based on the optimized parameters. In the method of the disclosed embodiment, during the iterative optimization phase, the mesh surface extracted during the iterative process is rendered using a rasterization operation. The loss is calculated between the rendered image and the image generated by the diffusion model. The parameters in the implicit signed distance prediction network are optimized through backpropagation to generate more realistic 3D geometry. This rasterized display rendering process takes significantly less time than the volume rendering process in the neural radiation field. Therefore, the optimization time for each iteration is greatly shortened, which can significantly reduce the time it takes to generate the target 3D mesh representation. For example, the method of the disclosed embodiment can directly obtain a textured target 3D mesh representation corresponding to the image to be reconstructed in about one minute.

[0116] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0117] Any of the 3D generation methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any of the 3D generation methods provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any of the 3D generation methods mentioned in the embodiments of the present disclosure. This will not be further described below.

[0118] Exemplary devices

[0119] Figure 6 Schematic diagram of the structure of a three-dimensional generation device provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiments of the present disclosure, such as Figure 6 The apparatus shown may include a first processing module 602 , a second processing module 604 , a third processing module 606 , and a fourth processing module 608 .

[0120] The first processing module 602 may be configured to generate a multi-view image based on the image to be reconstructed.

[0121] The second processing module 604 may be configured to obtain signed distance values corresponding to respective grid points in a preset three-dimensional space based on a signed distance prediction network; the grid points are vertices of a cube in the preset three-dimensional space.

[0122] The third processing module 606 may be configured to determine a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points.

[0123] The fourth processing module 608 may be configured to use the multi-view images as supervisory images and determine a textured target 3D mesh representation corresponding to the image to be reconstructed based on the first 3D mesh representation.

[0124] In some optional embodiments, based on any of the above-described three-dimensional generation device embodiments, the second processing module 604 may be specifically configured to: perform multi-resolution hash coding on each grid point in a predetermined three-dimensional space to obtain a multi-resolution coding feature; and predict a signed distance value corresponding to each grid point based on the multi-resolution coding feature.

[0125] In some optional embodiments, based on any of the above-described 3D generation device embodiments, the third processing module 606 may be specifically configured to: calculate, based on the signed distance values corresponding to each grid point, a set of points where the edge signed distances between any two adjacent grid points are a preset value; determine a first 3D mesh vertex set based on the set of points where the signed distances are the preset value; and determine a first 3D mesh representation based on the first 3D mesh vertex set.

[0126] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the supervision image may include a supervision normal map and a supervision color image corresponding to each perspective; the fourth processing module 608 may be specifically used to: determine the predicted normal map corresponding to each perspective, the predicted color image corresponding to the target perspective in each perspective, and the color corresponding to each three-dimensional mesh vertex based on the first three-dimensional mesh representation, the vertex normal vectors corresponding to each three-dimensional mesh vertex of the first three-dimensional mesh representation, and the observation perspective parameters. The observation perspective parameters may include the viewing direction corresponding to each perspective of the multi-perspective image. The predicted normal map and predicted color image of the target perspective are supervised based on the supervision image, and when the supervision result meets the preset conditions, the textured target three-dimensional mesh representation corresponding to the image to be reconstructed is determined based on the first three-dimensional mesh representation and the colors corresponding to each three-dimensional mesh vertex.

[0127] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the fourth processing module 608 can be specifically used to: extract the predicted normal map corresponding to each perspective based on the first three-dimensional grid representation and the viewing direction corresponding to each perspective; predict the color corresponding to each three-dimensional grid vertex based on each three-dimensional grid vertex represented by the first three-dimensional grid, the predicted normal map corresponding to each perspective, and the viewing direction, using a texture prediction network; and determine the predicted color image corresponding to the target perspective based on the color corresponding to each three-dimensional grid vertex.

[0128] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the fourth processing module 608 can be specifically used to: hash encode each three-dimensional mesh vertex to obtain encoding features; based on the predicted normal map and viewing direction corresponding to each three-dimensional mesh vertex and each viewing angle, use a multi-layer perceptron to predict the color corresponding to each three-dimensional mesh vertex.

[0129] In some optional embodiments, based on any of the above three-dimensional generation device embodiments, the fourth processing module 608 may be specifically configured to:

[0130] Based on the predicted color image corresponding to the target perspective, a predicted mask map corresponding to the target perspective is generated; based on the supervised color image corresponding to the target perspective, a supervised mask map corresponding to the target perspective is generated; based on the supervised color image, supervised normal map and supervised mask map corresponding to the target perspective, the predicted color image, predicted normal map and predicted mask map corresponding to the target perspective are supervised.

[0131] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the fourth processing module 608 can be specifically used to: determine a first loss based on the predicted color image and the supervised color image corresponding to the target perspective; determine a second loss based on the predicted normal map and the supervised normal map corresponding to the target perspective; determine a third loss based on the predicted mask map and the supervised mask map corresponding to the target perspective; determine a comprehensive loss based on the first loss, the second loss and the third loss; determine a supervision result based on the comprehensive loss; if the supervision result does not meet the preset conditions, update the parameters of the three-dimensional reconstruction algorithm based on the comprehensive loss, and repeat the steps of predicting the signed distance values corresponding to each grid point in the preset three-dimensional space based on the signed distance prediction network based on the updated parameters.

[0132] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the fourth processing module 608 may be specifically configured to: calculate at least one of the following losses: calculate the signed distance gradient of two adjacent grid points based on the signed distance values corresponding to each grid point; determine a fourth loss based on the signed distance gradient and a gradient threshold of two adjacent grid points; calculate a Laplace smoothness value based on the first three-dimensional grid representation as a fifth loss; calculate the negative cosine similarity of the normal vectors of two adjacent surfaces in the first three-dimensional grid representation; determine a sixth loss based on the negative cosine similarity and a similarity threshold of the normal vectors of two adjacent surfaces; and determine a comprehensive loss based on the first loss, the second loss, the third loss, and at least one of the fourth loss, the fifth loss, and the sixth loss.

[0133] In some optional embodiments, Figure 7 FIG. 1 is a schematic diagram of a three-dimensional generating device provided by another exemplary embodiment of the present disclosure. Figure 7 As shown, the apparatus may include: a diffusion model 702 , a signed distance prediction model 704 , a grid generation model 706 and a texture prediction model 708 .

[0134] The diffusion model 702 may be used to generate multi-view images based on the image to be reconstructed.

[0135] The signed distance prediction model 704 can be used to obtain signed distance values corresponding to each grid point in a preset three-dimensional space based on a signed distance prediction network; the grid points are vertices of a cube in the preset three-dimensional space.

[0136] The grid generation model 706 may be used to determine a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points.

[0137] The texture prediction model 708 can be used to determine a textured target 3D mesh representation corresponding to the image to be reconstructed based on the first 3D mesh representation, using the multi-view images as supervisory images.

[0138] In some optional embodiments, based on any of the aforementioned 3D generation device embodiments, the signed distance prediction model 704 may be specifically configured to: perform multi-resolution hash coding on each grid point in a predetermined 3D space to obtain a multi-resolution coding feature; and predict a signed distance value corresponding to each grid point based on the multi-resolution coding feature.

[0139] In some optional embodiments, based on any of the aforementioned 3D generation device embodiments, the mesh generation model 706 may be specifically configured to: calculate, based on the signed distance values corresponding to each grid point, a set of points where the edge signed distance between any two adjacent grid points is a preset value; determine a first 3D mesh vertex set based on the set of points where the signed distance is the preset value; and determine a first 3D mesh representation based on the first 3D mesh vertex set.

[0140] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the supervision image may include a supervision normal map and a supervision color image corresponding to each perspective; the texture prediction model 708 may be specifically used to: determine the predicted normal map corresponding to each perspective, the predicted color image corresponding to the target perspective in each perspective, and the color corresponding to each three-dimensional mesh vertex based on the first three-dimensional mesh representation, the vertex normal vectors corresponding to each three-dimensional mesh vertex of the first three-dimensional mesh representation, and the observation perspective parameters. The observation perspective parameters may include the viewing direction corresponding to each perspective of the multi-perspective image. The predicted normal map and predicted color image of the target perspective are supervised based on the supervision image, and when the supervision result meets the preset conditions, the textured target three-dimensional mesh representation corresponding to the to-be-reconstructed image is determined based on the first three-dimensional mesh representation and the colors corresponding to each three-dimensional mesh vertex.

[0141] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the texture prediction model 708 can be specifically used to: extract the predicted normal map corresponding to each perspective based on the first three-dimensional grid representation and the viewing direction corresponding to each perspective; predict the color corresponding to each three-dimensional grid vertex based on each three-dimensional grid vertex represented by the first three-dimensional grid, the predicted normal map corresponding to each perspective, and the viewing direction, using the texture prediction network; determine the predicted color image corresponding to the target perspective based on the color corresponding to each three-dimensional grid vertex.

[0142] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the texture prediction model 708 can be specifically used to: hash encode each three-dimensional mesh vertex to obtain encoding features; based on the predicted normal map and viewing direction corresponding to each three-dimensional mesh vertex and each viewing angle, use a multi-layer perceptron to predict the color corresponding to each three-dimensional mesh vertex.

[0143] In some optional embodiments, based on any of the above three-dimensional generation device embodiments, the texture prediction model 708 may be specifically used to:

[0144] Based on the predicted color image corresponding to the target perspective, a predicted mask map corresponding to the target perspective is generated; based on the supervised color image corresponding to the target perspective, a supervised mask map corresponding to the target perspective is generated; based on the supervised color image, supervised normal map and supervised mask map corresponding to the target perspective, the predicted color image, predicted normal map and predicted mask map corresponding to the target perspective are supervised.

[0145] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the texture prediction model 708 can be specifically used to: determine a first loss based on the predicted color image and the supervised color image corresponding to the target perspective; determine a second loss based on the predicted normal map and the supervised normal map corresponding to the target perspective; determine a third loss based on the predicted mask map and the supervised mask map corresponding to the target perspective; determine a comprehensive loss based on the first loss, the second loss and the third loss; determine a supervision result based on the comprehensive loss; if the supervision result does not meet the preset conditions, update the parameters of the three-dimensional reconstruction algorithm based on the comprehensive loss, and repeat the steps of predicting the signed distance values corresponding to each grid point in the preset three-dimensional space based on the signed distance prediction network based on the updated parameters.

[0146] In some optional embodiments, based on any of the above-mentioned three-dimensional generation device embodiments, the texture prediction model 708 can be specifically configured to: calculate at least one of the following losses: calculate the signed distance gradient of two adjacent grid points based on the signed distance values corresponding to each grid point; determine a fourth loss based on the signed distance gradient and gradient threshold of two adjacent grid points; calculate a Laplace smoothness value based on the first three-dimensional grid representation as a fifth loss; calculate the negative cosine similarity of the normal vectors of two adjacent surfaces in the first three-dimensional grid representation; determine a sixth loss based on the negative cosine similarity and similarity threshold of the normal vectors of two adjacent surfaces. Determine a comprehensive loss based on the first loss, the second loss, the third loss, and at least one of the fourth loss, the fifth loss, and the sixth loss.

[0147] The beneficial technical effects corresponding to the exemplary embodiments of the device disclosed herein can be found in the corresponding beneficial technical effects of the exemplary method section above, which will not be repeated here.

[0148] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0149] Memory for storing computer programs;

[0150] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the three-dimensional generation method described in any of the above embodiments of the present disclosure is implemented.

[0151] Figure 8 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 8 As shown, the electronic device includes one or more processors and memory.

[0152] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0153] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may execute the computer program products to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0154] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0155] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0156] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0157] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0158] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the method according to various embodiments of the present disclosure described in the above part of this specification.

[0159] The computer program product may be written in any combination of one or more programming languages to implement the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0160] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above part of this specification.

[0161] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0162] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0163] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0164] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0165] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0166] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0167] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0168] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0169] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A three-dimensional generation method, characterized in that: include: Generate multi-view images based on the image to be reconstructed; Based on the signed distance prediction network, the signed distance values corresponding to each grid point in the preset three-dimensional space are obtained; The grid points are vertices of a cube in the preset three-dimensional space; Determining a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points; Taking the multi-view image as a supervision image, based on the first three-dimensional grid representation, determine the textured target three-dimensional grid representation corresponding to the image to be reconstructed, the supervision image includes a supervision normal map and a supervision color image corresponding to each view, including: based on the first three-dimensional grid representation, the vertex normal vectors corresponding to each three-dimensional grid vertex of the first three-dimensional grid representation and the observation perspective parameters, determine the predicted normal map corresponding to each said view, the predicted color image corresponding to the target view in each said view, and the color corresponding to each said three-dimensional grid vertex; the observation perspective parameters include the viewing direction corresponding to each view of the multi-view image; supervise the predicted normal map and the predicted color image of the target view based on the supervision image and the comprehensive loss, and when the supervision result meets the preset conditions, based on the first three-dimensional grid representation and the vertex normal vectors corresponding to each said three-dimensional grid vertex The corresponding color is determined to determine the target three-dimensional grid representation with texture corresponding to the image to be reconstructed, the comprehensive loss is determined according to the first loss, the second loss, the third loss, and at least one of the fourth loss, the fifth loss and the sixth loss, the first loss is determined based on the predicted color image and the supervised color image corresponding to the target perspective, the second loss is determined based on the predicted normal map and the supervised normal map corresponding to the target perspective, the third loss is determined based on the predicted mask map and the supervised mask map corresponding to the target perspective, the fourth loss is determined based on the signed distance gradient and the gradient threshold of the two adjacent grid points calculated based on the signed distance values corresponding to each grid point; the fifth loss is the Laplace smoothness value calculated based on the first three-dimensional grid representation; the sixth loss is determined based on the negative cosine similarity and the similarity threshold of the normal vectors of the two adjacent surfaces in the first three-dimensional grid representation.

2. The method according to claim 1, characterized in that The method of obtaining the signed distance values corresponding to each grid point in the preset three-dimensional space based on the signed distance prediction network includes: Performing multi-resolution hash coding on each of the grid points in the preset three-dimensional space to obtain multi-resolution coding features; Based on the multi-resolution coding features, the symbol distance values corresponding to the grid points are predicted.

3. The method according to claim 1, characterized in that The determining, based on the signed distance values corresponding to the grid points, a first three-dimensional grid representation corresponding to the image to be reconstructed, includes: Based on the signed distance values corresponding to the grid points, a point set is calculated where the edge signed distances between any two adjacent grid points are preset values; Determining a first three-dimensional mesh vertex set based on a point set whose signed distance is the preset value; The first three-dimensional mesh representation is determined based on the first three-dimensional mesh vertex set.

4. The method according to claim 1, wherein The determining, based on the first three-dimensional mesh representation, vertex normal vectors corresponding to respective three-dimensional mesh vertices of the first three-dimensional mesh representation, and observation perspective parameters, of predicted normal maps corresponding to respective perspectives, predicted color images corresponding to target perspectives among the perspectives, and colors corresponding to respective three-dimensional mesh vertices, includes: extracting, based on the first three-dimensional grid representation and the viewing directions corresponding to the respective viewing angles, predicted normal maps corresponding to the respective viewing angles; Based on each of the three-dimensional mesh vertices represented by the first three-dimensional mesh, the predicted normal map corresponding to each of the viewing angles, and the viewing direction, using a texture prediction network, predicting a color corresponding to each of the three-dimensional mesh vertices; The predicted color image corresponding to the target viewing angle is determined based on the colors corresponding to the vertices of the three-dimensional mesh.

5. The method according to claim 4, characterized in that The predicting, based on each of the three-dimensional mesh vertices represented by the first three-dimensional mesh, the predicted normal map corresponding to each of the viewing angles, and the viewing direction, using a texture prediction network to predict a color corresponding to each of the three-dimensional mesh vertices, includes: Performing hash coding on each of the three-dimensional mesh vertices to obtain coding features; Based on each of the three-dimensional mesh vertices and the predicted normal maps corresponding to each of the viewing angles and the viewing direction, a multi-layer perceptron is used to predict the color corresponding to each of the three-dimensional mesh vertices.

6. The method according to claim 1, characterized in that Supervising the predicted normal map of the target perspective and the predicted color image based on the supervision image, including: generating a prediction mask image corresponding to the target perspective based on the predicted color image corresponding to the target perspective; Based on the supervised color image corresponding to the target perspective, generating a supervised mask image corresponding to the target perspective; Based on the supervised color image, the supervised normal map and the supervised mask map corresponding to the target perspective, the predicted color image, the predicted normal map and the predicted mask map corresponding to the target perspective are supervised.

7. The method according to claim 6, characterized in that The step of supervising the predicted color image, the predicted normal map, and the predicted mask map corresponding to the target perspective based on the supervised color image, the supervised normal map, and the supervised mask map corresponding to the target perspective includes: Determine the supervision results based on the comprehensive losses; When the supervision result does not meet the preset conditions, the parameters of the three-dimensional reconstruction algorithm are updated based on the comprehensive loss, and based on the updated parameters, the step of predicting the signed distance values corresponding to each grid point in the preset three-dimensional space based on the signed distance prediction network is repeated.

8. A three-dimensional generation device, characterized in that: include: A first processing module, configured to generate a multi-view image based on the image to be reconstructed; The second processing module is used to obtain the signed distance value corresponding to each grid point in the preset three-dimensional space based on the signed distance prediction network; The grid points are vertices of a cube in the preset three-dimensional space; a third processing module, configured to determine a first three-dimensional grid representation corresponding to the image to be reconstructed based on the signed distance values corresponding to the grid points; The fourth processing module is used to use the multi-view image as a supervision image, and based on the first three-dimensional grid representation, determine the textured target three-dimensional grid representation corresponding to the image to be reconstructed, the supervision image includes a supervision normal map and a supervision color image corresponding to each perspective, including: based on the first three-dimensional grid representation, the vertex normal vectors corresponding to each three-dimensional grid vertex of the first three-dimensional grid representation and the observation perspective parameters, determine the predicted normal map corresponding to each of the perspectives, the predicted color image corresponding to the target perspective in each of the perspectives, and the color corresponding to each of the three-dimensional grid vertices; the observation perspective parameters include the viewing direction corresponding to each perspective of the multi-view image; supervise the predicted normal map and the predicted color image of the target perspective based on the supervision image and the comprehensive loss, and when the supervision result meets the preset conditions, based on the first three-dimensional grid representation and each of the three-dimensional grid The colors corresponding to the grid vertices are respectively determined to determine the target three-dimensional grid representation with texture corresponding to the image to be reconstructed, the comprehensive loss is determined according to at least one of the first loss, the second loss, the third loss, and the fourth loss, the fifth loss and the sixth loss, the first loss is determined based on the predicted color image and the supervised color image corresponding to the target perspective, the second loss is determined based on the predicted normal map and the supervised normal map corresponding to the target perspective, the third loss is determined based on the predicted mask map and the supervised mask map corresponding to the target perspective, the fourth loss is determined based on the signed distance gradient and the gradient threshold of the two adjacent grid points calculated based on the signed distance values corresponding to each grid point; the fifth loss is the Laplace smoothness value calculated based on the first three-dimensional grid representation; the sixth loss is determined based on the negative cosine similarity and the similarity threshold of the normal vectors of the two adjacent surfaces in the first three-dimensional grid representation.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is configured to execute a computer program stored in the memory, and when the computer program is executed, implements the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images