Model training method and device, three-dimensional generation method and device, equipment and storage medium

Abstract

The invention provides a model training method and device, a three-dimensional generation method and device, equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: according to a two-dimensional reference image of a sample three-dimensional model under each view angle, generating a prediction rendering image under each view angle and a prediction SDF parameter of a sampling point by adopting an initial three-dimensional generation model; calculating a first loss parameter according to the real rendered image and the predicted rendered image under the plurality of visual angles; and determining a winding number SDF parameter of the sampling point as a real SDF parameter of the sampling point, calculating a second loss parameter according to the predicted SDF parameter and the real SDF parameter of the sampling point, and performing parameter adjustment on the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain a target three-dimensional generation model. According to the embodiment of the invention, the supervised training precision and spatial resolution of the model can be improved, and the geometric precision and details of the model are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a model training method, a three-dimensional generation method, and apparatus, equipment, and storage medium thereof. Background Art

[0002] Sparse perspective 3D generation uses 2D images from a small number of perspectives, such as 1 to 6 2D images, to generate a target 3D model. 3D generation technology plays an important role in 3D digital asset creation, 3D special effects production, virtual reality, augmented reality and other fields.

[0003] In recent years, algorithms for generating sparse-view 3D models based on pre-trained generative modules have rapidly developed and are playing a significant role in the field of 3D content generation. These technologies require supervised training of 3D generative models based on a 3D model training set.

[0004] However, current technologies do not take into account the true signed distance parameters of the sampling points during supervised training based on real three-dimensional models. This limits the accuracy of supervised training to a certain extent, resulting in low spatial resolution of the trained three-dimensional generative model, and limited geometric accuracy and fewer details in the generated three-dimensional model. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a model training method, a three-dimensional generation method, and its apparatus, equipment and storage medium to improve the supervised training accuracy of the three-dimensional generation model, improve the spatial resolution of the three-dimensional generation model, and thereby improve the geometric accuracy of the generated three-dimensional model and enrich the model details.

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0007] Acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles;

[0008] Generate, based on the two-dimensional reference images under the multiple viewing angles, predicted rendered images under the multiple viewing angles and predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model using an initial three-dimensional generation model;

[0009] Calculating a first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images under the multiple perspectives;

[0010] Obtaining the winding number of the sampling point with respect to the real three-dimensional triangular mesh of the sample three-dimensional model and the unsigned distance parameter of the sampling point;

[0011] determining a true signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point;

[0012] Calculating a second loss parameter of the initial three-dimensional generation model according to the predicted signed distance parameter of the sampling point and the true signed distance parameter of the sampling point;

[0013] The initial three-dimensional generation model is adjusted according to the first loss parameter and the second loss parameter to obtain a target three-dimensional generation model.

[0014] In a second aspect, an embodiment of the present application further provides a three-dimensional generation method, the method comprising:

[0015] Obtain two-dimensional reference images of the target object at multiple perspectives;

[0016] Generate predicted signed distance parameters for each sampling point in the target three-dimensional space corresponding to the target object using a pre-trained target three-dimensional generative model based on the two-dimensional reference images from the multiple perspectives, wherein the target three-dimensional generative model is a generative model obtained using the model training method described in the first aspect above;

[0017] Surface extraction is performed according to the predicted signed distance parameters of each sampling point to obtain a three-dimensional model of the target object.

[0018] In a third aspect, an embodiment of the present application further provides a model training device, comprising:

[0019] A first acquisition module is used to acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles;

[0020] A first generation module is configured to generate, based on the two-dimensional reference images under the multiple viewing angles, predicted rendered images under the multiple viewing angles and predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model using an initial three-dimensional generation model;

[0021] A first calculation module is configured to calculate a first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images under the multiple perspectives;

[0022] a second calculation module, configured to obtain a winding number of the sampling point with respect to the real three-dimensional triangular mesh of the sample three-dimensional model and an unsigned distance parameter of the sampling point; determine a real signed distance parameter of the sampling point based on the winding number of the sampling point and the unsigned distance parameter of the sampling point; and calculate a second loss parameter of the initial three-dimensional generation model based on the predicted signed distance parameter of the sampling point and the real signed distance parameter of the sampling point;

[0023] A training module is used to adjust the parameters of the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain a target three-dimensional generation model.

[0024] In a fourth aspect, an embodiment of the present application further provides a three-dimensional generation device, comprising:

[0025] A second acquisition module is used to acquire two-dimensional reference images of the target object at multiple viewing angles;

[0026] a second generation module, configured to generate, based on the two-dimensional reference images from the multiple perspectives, predicted signed distance parameters for each sampling point in the target three-dimensional space corresponding to the target object using a pre-trained target three-dimensional generation model, wherein the target three-dimensional generation model is a generation model obtained using the model training method described in the first aspect above;

[0027] The extraction module is used to perform surface extraction according to the predicted signed distance parameters of each sampling point to obtain a three-dimensional model of the target object.

[0028] In the fifth aspect, an embodiment of the present application also provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the model training method as described in any one of the first aspects or the steps of the three-dimensional generation method described in the second aspect.

[0029] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method described in any one of the first aspects or the steps of the three-dimensional generation method described in the second aspect are executed.

[0030] The model training method, three-dimensional generation method, and its apparatus, equipment, and storage medium provided in the embodiments of the present application can use an initial three-dimensional generation model to generate predicted rendering images under multiple perspectives and predicted SDF parameters of sampling points in the corresponding three-dimensional space based on two-dimensional reference images of the sample three-dimensional model under multiple perspectives, and calculate the first loss parameter of the initial three-dimensional generation model based on the real rendered images and predicted rendered images under multiple perspectives. It can also obtain the winding number SDF parameter of the sampling point as the real SDF parameter based on the obtained winding number and unsigned distance parameter of the sampling point, and calculate the second loss parameter of the initial three-dimensional generation model based on the predicted SDF parameter and the real SDF parameter of the sampling point, and then adjust the parameters of the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain the target three-dimensional model. The technical solution provided in the present application adopts both the rendering loss and the SDF prediction loss of the model in the process of supervised training of the initial three-dimensional generation model. At the same time, for the sample three-dimensional model, the winding number SDF parameter of the sampling point is used as the true SDF parameter, thereby improving the accurate acquisition of the true SDF parameter of the sampling point, improving the accuracy of supervised training, and improving the spatial resolution of the three-dimensional generation model, thereby improving the geometric accuracy of the three-dimensional model in the three-dimensional generation process of the two-dimensional image based on sparse perspective, and enriching the details of the three-dimensional model.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0034] Figure 2 A flow chart of a method for determining the true SDF parameters of a sampling point in a model training method provided in an embodiment of the present application;

[0035] Figure 3 Provides a basic structural diagram of a three-dimensional generation module for the embodiment of the present application;

[0036] Figure 4 A flowchart of a method for executing a three-dimensional generation model in a model training method provided in an embodiment of the present invention;

[0037] Figure 5 A network structure diagram of a three-dimensional feature decoder provided in an embodiment of the present application;

[0038] Figure 6 A flowchart of a method for implementing feature decoding by a three-dimensional feature decoder provided in an embodiment of the present application;

[0039] Figure 7 A flowchart for obtaining a rendered image in a model training method provided in an embodiment of the present application;

[0040] Figure 8 A flowchart of a method for calculating rendering loss in a model training method is provided for an embodiment of the present application;

[0041] Figure 9 A schematic diagram of the structure of a three-dimensional large model is provided for the embodiment of the present application;

[0042] Figure 10 A schematic diagram of the rendering process of a differentiable grating rendering module in a model training method is provided for an embodiment of the present application;

[0043] Figure 11 A flowchart of a three-dimensional generation method provided in an embodiment of the present application;

[0044] Figure 12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0045] Figure 13 A schematic structural diagram of a three-dimensional generation device provided in an embodiment of the present application;

[0046] Figure 14 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0048] The three-dimensional generation method in one embodiment of the present disclosure can be applied to the construction of three-dimensional digital assets such as virtual three-dimensional models in any type of three-dimensional virtual scene, such as a three-dimensional game scene, a three-dimensional animation scene, a three-dimensional virtual scene based on virtual reality, and a three-dimensional virtual scene based on augmented reality. It should be noted that the technology used in the solution of the present application is a three-dimensional generation technology, which is different from the three-dimensional reconstruction technology, which are two different branches in the fields of computer graphics and computer vision. For three-dimensional reconstruction technology, its input image is usually a real image collected for a real object, which aims to restore a three-dimensional scene or a three-dimensional model. In the three-dimensional generation technology, its input image can be a virtual image generated by a computer program, or a hand-drawn image, etc., which aims to digitally represent a three-dimensional object, that is, to generate a three-dimensional model. In the specific execution process, its three-dimensional generation method can be run on a local terminal device or a server. When the three-dimensional generation method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0049] In order to facilitate a better understanding of the three-dimensional generation method provided in the embodiments of the present application, the technical terms involved in the following embodiments of the present application are first explained as follows.

[0050] Technical term explanation:

[0051] SDF: Signed Distance Function (SDF), which is an implicit representation of the surface of a 3D model. It can determine the distance from a 3D point to the surface of an object in a finite region in 3D space and also defines the sign of the distance. For example, if the point is inside the object, it is negative; if the point is outside the object, it is positive; if the point is on the surface of the object, it is 0.

[0052] MLP: Multilayer Perceptron (MLP), a type of neural network architecture.

[0053] Transformer: A deep neural network architecture, mainly composed of an input part, an encoding part, a decoding part, and an output part. The encoding part is a multi-layer encoder, which contains multiple encoder layers, each encoder layer consists of a multi-head self-attention sublayer and a feedforward fully connected sublayer; the decoding part is a multi-layer decoder, which contains multiple decoder layers, each decoder layer consists of a masked multi-head self-attention sublayer, a multi-head attention sublayer, and a feedforward fully connected sublayer.

[0054] Triplane: A 3D feature representation method that represents 3D spatial features as features of three orthogonal planes: the XY plane, the XZ plane, and the ZY plane. For a point (x, y, z) in space, it is projected onto each of the three orthogonal planes and interpolated to obtain feature vectors in each plane. These are then concatenated to form the final feature vector.

[0055] Marching Cubes: A surface extraction algorithm, also known as a surface rendering algorithm, is used to extract isosurfaces from volumetric data. The core concept is to define a set of cubes (voxels) and vertex scalar values. The algorithm then compares the vertices to a user-specified threshold, determines which edges of the voxels intersect the isosurface, creates a triangular patch, and connects all cube faces on the isosurface boundary to form a surface.

[0056] FlexiCubes: Flexible Isosurface Extraction for Gradient-Based MeshOptimization (FlexiCubes) is a differentiable isosurface extraction algorithm. In this algorithm, the surface of an object is represented by a cubic mesh. Each vertex in the cube mesh stores the SDF value and local deformation parameters. Both the SDF value and the local deformation vector are optimized using a neural network.

[0057] In order to clearly describe the three-dimensional generation method provided in the embodiment of the present application, the model training method provided in the embodiment of the present application is first explained and illustrated by examples through multiple embodiments in combination with multiple drawings, so as to clearly illustrate the pre-trained network model used in the three-dimensional generation method.

[0058] Figure 1 This is a flow chart of a model training method provided in an embodiment of the present application. The model training method is an end-to-end model training method. Figure 1 As shown, the model training method may include:

[0059] S101: Acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles.

[0060] During the specific implementation process, any 3D model can be obtained from a preset 3D model dataset, i.e., a training dataset, as a sample 3D model. The preset 3D model dataset also includes, for each 3D model, 2D reference images and real-world rendering images of the 3D model at multiple perspectives. Therefore, when a sample 3D model is obtained from the preset 3D model dataset, 2D reference images and real-world rendering images of the sample 3D model at multiple perspectives can be obtained. The model training method mentioned in this application is primarily used for sparse perspective 3D generation. Therefore, in this embodiment, the number of perspectives can be any integer between 2 and 6.

[0061] S102 : Based on the two-dimensional reference images under multiple viewing angles, an initial three-dimensional generation model is used to generate predicted rendering images under multiple viewing angles, as well as predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model.

[0062] The initial 3D generative model can be a pre-built end-to-end network model. Before executing this step, the parameters of the initial 3D generative model can be initialized. During the implementation process, two-dimensional reference images from multiple perspectives can be input into the initial 3D generative model to obtain predicted rendered images from multiple perspectives. At the same time, random sampling of 3D points in the 3D space corresponding to the sample 3D model is performed to obtain sampling points. The initial 3D generative model is used to perform SDF prediction on the sampling points to obtain the predicted signed distance parameters of the sampling points, i.e., the predicted SDF parameters.

[0063] S103: Calculate a first loss parameter of an initial three-dimensional generation model based on the real rendered images and the predicted rendered images under multiple perspectives.

[0064] For example, the rendering error at each perspective can be calculated based on the real rendered image and the predicted rendered image at each perspective, and then the first loss parameter of the initial 3D generation model can be calculated based on the rendering errors at multiple perspectives. The first loss parameter can also be called a rendering loss value, which can be expressed as L R .

[0065] S104 , obtaining the winding number of the sampling point with respect to the real three-dimensional triangular mesh of the sample three-dimensional model and the unsigned distance parameter of the sampling point.

[0066] In a specific implementation, for example, a preset winding number calculation method can be used to obtain the winding number of a sampling point relative to the true three-dimensional triangular mesh of the sample three-dimensional model, and a preset unsigned distance method can be used to obtain an unsigned distance function (UDF) parameter of the sampling point. The preset winding number calculation method can be, for example, a winding number (winding_number) calculation function. The unsigned distance parameter UFD(q) of the sampling point can represent the closest distance from the sampling point q to all triangles on the surface S of the true three-dimensional triangular mesh.

[0067] For example, for a sampling point q, the winding number w of the sampling point q can be obtained using the following formula (1) based on the coordinates of the sampling point q in the preset direction on the surface S of the real three-dimensional triangle mesh and the corresponding preset unit normal vector: S (q).

[0068]

[0069] Wherein, x is the coordinate of the sampling point on the surface S in the preset direction, and n is the unit normal vector of x.

[0070] S105 : Determine the real signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point.

[0071] After obtaining the winding number and UDF parameter of the sampling point, the winding number SDF parameter of the sampling point can be determined as the true signed distance parameter of the sampling point, that is, the true SDF parameter, based on the winding number and the unsigned distance parameter of the sampling point.

[0072] S106 , calculating a second loss parameter of the initial three-dimensional generation model according to the predicted signed distance parameter of the sampling point and the true signed distance parameter of the sampling point.

[0073] For example, the minimum mean square error can be calculated based on the predicted symbol distance parameter and the true symbol distance parameter of each sampling point as the second loss parameter of the sampling point, that is, the SDF loss value. In actual application scenarios, for example, the second loss parameter can be calculated using the following formula (2).

[0074] L SDF =‖SDF net (q)-SDF w (q)‖ 2 ...Formula (2)

[0075] Among them, L SDF is the second loss parameter, namely the SDF loss value, SDF net (q) is the predicted symbol distance parameter of sampling point q, SDF w(q) is the true signed distance parameter of sampling point q.

[0076] S107. Adjust the parameters of the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain a target three-dimensional generation model.

[0077] In a possible implementation method, a weighted summation of the first loss parameter and the second loss parameter can be performed to obtain an overall loss parameter as a target loss parameter, and based on the target loss parameter, a gradient back propagation algorithm is used to update the network parameters of the initial three-dimensional generative model, and then the above S102-S107 are re-executed to retrain the adjusted three-dimensional generative model based on two-dimensional reference images and real rendering images under multiple perspectives of the sample three-dimensional model until a preset stop iteration condition is met, and the three-dimensional generative model that meets the preset stop iteration condition is determined as the target three-dimensional generative model. The preset stop iteration condition can be, for example, that the retrained three-dimensional generative model reaches a preset network convergence condition, or reaches a preset maximum number of iterations. The preset network convergence condition can be that the recalculated target loss parameter is less than or equal to a preset loss parameter threshold.

[0078] It should be noted that when the target three-dimensional generation model is obtained, the model parameters of the target three-dimensional generation model can be recorded and stored, so that the target three-dimensional generation model can be applied by loading the model parameters during the model generation process.

[0079] The model training method provided in the present application can use an initial three-dimensional generation model to generate predicted rendering images under multiple perspectives and predicted SDF parameters of sampling points in the corresponding three-dimensional space based on two-dimensional reference images of the sample three-dimensional model under multiple perspectives, and calculate the first loss parameter of the initial three-dimensional generation model based on the real rendered images and predicted rendered images under multiple perspectives. It can also obtain the winding number SDF parameter of the sampling point as the real SDF parameter based on the obtained winding number and unsigned distance parameter of the sampling point, and calculate the second loss parameter of the initial three-dimensional generation model based on the predicted SDF parameter and the real SDF parameter of the sampling point, and then adjust the parameters of the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain the target three-dimensional model. The technical solution provided in the present application adopts both the rendering loss and the SDF prediction loss of the model in the process of supervised training of the initial three-dimensional generation model. At the same time, for the sample three-dimensional model, the winding number SDF parameter of the sampling point is used as the true SDF parameter, thereby improving the accurate acquisition of the true SDF parameter of the sampling point, improving the accuracy of supervised training, and improving the spatial resolution of the three-dimensional generation model, thereby improving the geometric accuracy of the three-dimensional model in the three-dimensional generation process of the two-dimensional image based on sparse perspective, and enriching the details of the three-dimensional model.

[0080] The specific implementation of the real SDF parameters of the sampling points will be described in detail below in conjunction with the embodiments. Figure 2 This is a flow chart of a method for determining the true SDF parameters of a sampling point in a model training method provided in an embodiment of the present application. Figure 2 As shown, in S105 shown above, determining the true signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point may include:

[0081] S201: Determine the sign of the sampling point according to the winding number of the sampling point.

[0082] In one possible implementation, the winding number of the sampling point can be compared with a preset parameter threshold, and the sign of the sampling point can be determined based on the comparison result. The preset parameter threshold can be, for example, a value greater than 0 and less than 1, such as 0.5.

[0083] For example, if the winding number of the sampling point is less than a preset parameter threshold, the sign of the sampling point is determined to be positive; if the winding number of the sampling point is greater than or equal to the preset parameter threshold, the sign of the sampling point is determined to be negative. The sign of the sampling point can be used to indicate whether the sampling point is inside or outside the real 3D triangular mesh of the sample 3D model.

[0084] S202: Determine a true signed distance parameter of the sampling point according to the sign of the sampling point and the unsigned distance parameter of the sampling point.

[0085] In a possible implementation, the signed and unsigned distance parameters of the sampling points may be combined to obtain the true signed distance parameters of the sampling points.

[0086] In practical application examples, the winding number w of the sampling point can be S (q) and the unsigned distance parameter UDF(q) of the sampling point, and the true signed distance parameter SDF(q) of the sampling point is determined using the following formula (3).

[0087] SDF w (q)=sign(0.5-w S (q))*UDF(q)......Formula (3)

[0088] The method provided in this embodiment can determine the sign of the sampling point by the winding number of the sampling point, and then the signed and unsigned distance parameters of the sampling point to obtain the true signed distance parameters of the sampling point, thereby achieving accurate acquisition of the true signed distance parameters of the sampling point and improving the accuracy of supervised training based on the true signed distance parameters.

[0089] In a possible implementation example, the initial 3D generative model can be a large model including multiple model structures, which can be called a 3D generative large model. The following continues with the framework example of the 3D generative model to further explain the implementation principle of the 3D generative model in detail. Figure 3 The basic structural diagram of the three-dimensional generation module is provided for the embodiment of this application. Figure 3 It can be seen that, for example, before executing the model training method described in the above embodiment, it is possible to construct Figure 3 The three-dimensional generation model shown may include: a two-dimensional feature encoder 301, a three-dimensional feature decoder 302, a first prediction network 303, and a second prediction network 304.

[0090] The two-dimensional feature encoder 301 can perform predictions based on the input two-dimensional reference images from multiple perspectives, obtaining and outputting two-dimensional image features from multiple perspectives, namely, two-dimensional feature maps. The three-dimensional feature decoder 302 can be a Tansfomer structure decoding network that can perform three-dimensional feature prediction on the two-dimensional image features of the input multiple images to obtain three-dimensional spatial features. After obtaining the three-dimensional spatial features, feature interpolation can be performed on the sampling points based on the three-dimensional spatial features to obtain interpolated features of the sampling points. The first prediction network 303 can calculate SDF parameters based on the interpolated features of the input sampling points to obtain predicted SDF parameters of the sampling points. Therefore, the first prediction network 303 can also be referred to as an SDF prediction network. Simultaneously, the second prediction network 304 can also perform rendering color prediction based on the interpolated features of the input sampling points to obtain the rendering color of the sampling points. Therefore, the second prediction network 304 can also be referred to as a color prediction network or an RGB prediction network. Both the first prediction network 303 and the second prediction network 304 can be neural networks with MLP structures.

[0091] During the model training process, a rendering module of a surface extraction algorithm and a rasterization rendering algorithm is also introduced. The rendering module can render at multiple perspectives based on the predicted symbol distance parameters and rendering colors of the sampling points, thereby obtaining predicted rendered images at multiple perspectives. For example, a surface extraction algorithm such as the Marching Cubes algorithm can be used to extract a table grid based on the predicted symbol distance parameters of the sampling points, and a rasterization rendering algorithm such as the differentiable rasterization rendering algorithm can be used to render the extracted surface grid at multiple perspectives based on the rendering colors of the sampling points, thereby obtaining predicted rendered images at multiple perspectives.

[0092] The following continues to describe in detail the implementation of determining the predicted signed distance parameters of the predicted rendered images and sampling points under multiple viewing angles in the above method embodiment in conjunction with the method flow chart and the structure of the 3D generation model. Figure 4 A flow chart of a method for generating a three-dimensional model in a model training method provided by an embodiment of the present invention. Figure 3 The structure of the three-dimensional generation model, accordingly, the initial three-dimensional generation model may include: a two-dimensional feature encoder, a three-dimensional feature decoder, a first prediction network, and a second prediction network. In this embodiment, the model parameters of each network module in the initial three-dimensional generation model are corresponding initialization parameters. Figure 4 As shown, in S102 described in the above embodiment, based on the two-dimensional reference images under multiple perspectives, the initial three-dimensional generation model is used to generate predicted rendered images under multiple perspectives, and the predicted signed distance parameters of the sample three-dimensional model corresponding to the sampling points in the three-dimensional space may specifically include:

[0093] S401 : performing feature encoding using a two-dimensional feature encoder based on two-dimensional reference images at multiple viewing angles to obtain two-dimensional image features at multiple viewing angles.

[0094] Specifically, the two-dimensional reference images under multiple perspectives can be input into the two-dimensional feature encoder for feature encoding to obtain the two-dimensional image features of multiple perspectives. The dimension of the two-dimensional image features of each perspective can be expressed as H×W×F, where H represents the height of the two-dimensional reference image, W represents the height of the two-dimensional reference image, and F represents the number of feature channels of the two-dimensional reference image.

[0095] S402: Based on the two-dimensional image features of multiple viewing angles, a three-dimensional feature decoder is used to perform feature decoding to obtain three-dimensional spatial features.

[0096] Specifically, the two-dimensional image features of multiple viewing angles may be input into a three-dimensional feature decoder for feature decoding to obtain and output three-dimensional spatial features.

[0097] In an implementation example, the three-dimensional spatial feature can be, for example, a three-dimensional spatial feature based on vector-matrix decomposition representation. Accordingly, the three-dimensional feature decoder can be called a three-dimensional feature decoder based on vector-matrix decomposition representation of tensors. The three-dimensional spatial feature based on vector-matrix decomposition representation is actually a combination of spatial features in multiple directions, wherein the spatial feature in each direction is the product of the feature vector in that direction and the feature matrix of the orthogonal plane corresponding to that direction. If the two-dimensional image features of multiple perspectives are F1,…F i ,…,F6, the three-dimensional feature decoder is a Transformer decoding network, then the three-dimensional feature decoder can be expressed as:

[0098] {V X ,M YZ ; V Y ,M XZ ; V Z ,M XZ}=Transformer(F1,…F i ,…,F6)

[0099] Among them, the three-dimensional spatial features output by the three-dimensional feature decoder based on vector-matrix decomposition representation can be expressed as {V X ,M YZ ; V Y ,M XZ ; V Z ,M XY}, specifically a combination of spatial features in three directions. Among them, the spatial feature in the X direction is: the feature vector of the X axis The characteristic matrix of the plane {V X ,M YZ}, the spatial characteristics of the Y direction are: the characteristic vector of the Y axis The characteristic matrix of the plane {V Y ,M XZ}, the spatial characteristics of the Z direction are: the characteristic vector of the Z axis The characteristic matrix of the plane {V Z ,M XY}. {V X ,V Z ,V Y} are the eigenvectors of X-axis, Y-axis and Z-axis respectively; {M YZ ,M XZ ,M XY} are the characteristic matrices of the YZ plane, XZ plane, and XY plane respectively.

[0100] In another example, the three-dimensional spatial features may also be represented in other forms, to which the embodiments of the present application are not limited.

[0101] S403: Perform feature interpolation processing on the sampling points according to the three-dimensional spatial features to obtain interpolation features of the sampling points.

[0102] In a possible implementation, the sampling point can be projected into the three-dimensional space feature to obtain the projection position of the projection point in the three-dimensional space feature. Based on the projection position, the adjacent points of the projection point are found, and the interpolation features of the sampling point are obtained by performing linear interpolation processing on the features of the adjacent points in the three-dimensional space feature.

[0103] In a specific implementation example, if the three-dimensional spatial feature is a three-dimensional spatial feature represented based on vector-matrix decomposition, that is, the three-dimensional spatial feature includes: spatial features in multiple directions, and the spatial feature in each direction is: the product of the feature vector in each direction and the feature matrix of the orthogonal plane corresponding to each direction, then the sampling point can be projected onto the feature vector in each direction and the feature matrix of the corresponding orthogonal plane for interpolation processing to obtain the interpolation feature of the sampling point in each direction and the interpolation feature of the corresponding orthogonal plane, and the interpolation feature of the sampling point in each direction and the interpolation feature of the corresponding orthogonal plane are multiplied to obtain the interpolation feature of the sampling point in each direction, and then the interpolation features of the sampling point in multiple directions are spliced to obtain the interpolation feature of the sampling point.

[0104] {V X ,M YZ ; V Y ,M XZ ; V Z ,M XZ Taking the three-dimensional spatial feature as an example, the sampling points can be projected onto the feature vector V of the X-axis X and the characteristic matrix M of the YZ plane YZ , get the interpolation feature V of the sampling point on the X axis X Interpolation features M in the (x) and YZ planes YZ (y,z), and the interpolation feature V of the sampling point on the X axis x Interpolation features M in the (x) and YZ planes YZ (y,z) is multiplied to obtain the interpolation feature V of the sampling point in the X direction X (x)*M YZ (y,z); Similarly, the interpolation feature V of the sampling point in the Y direction can also be obtained Y (y)*M xz (x,z) and the interpolation feature V in the Z direction z (z)*M XZ (x,z).

[0105] After obtaining the interpolation feature V of the sampling point in the X direction X (x)*M Yz (y,z), interpolation feature V in the Y direction Y (y)*M XZ (x,z) and the interpolation feature V in the Z direction Z (z)*M XZ (x,z), the following calculation formula (4) can be used:

[0106] G xyz ={V X (x)*M YZ (y,z),V Y(y)*M XZ (x,z),V Z (z)*M XZ (x,z)}……Formula (4)

[0107] Perform feature splicing to obtain the interpolation feature G of the sampling point xyz .

[0108] S404: Using a first prediction network to perform prediction based on the interpolation characteristics of the sampling points, to obtain predicted signed distance parameters of the sampling points.

[0109] S405 : Using a second prediction network to perform prediction based on the interpolation features of the sampling points, to obtain the rendering color of the sampling points.

[0110] After executing S403 to obtain the interpolation features of the sampling points, the interpolation features of the sampling points can be input into the first prediction network and the second prediction network respectively for SDF parameter prediction and color prediction to obtain the predicted signed distance parameter and rendering color of the sampling points.

[0111] S406 , performing rendering at multiple viewing angles according to the predicted symbol distance parameters of the sampling points and the rendering colors of the sampling points, to obtain predicted rendered images at multiple viewing angles.

[0112] In a possible implementation, a rendering module based on a surface extraction algorithm and a rasterization rendering algorithm can be used to perform rendering at multiple viewpoints. Specifically, the surface extraction algorithm can be used to extract a table grid based on the predicted signed distance parameters of the sampling points. Then, based on the rendering colors of the sampling points, a rasterization rendering algorithm can be used to rasterize and render the extracted surface grid at multiple viewpoints, resulting in predicted rendered images at multiple viewpoints.

[0113] The method provided in the embodiment of the present application can use a two-dimensional feature encoder to perform feature encoding on a two-dimensional reference image of multiple perspectives during the model training process to obtain two-dimensional image features at each perspective, use a three-dimensional feature decoder to perform spatial feature decoding on the two-dimensional image features of multiple perspectives to obtain three-dimensional spatial features, and use the three-dimensional spatial features to perform feature interpolation processing on the sampling points to obtain the interpolation features of each sampling point; and also use a first prediction network to perform prediction based on the interpolation features of the sampling points and a second prediction network to perform predictions respectively to obtain the predicted symbol distance parameters and rendering colors of the sampling points, and then render them respectively under multiple perspectives based on the predicted symbol distance parameters of the sampling points and the rendering colors of the sampling points to obtain predicted rendered images under multiple perspectives, thereby achieving accurate acquisition of predicted rendered images under each perspective during the model training process. Moreover, the method of this embodiment also provides three-dimensional spatial features based on vector-matrix decomposition representation, which effectively avoids the defect of insufficient three-dimensional feature representation accuracy brought about by the use of three-plane three-dimensional spatial feature representation, and achieves the accuracy and representation accuracy of accurate feature representation of three-dimensional space, especially for feature representation of complex three-dimensional space.

[0114] The following embodiment of the present application also provides a specific network structure of a three-dimensional feature decoder as an implementation example of a three-dimensional feature decoder based on a vector-matrix decomposition representation of a tensor, to explain in detail the implementation details of three-dimensional spatial features based on a vector-matrix decomposition representation. The following is a detailed description of the specific network structure of the three-dimensional feature decoder and its corresponding method flow. Figure 5 A network structure diagram of a three-dimensional feature decoder provided in an embodiment of the present application, Figure 6 This is a flow chart of a method for implementing feature decoding by a three-dimensional feature decoder provided in an embodiment of the present application. Figure 5 It can be seen that the three-dimensional feature decoder provided in the embodiment of the present application includes: a vector feature decoder, a plane feature decoder, a vector feature linear layer, a plane feature linear layer and a feature output layer.

[0115] Continue to refer Figure 6 As in S402 of the above embodiment, a three-dimensional feature decoder is used to perform feature decoding based on the two-dimensional image features of multiple perspectives, and the three-dimensional spatial features obtained may include:

[0116] S601: Based on preset initial position codes in multiple directions and two-dimensional image features of multiple viewing angles, a vector feature decoder is used to perform feature decoding to generate initial feature vectors in multiple directions.

[0117] The vector feature decoder may be a first multi-layer cross-attention decoder, such as a first multi-layer cross-attention Transformer decoder, which may perform cross-attention learning on the initial position encodings in multiple directions perpendicular to each other based on the two-dimensional image features of multiple perspectives, i.e., feature maps of multiple perspectives, to obtain initial feature vectors in multiple directions. The initial position encodings in multiple directions are random initialization values in multiple directions. In the process of performing cross-attention learning using the first multi-layer cross-attention decoder, the initial position encodings in multiple directions are continuously updated based on the two-dimensional image features of multiple perspectives until the initial feature vectors in multiple directions are obtained.

[0118] S602: According to the preset initial position codes of the multiple planes and the two-dimensional image features of the multiple viewing angles, a plane feature decoder is used to perform feature decoding to generate initial feature matrices of the multiple planes.

[0119] The plane feature decoder can be a second multi-layer cross-attention decoder, such as a second multi-layer cross-attention Transformer decoder, which can perform cross-attention learning on the initial position encodings of multiple planes based on the two-dimensional image features of multiple perspectives, that is, the feature maps of multiple perspectives, to obtain initial feature matrices of multiple mutually perpendicular planes. The initial position encodings of the multiple planes are random initialization values of the multiple planes. In the process of cross-attention learning using the second multi-layer cross-attention decoder, the initial position encodings of the multiple planes are continuously updated according to the two-dimensional image features of the multiple perspectives until the initial feature vectors of the multiple planes are obtained.

[0120] S603: Use a vector feature linear layer to perform linear processing on the initial feature vectors in multiple directions to obtain feature vectors in multiple directions.

[0121] The vector feature linear layer can be a linear mapping relationship that stores a preset feature dimension, which can be expressed as a weight W KxL Correspondingly, if the feature dimension of the feature vector in each direction output by the vector feature decoder can be expressed as DxL, the vector feature linear layer can perform linear mapping processing on the feature dimensions of the initial feature vectors in multiple directions to obtain feature vectors in multiple directions. The linear mapping processing can be a vector feature linear layer that maps the initial feature vectors in multiple directions from dimension L to dimension K according to the linear mapping relationship to obtain feature vectors in the corresponding directions with feature dimensions of DxK.

[0122] Assume that the dimension of the initial position encoding in each direction is DxL, and the dimension of the initial position encoding in the three directions can be expressed as 3xDxL. The dimension of the initial feature vector in each direction obtained by the vector feature decoder is also DxL. Correspondingly, the dimension of the initial feature vector in the three directions can be expressed as 3xDxL. The initial feature vector in each direction is mapped from dimension L to dimension K using the vector feature linear layer, and the dimension of the feature vectors in multiple directions can be expressed as 3xDxK. Among them, the feature vectors in multiple directions can be expressed as {V X ,V Z ,V Y} are the eigenvectors of the X-axis, Y-axis, and Z-axis respectively.

[0123] S604: Use a plane feature linear layer to perform linear processing on the initial feature matrices of the multiple planes to obtain feature matrices of the multiple planes.

[0124] Similarly, the plane feature linear layer can be a linear mapping relationship that stores a preset feature dimension, which can be expressed as a weight W KxL Correspondingly, if the feature dimensions of the feature matrix of each plane output by the plane feature decoder can be expressed as HxWxL, the plane feature linear layer can perform linear mapping processing on the feature dimensions of the initial feature matrices of multiple planes to obtain feature vectors in multiple directions. Here, the linear mapping processing can be a plane feature linear layer that maps the initial feature matrices of multiple planes from L dimensions to K dimensions according to the linear mapping relationship to obtain feature vectors in corresponding directions with feature dimensions of HxWxK.

[0125] Assume that the dimension of the initial position encoding of each plane is HxWxL, the dimension of the initial position encoding of the three planes can be expressed as 3xHxWxL, and the dimension of the initial feature matrix of each plane obtained by the plane feature decoder is also HxWxL. Correspondingly, the dimension of the initial feature vector of the three planes can be expressed as 3xHxWxL. The initial feature vector of each plane is mapped from dimension L to dimension K using the plane feature linear layer, and the dimension of the feature vectors of the multiple planes can be expressed as 3xHxWxK. Among them, the feature vectors of the multiple planes can be expressed as {M YZ ,M XZ ,M XY} are the eigenvectors of the YZ plane, XZ plane, and XY plane respectively.

[0126] S605: Using the feature output layer, perform a product operation on the feature vector in each direction and the feature matrix of the orthogonal plane corresponding to each direction to obtain the spatial feature in each direction.

[0127] In the specific implementation process, the feature output layer can be used to combine the feature vectors in multiple directions and the feature matrices in multiple planes, and the three-dimensional spatial features based on the vector-matrix decomposition representation can be expressed as {V X ,M YZ ; V Y ,M XZ ; V Z ,M XY In the process of feature combination, the feature vector of each direction and the feature matrix of the corresponding orthogonal plane can be multiplied to obtain the spatial features of each direction. For example, the feature vector V of the X axis can be X and the characteristic matrix M of the YZ plane YZ Perform product operation to obtain the spatial feature {V X ,M YZ}; The Y-axis eigenvector V Y and the characteristic matrix M of the XZ plane XZ Perform product operation to obtain the spatial feature {V Y ,M XZ}; The Z-axis eigenvector V Z and the characteristic matrix M of the XY plane XY Perform product operation to obtain the spatial feature {V Z ,M XY}.

[0128] The method of this embodiment provides a three-dimensional feature decoder based on tensor vector-matrix decomposition representation to obtain three-dimensional spatial features based on vector-matrix decomposition representation. Since it obtains initial feature vectors in multiple directions and initial feature matrices in multiple planes based on vector feature decoder and plane feature decoder respectively, and then uses vector feature linear layer and plane feature linear layer to perform linear mapping processing of feature dimensions on the initial feature vectors in each direction and the initial feature matrix of each plane respectively, the feature vectors in each direction and the initial feature matrix of each plane can be made more accurate, so that the three-dimensional spatial features finally obtained based on vector-matrix decomposition representation can realize the real feature expression of the three-dimensional space, thereby effectively avoiding the defect of insufficient three-dimensional feature representation accuracy brought about by the three-plane three-dimensional space feature representation, and realizing the accuracy and representation precision of accurate feature representation of the three-dimensional space, especially for the feature representation of complex three-dimensional space.

[0129] In some other possible implementation examples, embodiments of the present application further provide a rendering module based on a surface extraction algorithm, a triangular projection algorithm, and a rasterization rendering algorithm. The surface extraction algorithm is a Marching Cubes algorithm, and the rasterization rendering algorithm can be a differentiable raster rendering algorithm. Therefore, the rendering module can also be referred to as a differentiable raster rendering module based on Marching Cubes combined with triangular projection. Rendering using this rendering module is described in detail below in conjunction with embodiments. Figure 7 A flowchart for obtaining a rendered image in a model training method provided in an embodiment of the present application is shown in FIG. Figure 7 As shown in S406 above, rendering is performed at multiple viewing angles according to the predicted symbol distance parameter of the sampling point and the rendering color of the sampling point, and the predicted rendered images at multiple viewing angles may include:

[0130] S701 , extracting a surface mesh according to the predicted signed distance parameters of the sampling points to obtain an initial triangular mesh.

[0131] For example, a preset surface extraction algorithm, such as Marching Cubes, can be used to extract a surface mesh based on the predicted signed distance parameters of the sampling points to obtain an initial triangular mesh. Specifically, a preset surface extraction algorithm can be used to extract sampling points with an SDF parameter of 0 based on the predicted signed distance parameters of the sampling points to construct a mesh as the initial triangular mesh.

[0132] S702: Use the first prediction network to predict the initial triangular mesh to obtain the signed distance parameter and normal vector of each vertex on the initial triangular mesh.

[0133] In one possible implementation, feature interpolation can be performed on each vertex based on the coordinate position of each vertex on the initial triangular mesh and the three-dimensional spatial features to obtain the interpolation features of each vertex, and the interpolation features of each vertex are input into the first prediction network for prediction to obtain the signed distance parameters and normal vectors of each vertex.

[0134] S703 : Project the initial triangular mesh according to the signed distance parameter and normal vector of each vertex to obtain a projected triangular mesh.

[0135] In a possible implementation, the triangle projection algorithm can be a closest projection algorithm based on a signed distance parameter. Specifically, based on the signed distance parameter and normal vector of each vertex, each vertex of the initial triangle mesh can be projected to the closest point of the signed distance parameter. By projecting each vertex in the initial triangle mesh, the initial triangle mesh is projected to obtain a projected triangle mesh. By projecting the triangle mesh, fine-tuning of the triangle mesh based on the signed distance parameter can be achieved.

[0136] For example, according to the signed distance parameter of each vertex, each vertex can be translated along the opposite direction of the normal vector of each vertex by the corresponding signed distance parameter to obtain the translated vertices, and then obtain the projected triangular mesh composed of the translated vertices.

[0137] S704 , rendering the projected triangular mesh at multiple viewing angles according to the rendering colors of the sampling points, to obtain predicted rendering images at multiple viewing angles.

[0138] The preset rasterization rendering algorithm is, for example, a differentiable rasterization rendering algorithm. Specifically, the projected triangle mesh is subjected to differentiable rasterization rendering at each viewing angle according to the rendering color of the sampling point, thereby obtaining a predicted rendering image at each viewing angle.

[0139] The method provided in the embodiment of the present application can realize the projection of the initialized triangular mesh by projecting each vertex in the initial triangular mesh based on the nearest signed distance parameter before rendering the triangular mesh during the model training process, realize the micro-adjustment of the triangular mesh based on the signed distance parameter, and improve the rendering accuracy of the triangular mesh. At the same time, it avoids the defects of large video memory consumption and limited gradient back propagation accuracy brought about by the use of differentiable isosurface extraction algorithms such as the FlexiCubes algorithm. While reducing video memory consumption, it also improves the model training accuracy.

[0140] Based on the above embodiments, the present application also provides a possible implementation example of calculating rendering loss. In this embodiment, the real rendered image and the predicted rendered image at each viewing angle can be respectively a type of rendered image. For example, the real rendered image at each viewing angle may include: a real color image, a real binary mask image, a real depth map, and a real normal map. Correspondingly, the predicted rendered image at each viewing angle may include: a predicted color image, a predicted binary mask image, a predicted depth map, and a predicted normal map. Figure 8 A flow chart of a method for calculating rendering loss in a model training method is provided for an embodiment of the present application. Figure 8 As shown, in the above method embodiment, in S103, calculating the first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images under multiple perspectives may include:

[0141] S801. Calculate the color error, binary mask error, depth error, and normal error at each viewing angle based on the true color image, true binary mask image, true depth map, and true normal map at each viewing angle, as well as the predicted color image, predicted binary mask image, predicted depth map, and predicted normal map at each viewing angle.

[0142] Specifically, the real color image under each viewing angle can be and predicted color image ci , calculate the color error L at each viewing angle C ; According to the real binary mask image at each perspective and predicted binary mask image M i , calculate the binary mask error L at each viewing angle M ; According to the real depth map at each perspective and the predicted depth map D i , calculate the depth error L at each viewing angle D ; According to the real normal map at each viewing angle and the predicted normal map n i , calculate the normal error L at each viewing angle N .

[0143]

[0144] Wherein, P is the preset total number of pixels of the rendered image.

[0145] S802: Calculate a first loss parameter based on color errors, binary mask errors, depth errors, and normal errors under multiple viewing angles.

[0146] Specifically, the color error L under multiple viewing angles can be C , binary mask error L M , depth error L D And the normal error L N , perform weighted summation of each error to obtain the first loss parameter L R In one example, the formula L R =L C +L M +L D +L N Calculate the first loss parameter L R Of course, the calculation formula of the first loss parameter is not limited to the above example explanation, and this application does not impose any restrictions on this.

[0147] The method provided in the embodiment of the present application can calculate the first loss parameter as the rendering parameter by calculating the error between each predicted rendering image and the corresponding true value, that is, the real rendered image, at each perspective, thereby realizing the calculation of the rendering loss of the model in multiple dimensions, improving the calculation accuracy of the rendering loss, and effectively ensuring the supervised training accuracy during the model training process.

[0148] The three-dimensional models provided in the above embodiments of the present application can, for example, be three-dimensional generation of large modules. In order to more clearly understand the model training methods provided in the above embodiments of the present application, the model training is explained in detail below in combination with the specific model framework of the three-dimensional generation of large models. Figure 9The present invention provides a schematic diagram of the structure of a three-dimensional large model. Before model training, a three-dimensional large model can be constructed. Figure 9 As shown, the large 3D generation model includes: a 2D feature encoder, a 3D feature decoder, a first prediction network, and a second prediction network. The 3D feature decoder can be a 3D feature decoder based on a vector-matrix decomposition representation of a tensor, also known as a Transformer decoder based on a tensor decomposition representation. The first prediction network and the second prediction network are respectively an SDF prediction network and a color prediction network, both of which are neural networks with an MLP structure.

[0149] During model training, a 3D model, 2D reference images of the 3D model at six viewing angles, and real rendered images at six viewing angles are randomly selected from the 3D model dataset. The real rendered images at each viewing angle can include: a real RGB image, a real binary mask image, a real depth map, and a real normal map.

[0150] Subsequently, the 2D reference images under the six viewing angles can be input into the two-dimensional feature encoder, which performs feature extraction to obtain the 2D image features F1,…F under the six viewing angles. i ,…,F6.

[0151] However, the 2D image features F1,…F under 6 viewing angles are i ,…,F6 is input to the 3D feature decoder for feature decoding, and the 3D spatial feature {V X ,M YZ ; V Y ,M XZ ; V Z ,M XZ}.

[0152] Next, randomly select 3D sampling points (x, y, z) in the three-dimensional space, and project the selected sampling points onto the feature vectors in each direction of the 3D space feature and the feature matrix of the corresponding orthogonal plane, and obtain the interpolation features of the sampling points in each direction and the interpolation features of the corresponding orthogonal plane, and multiply them to obtain the interpolation features of the sampling points in the three directions. Finally, the interpolation features of all directions are spliced together to obtain the interpolation features G of the sampling points. xyz .

[0153] Next, the interpolation feature G of each sampling point is xyz The data is input into the first prediction network to obtain the predicted SDF parameters and normal vectors of each sampling point; at the same time, it is also input into the second prediction network to obtain the rendering color of the sampling point.

[0154] During the model training process, a differentiable raster rendering module based on surface extraction algorithm, triangle projection algorithm and rasterization rendering algorithm is also introduced, as shown below in conjunction with the accompanying drawings. Figure 10 The present invention provides a schematic diagram of the rendering process of a differentiable raster rendering module in a model training method. Figure 10 For sampling points in 3D extraction within a preset three-dimensional space, the first prediction network is used to obtain the predicted SDF parameters and normal vectors for each sampling point. The predicted SDF parameters and normals for each sampling point are then extracted using a surface extraction algorithm to obtain a triangular mesh. For each vertex in the triangular mesh, the first preset network is used to predict the SDF parameters and normal vectors for each vertex. Each vertex is then translated in the opposite direction of its normal vector by a corresponding signed distance parameter to obtain a projected triangular mesh consisting of the translated vertices.

[0155] Next, based on the rendering color of each sampling point, a differentiable raster rendering algorithm is used to perform differentiable raster rendering on the projected triangle mesh at each viewing angle, obtaining a predicted RGB image, a predicted binary mask image, a predicted depth map, and a predicted normal map at each viewing angle. The error between each predicted image and the corresponding real image is then calculated to obtain the rendering error.

[0156] At the same time, for randomly selected sampling points on the selected 3D model, the winding number SDF parameter of each sampling point is also calculated as the true SDF parameter. After obtaining the true SDF parameter of the sampling point, the SDF loss can be obtained based on the minimum mean square error between the true SDF parameter of the sampling point and the predicted SDF parameter obtained above.

[0157] Next, the rendering loss and SDF loss are weighted and summed to obtain the target loss parameter. Based on this target loss parameter, the gradient backpropagation algorithm is used to update the network parameters of the large 3D generative model. These steps are then repeated until the network converges or reaches the preset maximum iteration condition. The model parameters are saved to obtain the trained target 3D generative model.

[0158] After the three-dimensional generation large model is trained using the model training method provided in the embodiment of the present application, the three-dimensional model generated by the trained three-dimensional generation large model, that is, the three-dimensional triangular mesh model has a smaller error and occupies less video memory during training. The three-dimensional generation large model is evaluated based on a public data set, and its model accuracy is improved from 0.85 of the traditional three-dimensional generation model to 0.88. Secondly, the video memory consumption during the model training process is also reduced. For example, the traditional model training method has the limitation that it can only be trained under the A100 graphics card with 40G video memory, while the model training method provided in the embodiment of the present application can be trained on a 3090 graphics card with 24G video memory.

[0159] In the case of obtaining a target three-dimensional generation model by adopting the model training method provided by any of the above embodiments, the embodiment of the present application also provides an implementation example of three-dimensional generation using the target three-dimensional generation model. The device body for executing the three-dimensional generation method can be the same hardware body as the device body for the above-mentioned model training method, or it can be a different hardware body, and the embodiment of the present application is not limited to this. Before executing the three-dimensional generation method, the pre-trained model parameters are first imported, and the three-dimensional generation method provided by the embodiment of the present application can be executed in combination with the model. Among them, the pre-trained model parameters may at least include: the network model parameters of the target three-dimensional generation model mentioned above.

[0160] The specific implementation process of three-dimensional generation is explained and illustrated in detail below with reference to the accompanying drawings. Figure 11 A flowchart of a three-dimensional generation method provided in an embodiment of the present application is shown in FIG. Figure 11 As shown, the method may include:

[0161] S1101: Acquire two-dimensional reference images of a target object at multiple viewing angles.

[0162] S1102 : Based on the two-dimensional reference images under multiple viewing angles, a pre-trained target three-dimensional generation model is used to generate predicted signed distance parameters of each sampling point in the target three-dimensional space corresponding to the target object.

[0163] The target three-dimensional generative model is a generative model obtained by using any of the model training methods described in the embodiments.

[0164] In a possible implementation, a 2D feature encoder in a target 3D generative model can be used to perform feature encoding based on 2D reference images from multiple viewpoints to obtain 2D image features from multiple viewpoints. Based on the 2D image features from multiple viewpoints, a 3D feature decoder in the target 3D generative model can be used to perform feature decoding to obtain 3D spatial features. Based on the 3D spatial features, feature interpolation processing is performed on the sampling points to obtain interpolated features of the sampling points. Based on the interpolated features of the sampling points, a first prediction network in the target 3D generative model is then used to perform prediction to obtain predicted signed distance parameters and normal vectors for the sampling points. Correspondingly, based on the interpolated features of the sampling points, a second prediction network in the target 3D generative model can be used to perform prediction to obtain predicted colors for the sampling points.

[0165] S1103 : Perform surface extraction based on the predicted signed distance parameters of the sampling points to obtain a three-dimensional model of the target object.

[0166] The 3D model can be a colorless 3D mesh model or a colored 3D mesh model. Specifically, a surface extraction algorithm can be used to extract the surface based on the predicted signed distance parameters and normal vectors of the sampling points to obtain a 3D surface mesh, i.e., a colorless 3D mesh model. Alternatively, the 3D surface mesh can be rendered based on the predicted colors of the sampling points to obtain a colored 3D mesh model.

[0167] The three-dimensional generation method provided in the embodiment of the present application uses the winding number SDF parameters of the sampling points as the true SDF parameters during the training of the target three-dimensional generation model, thereby improving the accurate acquisition of the true SDF parameters of the sampling points, improving the accuracy of supervised training, improving the spatial resolution of the three-dimensional generation model, and improving the geometric accuracy and details of the generated three-dimensional model.

[0168] The following describes the model training device, three-dimensional generation device, equipment and storage medium provided in this application for execution. The specific implementation process and technical effects are described above and will not be repeated below.

[0169] Figure 12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 12 As shown, the model training device 1200 may include:

[0170] The first acquisition module 1201 is configured to acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles.

[0171] The first generation module 1202 is used to generate predicted rendering images under multiple perspectives and predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model based on the two-dimensional reference images under multiple perspectives using the initial three-dimensional generation model.

[0172] The first calculation module 1203 is used to calculate a first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images under multiple perspectives.

[0173] The second calculation module 1204 is used to obtain the winding number of the sampling point for the real three-dimensional triangular mesh of the sample three-dimensional model and the unsigned distance parameter of the sampling point; determine the real signed distance parameter of the sampling point based on the winding number of the sampling point and the unsigned distance parameter of the sampling point; calculate the second loss parameter of the initial three-dimensional generation model based on the predicted signed distance parameter of the sampling point and the real signed distance parameter of the sampling point.

[0174] The training module 1205 is used to adjust the parameters of the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain the target three-dimensional generation model.

[0175] The model training device provided in this embodiment uses both the model's rendering loss and the model's SDF prediction loss during the supervised training of the initial three-dimensional generation model. At the same time, for the sample three-dimensional model, the winding number SDF parameter of the sampling point is used as the true SDF parameter, thereby improving the accurate acquisition of the true SDF parameter of the sampling point, improving the accuracy of supervised training, and improving the spatial resolution of the three-dimensional generation model. This further improves the geometric accuracy of the three-dimensional model in the three-dimensional generation process using two-dimensional images based on sparse perspectives, and enriches the details of the three-dimensional model.

[0176] In a feasible implementation, the second calculation module 1204 is specifically configured to: determine the sign of the sampling point according to the winding number of the sampling point; and determine the true signed distance parameter of the sampling point according to the sign of the sampling point and the unsigned distance parameter of the sampling point.

[0177] In a feasible implementation scheme, the second calculation module 1204 is specifically used to: if the winding number of the sampling point is less than the preset parameter threshold, determine that the sign of the sampling point is positive; if the winding number of the sampling point is greater than or equal to the preset parameter threshold, determine that the sign of the sampling point is negative.

[0178] In one feasible implementation, the initial 3D generation model includes: a 2D feature encoder, a 3D feature decoder, a first prediction network, and a second prediction network. The first generation module 1202 is specifically configured to: perform feature encoding using a 2D feature encoder based on 2D reference images from multiple perspectives to obtain 2D image features from multiple perspectives; perform feature decoding using a 3D feature decoder based on the 2D image features from multiple perspectives to obtain 3D spatial features; perform feature interpolation processing on the sampling points based on the 3D spatial features to obtain interpolation features of the sampling points; perform prediction using a first prediction network based on the interpolation features of the sampling points to obtain predicted symbol distance parameters of the sampling points; perform prediction using a second prediction network based on the interpolation features of the sampling points to obtain rendering colors of the sampling points; and perform rendering from multiple perspectives based on the predicted symbol distance parameters and rendering colors of the sampling points to obtain predicted rendered images from multiple perspectives.

[0179] In one feasible implementation, the three-dimensional spatial features include: spatial features in multiple directions, where the spatial features in each direction are: the product of the feature vector in each direction and the feature matrix of the orthogonal plane corresponding to each direction. The first generation module 1202 is specifically configured to: project the sampling points onto the feature vector in each direction and the feature matrix of the corresponding orthogonal plane, respectively, and perform interpolation processing to obtain the interpolation features of the sampling points in each direction and the interpolation features of the corresponding orthogonal plane; multiply the interpolation features of the sampling points in each direction with the interpolation features of the corresponding orthogonal plane to obtain the interpolation features of the sampling points in each direction; and splice the interpolation features of the sampling points in multiple directions to obtain the interpolation features of the sampling points.

[0180] In a feasible implementation, the three-dimensional feature decoder includes: a vector feature decoder, a plane feature decoder, a vector feature linear layer, a plane feature linear layer, and a feature output layer.

[0181] The first generation module 1202 is specifically used to: use a vector feature decoder to perform feature decoding based on the preset initial position codes of multiple directions and the two-dimensional image features of multiple perspectives to generate initial feature vectors in multiple directions; use a plane feature decoder to perform feature decoding based on the preset initial position codes of multiple planes and the two-dimensional image features of multiple perspectives to generate initial feature matrices of multiple planes; use a vector feature linear layer to linearly process the initial feature vectors of multiple directions to obtain feature vectors in multiple directions; use a plane feature linear layer to linearly process the initial feature matrices of multiple planes to obtain feature matrices in multiple directions; use a feature output layer to multiply the feature vector of each direction with the feature matrix of the orthogonal plane corresponding to each direction to obtain the spatial features of each direction.

[0182] In a feasible implementation scheme, the first generation module 1202 is specifically used to: extract the surface mesh according to the predicted signed distance parameters of the sampling points to obtain an initial triangular mesh; use the first prediction network to predict the initial triangular mesh to obtain the signed distance parameters and normal vectors of each vertex on the initial triangular mesh; project the initial triangular mesh according to the signed distance parameters and normal vectors of each vertex to obtain a projected triangular mesh; render the projected triangular mesh at multiple perspectives according to the rendering color of the sampling points to obtain predicted rendered images at multiple perspectives.

[0183] In a feasible embodiment, the real rendered image at each viewing angle includes: a real color image, a real binary mask image, a real depth map and a real normal map, and the predicted rendered image at each viewing angle includes: a predicted color image, a predicted binary mask image, a predicted depth map and a predicted normal map.

[0184] The first calculation module 1203 is specifically used to: calculate the color error, binary mask error, depth error and normal error at each perspective based on the true color image, true binary mask image, true depth map and true normal map at each perspective, as well as the predicted color image, predicted binary mask image, predicted depth map and predicted normal map at each perspective; calculate the first loss parameter based on the color error, binary mask error, depth error and normal error at multiple perspectives.

[0185] Figure 13 A schematic diagram of the structure of a three-dimensional generation device provided in an embodiment of the present application is shown in FIG. Figure 13 As shown, the three-dimensional generating device 1300 may include:

[0186] The second acquisition module 1301 is used to acquire two-dimensional reference images of the target object at multiple viewing angles;

[0187] The second generation module 1302 generates predicted signed distance parameters of each sampling point in the target three-dimensional space corresponding to the target object based on the two-dimensional reference images from multiple perspectives using a pre-trained target three-dimensional generation model, wherein the target three-dimensional generation model is a generation model obtained using the model training method described in any of the above embodiments.

[0188] The extraction module 1303 is used to perform surface extraction based on the predicted signed distance parameters of each sampling point to obtain a three-dimensional model of the target object.

[0189] In the three-dimensional generation method provided in this embodiment, since the winding number SDF parameters of the sampling points are used as the true SDF parameters during the training of the target three-dimensional generation model, the accurate acquisition of the true SDF parameters of the sampling points is improved, the accuracy of supervised training is improved, the spatial resolution of the three-dimensional generation model is improved, and the geometric accuracy and details of the generated three-dimensional model are also improved.

[0190] Figure 14 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 14 As shown, the computer device includes: a processor 1401, a storage medium 1402 and a bus 1403. The storage medium 1402 stores machine-readable instructions executable by the processor 1401. When the computer device runs a model training method as described in the embodiment, the processor 1401 communicates with the storage medium 1402 via the bus 1403. The processor 1401 executes the machine-readable instructions to perform the following steps:

[0191] Acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles;

[0192] Based on the two-dimensional reference images under multiple viewing angles, an initial three-dimensional generation model is used to generate predicted rendered images under multiple viewing angles, as well as predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model;

[0193] Calculating a first loss parameter of an initial 3D generation model based on the real rendered images and the predicted rendered images under multiple perspectives;

[0194] Obtain the winding number of the sampling point for the real three-dimensional triangular mesh of the sample three-dimensional model and the unsigned distance parameter of the sampling point;

[0195] Determine the true signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point;

[0196] Calculating a second loss parameter of the initial three-dimensional generation model according to the predicted signed distance parameter of the sampling point and the true signed distance parameter of the sampling point;

[0197] According to the first loss parameter and the second loss parameter, the initial three-dimensional generation model is adjusted to obtain the target three-dimensional generation model.

[0198] In a feasible implementation manner, when the processor 1401 determines the real signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point, the processor 1401 is specifically configured to:

[0199] Determine the sign of the sampling point based on the winding number of the sampling point;

[0200] The true signed distance parameter of the sampling point is determined according to the sign of the sampling point and the unsigned distance parameter of the sampling point.

[0201] In a feasible implementation, when determining the sign of a sampling point according to the winding number of the sampling point, the processor 1401 is specifically configured to:

[0202] If the winding number of the sampling point is less than the preset parameter threshold, the sign of the sampling point is determined to be positive;

[0203] If the winding number of the sampling point is greater than or equal to the preset parameter threshold, the sign of the sampling point is determined to be negative.

[0204] In one feasible implementation, the initial 3D generation model includes: a 2D feature encoder, a 3D feature decoder, a first prediction network, and a second prediction network. When the processor 1401 generates predicted rendered images from multiple perspectives using the initial 3D generation model based on 2D reference images from multiple perspectives, as well as predicted signed distance parameters of sampling points in the 3D space corresponding to the sample 3D model, the processor 1401 is specifically configured to:

[0205] Based on the two-dimensional reference images under multiple viewing angles, a two-dimensional feature encoder is used to perform feature encoding to obtain two-dimensional image features under multiple viewing angles;

[0206] According to the two-dimensional image features of multiple perspectives, a three-dimensional feature decoder is used to decode the features and obtain three-dimensional spatial features;

[0207] According to the three-dimensional spatial characteristics, the sampling points are subjected to feature interpolation processing to obtain the interpolation features of the sampling points;

[0208] According to the interpolation characteristics of the sampling points, a first prediction network is used to perform prediction to obtain the predicted signed distance parameters of the sampling points;

[0209] According to the interpolation characteristics of the sampling points, the second prediction network is used to predict and obtain the rendering color of the sampling points;

[0210] Rendering is performed at multiple viewing angles according to the predicted signed distance parameters of the sampling points and the rendering colors of the sampling points to obtain predicted rendered images at multiple viewing angles.

[0211] In one feasible implementation, the three-dimensional spatial features include: spatial features in multiple directions, where the spatial features in each direction are: the product of the feature vector in each direction and the feature matrix of the orthogonal plane corresponding to each direction. When the processor 1401 performs feature interpolation processing on the sampling points based on the three-dimensional spatial features to obtain the interpolation features of the sampling points, it is specifically configured to:

[0212] The sampling points are projected onto the eigenvectors in each direction and the eigenmatrix of the corresponding orthogonal plane for interpolation processing to obtain the interpolation features of the sampling points in each direction and the interpolation features of the corresponding orthogonal plane;

[0213] Multiply the interpolation features of the sampling point in each direction with the interpolation features of the corresponding orthogonal plane to obtain the interpolation features of the sampling point in each direction;

[0214] The interpolation features of the sampling points in multiple directions are spliced to obtain the interpolation features of the sampling points.

[0215] In one feasible implementation, the three-dimensional feature decoder includes: a vector feature decoder, a plane feature decoder, a vector feature linear layer, a plane feature linear layer, and a feature output layer. When the processor 1401 performs feature decoding based on the two-dimensional image features of multiple perspectives using the three-dimensional feature decoder to obtain three-dimensional spatial features, it is specifically configured to:

[0216] Based on the preset initial position codes in multiple directions and the two-dimensional image features of multiple perspectives, a vector feature decoder is used to perform feature decoding to generate initial feature vectors in multiple directions;

[0217] Based on the preset initial position codes of multiple planes and the two-dimensional image features of multiple perspectives, a plane feature decoder is used to perform feature decoding to generate initial feature matrices of multiple planes;

[0218] The vector feature linear layer is used to perform linear processing on the initial feature vectors in multiple directions to obtain feature vectors in multiple directions;

[0219] The plane feature linear layer is used to perform linear processing on the initial feature matrices of multiple planes to obtain feature matrices in multiple directions;

[0220] The feature output layer is used to multiply the feature vector in each direction with the feature matrix of the orthogonal plane corresponding to each direction to obtain the spatial features in each direction.

[0221] In a feasible implementation, the processor 1401, when performing rendering at multiple viewing angles based on the predicted signed distance parameters of the sampling points and the rendering colors of the sampling points to obtain predicted rendered images at multiple viewing angles, is specifically configured to:

[0222] According to the predicted signed distance parameters of the sampling points, the surface mesh is extracted to obtain the initial triangular mesh;

[0223] The first prediction network is used to predict the initial triangular mesh to obtain the signed distance parameter and normal vector of each vertex on the initial triangular mesh;

[0224] Project the initial triangular mesh according to the signed distance parameter and normal vector of each vertex to obtain the projected triangular mesh;

[0225] According to the rendering color of the sampling point, the projected triangular mesh is rendered at multiple viewing angles to obtain predicted rendering images at multiple viewing angles.

[0226] In a feasible embodiment, the real rendered image at each viewing angle includes: a real color image, a real binary mask image, a real depth map, and a real normal map; the predicted rendered image at each viewing angle includes: a predicted color image, a predicted binary mask image, a predicted depth map, and a predicted normal map. The processor 1401, when executing the calculation of the first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images at multiple viewing angles, may include:

[0227] According to the true color image, true binary mask image, true depth map and true normal map at each viewpoint, as well as the predicted color image, predicted binary mask image, predicted depth map and predicted normal map at each viewpoint, the color error, binary mask error, depth error and normal error at each viewpoint are calculated respectively;

[0228] A first loss parameter is calculated based on color errors, binary mask errors, depth errors, and normal errors under multiple viewing angles.

[0229] When a computer device runs a three-dimensional generation method as described in the embodiment, the processor 1401 communicates with the storage medium 1402 via the bus 1403, and the processor 1401 executes the machine-readable instructions to perform the following steps:

[0230] Obtain two-dimensional reference images of the target object at multiple perspectives;

[0231] Generate predicted signed distance parameters for each sampling point in the target three-dimensional space corresponding to the target object using a pre-trained target three-dimensional generative model based on two-dimensional reference images from multiple perspectives, wherein the target three-dimensional generative model is a generative model obtained using the model training method described in any of the above embodiments;

[0232] Surface extraction is performed based on the predicted signed distance parameters of each sampling point to obtain a three-dimensional model of the target object.

[0233] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor when it is running, and the processor executes the steps of any of the above-mentioned model training methods or three-dimensional generation methods.

[0234] An embodiment of the present application also provides a computer program product, which includes program instructions. When the program instructions are executed by a processor, the steps of any of the above-mentioned model training methods or three-dimensional generation methods are executed.

[0235] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0236] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0238] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0239] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0240] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0241] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: include: Acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles; Generate, based on the two-dimensional reference images under the multiple viewing angles, predicted rendered images under the multiple viewing angles and predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model using an initial three-dimensional generation model; Calculating a first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images under the multiple perspectives; Obtaining the winding number of the sampling point with respect to the real three-dimensional triangular mesh of the sample three-dimensional model and the unsigned distance parameter of the sampling point; determining a true signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point; Calculating a second loss parameter of the initial three-dimensional generation model according to the predicted signed distance parameter of the sampling point and the true signed distance parameter of the sampling point; The initial three-dimensional generation model is adjusted according to the first loss parameter and the second loss parameter to obtain a target three-dimensional generation model.

2. The method according to claim 1, characterized in that Determining the true signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point includes: Determining the sign of the sampling point according to the winding number of the sampling point; A true signed distance parameter of the sampling point is determined according to the sign of the sampling point and the unsigned distance parameter of the sampling point.

3. The method according to claim 2, characterized in that The step of determining the sign of the sampling point according to the winding number of the sampling point includes: If the winding number of the sampling point is less than a preset parameter threshold, determining that the sign of the sampling point is positive; If the winding number of the sampling point is greater than or equal to the preset parameter threshold, the sign of the sampling point is determined to be negative.

4. The method according to claim 1, wherein The initial three-dimensional generation model includes: a two-dimensional feature encoder, a three-dimensional feature decoder, a first prediction network, and a second prediction network; The method of generating predicted rendered images under the multiple perspectives and predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model using an initial three-dimensional generation model based on the two-dimensional reference images under the multiple perspectives includes: Performing feature encoding using the two-dimensional feature encoder according to the two-dimensional reference images under the multiple viewing angles to obtain two-dimensional image features of the multiple viewing angles; According to the two-dimensional image features of the multiple perspectives, the three-dimensional feature decoder is used to perform feature decoding to obtain three-dimensional spatial features; Performing feature interpolation processing on the sampling points according to the three-dimensional spatial features to obtain interpolation features of the sampling points; According to the interpolation characteristics of the sampling points, the first prediction network is used to perform prediction to obtain a predicted signed distance parameter of the sampling points; According to the interpolation characteristics of the sampling point, the second prediction network is used to perform prediction to obtain the rendering color of the sampling point; Rendering is performed respectively under the multiple viewing angles according to the predicted signed distance parameters of the sampling points and the rendering colors of the sampling points to obtain predicted rendered images under the multiple viewing angles.

5. The method according to claim 4, characterized in that The three-dimensional spatial features include: spatial features in multiple directions, where the spatial features in each direction are: the product of the feature vector in each direction and the feature matrix of the orthogonal plane corresponding to each direction; The performing feature interpolation processing on the sampling points according to the three-dimensional spatial features to obtain interpolation features of the sampling points includes: Projecting the sampling points onto the feature vectors in each direction and the feature matrix of the corresponding orthogonal plane for interpolation processing, respectively, to obtain the interpolation features of the sampling points in each direction and the interpolation features of the corresponding orthogonal plane; Multiplying the interpolation feature of the sampling point in each direction and the interpolation feature of the corresponding orthogonal plane to obtain the interpolation feature of the sampling point in each direction; The interpolation features of the sampling point in the multiple directions are spliced to obtain the interpolation features of the sampling point.

6. The method according to claim 5, characterized in that The three-dimensional feature decoder includes: a vector feature decoder, a plane feature decoder, a vector feature linear layer, a plane feature linear layer and a feature output layer; The step of decoding the two-dimensional image features from the multiple viewing angles using the three-dimensional feature decoder to obtain three-dimensional spatial features includes: According to the preset initial position codes of the multiple directions and the two-dimensional image features of the multiple perspectives, the vector feature decoder is used to perform feature decoding to generate initial feature vectors of the multiple directions; According to the preset initial position codes of the multiple planes and the two-dimensional image features of the multiple perspectives, the plane feature decoder is used to perform feature decoding to generate initial feature matrices of the multiple planes; Using the vector feature linear layer to perform linear processing on the initial feature vectors in the multiple directions to obtain feature vectors in the multiple directions; Using the plane feature linear layer to perform linear processing on the initial feature matrices of the multiple planes to obtain feature matrices in the multiple directions; The feature output layer is used to perform a product operation on the feature vector in each direction and the feature matrix of the orthogonal plane corresponding to each direction to obtain the spatial feature in each direction.

7. The method according to claim 4, characterized in that The rendering is performed respectively at the multiple viewing angles according to the predicted signed distance parameter of the sampling point and the rendering color of the sampling point to obtain the predicted rendered images at the multiple viewing angles, including: Extracting a surface mesh according to the predicted signed distance parameters of the sampling points to obtain an initial triangular mesh; Using the first prediction network, predicting the initial triangular mesh to obtain a signed distance parameter and a normal vector of each vertex on the initial triangular mesh; Projecting the initial triangular mesh according to the signed distance parameters and normal vectors of each vertex to obtain a projected triangular mesh; The projected triangular mesh is rendered respectively at the multiple viewing angles according to the rendering colors of the sampling points to obtain predicted rendered images at the multiple viewing angles.

8. The method according to claim 1, characterized in that The real rendered image at each viewing angle includes: a real color image, a real binary mask image, a real depth map, and a real normal map; the predicted rendered image at each viewing angle includes: a predicted color image, a predicted binary mask image, a predicted depth map, and a predicted normal map; The calculating the first loss parameter of the initial 3D generation model according to the real rendered images and the predicted rendered images under the multiple perspectives includes: According to the true color image, true binary mask image, true depth map and true normal map at each viewing angle, and the predicted color image, predicted binary mask image, predicted depth map and predicted normal map at each viewing angle, respectively calculate the color error, binary mask error, depth error and normal error at each viewing angle; The first loss parameter is calculated according to the color error, binary mask error, depth error and normal error at each viewing angle.

9. A three-dimensional generation method, characterized in that: The method comprises: Obtain two-dimensional reference images of the target object at multiple perspectives; generating predicted signed distance parameters for each sampling point in the target three-dimensional space corresponding to the target object using a pre-trained target three-dimensional generative model based on the two-dimensional reference images from the multiple perspectives, wherein the target three-dimensional generative model is a generative model obtained using the model training method described in any one of claims 1 to 8; Surface extraction is performed according to the predicted signed distance parameters of each sampling point to obtain a three-dimensional model of the target object.

10. A model training device, characterized in that: include: A first acquisition module is used to acquire a sample three-dimensional model and two-dimensional reference images of the sample three-dimensional model at multiple viewing angles; A first generation module is configured to generate, based on the two-dimensional reference images under the multiple viewing angles, predicted rendered images under the multiple viewing angles and predicted signed distance parameters of sampling points in the three-dimensional space corresponding to the sample three-dimensional model using an initial three-dimensional generation model; A first calculation module is configured to calculate a first loss parameter of the initial 3D generation model based on the real rendered images and the predicted rendered images under the multiple perspectives; A second calculation module is used to obtain the winding number of the sampling point with respect to the real three-dimensional triangular mesh of the sample three-dimensional model and the unsigned distance parameter of the sampling point; determining a true signed distance parameter of the sampling point according to the winding number of the sampling point and the unsigned distance parameter of the sampling point; Calculating a second loss parameter of the initial three-dimensional generation model according to the predicted signed distance parameter of the sampling point and the true signed distance parameter of the sampling point; A training module is used to adjust the parameters of the initial three-dimensional generation model according to the first loss parameter and the second loss parameter to obtain a target three-dimensional generation model.

11. A three-dimensional generation device, characterized in that: include: A first acquisition module is used to acquire two-dimensional reference images of the target object at multiple viewing angles; a second generation module, configured to generate, based on the two-dimensional reference images from the multiple perspectives, predicted signed distance parameters for each sampling point in the target three-dimensional space corresponding to the target object using a pre-trained target three-dimensional generation model, wherein the target three-dimensional generation model is a generation model obtained using the model training method according to any one of claims 1 to 8; The extraction module is used to perform surface extraction according to the predicted signed distance parameters of each sampling point to obtain a three-dimensional model of the target object.

12. A computer device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the model training method according to any one of claims 1 to 8 or the three-dimensional generation method according to claim 9.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the model training method according to any one of claims 1 to 8 or the three-dimensional generation method according to claim 9.

Images