Model training method, data generation method, and device thereof

Abstract

The present disclosure provides a model training method, a data generation method and an apparatus and device thereof. The method comprises: obtaining a target training sample; the target training sample comprises: mesh data of an initial three-dimensional model, an initial reference image and sparse voxel reference features; inputting the initial noisy voxel and the initial reference image into a diffusion model to be trained to obtain first sparse voxel estimation features representing geometric information of the initial three-dimensional model and material information of an initial material of a reference three-dimensional model surface corresponding to a view angle of the initial reference image; obtaining a target loss value based on difference information between the sparse voxel reference features and the first sparse voxel estimation features; and training the diffusion model to be trained using the target loss value, so that the trained target diffusion model can output sparse voxel features representing the geometric information and the material information.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a model training method, a data generation method, and an apparatus or device thereof. Background Technology

[0002] With the development of 3D model reconstruction technology, many reconstruction schemes such as implicit volume optimization and explicit volume optimization have emerged in the field of 3D model reconstruction. However, existing reconstruction schemes still have the following problems: the generated material details are blurred and flickering due to the reliance on multi-view intermediate representation, and the implementation cost is high due to the lack of material information required for 3D model reconstruction, making it difficult to meet the high-quality requirements of 3D model reconstruction. Summary of the Invention

[0003] This disclosure provides a model training method, a data generation method, and an apparatus or device thereof to solve or alleviate one or more technical problems in the prior art.

[0004] Firstly, this disclosure provides a method for training generative models, including: Obtain target training samples; the target training samples include: mesh data of the initial 3D model, an initial reference image, and sparse voxel reference features; wherein, the sparse voxel reference features are obtained by encoding sparse voxels carrying information related to the geometry and initial material of the initial 3D model; the initial reference image is a 2D image of the reference 3D model with the initial material; the reference 3D model is structurally related to the initial 3D model; The mesh data of the initial 3D model is preprocessed to obtain initial noisy voxels; The initial noisy voxel and the initial reference image are input into the diffusion model to be trained to obtain the first sparse voxel estimation features that represent the geometric information of the initial 3D model and the material information of the initial material of the reference 3D model surface under the viewpoint corresponding to the initial reference image. Based on the difference information between the sparse voxel reference features and the first sparse voxel estimation features, the target loss value is obtained. The target loss value is used to train the diffusion model to be trained so that the trained target diffusion model can output sparse voxel features that represent geometric and material information.

[0005] Secondly, this disclosure provides a method for generating data of a three-dimensional model with materials, including: Obtain the mesh data and target reference image of the target 3D model; the target reference image is a 2D image of the reference 3D model with the target material; the reference 3D model is structurally related to the target 3D model. The mesh data of the target 3D model is preprocessed to obtain noisy target voxels; The target noisy voxel and the target reference image are input into the target diffusion model to obtain sparse voxel prediction features; wherein, the target diffusion model is trained using the model training method described above; The target sparse voxel is obtained by using the predicted features of the sparse voxel. Based on the target sparse voxel, at least one of the following is obtained: Mesh data of a target 3D model with the target material. A rendered image of a target 3D model with the target material.

[0006] Thirdly, this disclosure provides a model training apparatus for a generative model, comprising: The first acquisition unit is used to acquire target training samples; the target training samples include: mesh data of an initial 3D model, an initial reference image, and sparse voxel reference features; wherein, the sparse voxel reference features are obtained by encoding sparse voxels carrying information related to the geometry and initial material of the initial 3D model; the initial reference image is a 2D image of a reference 3D model with the initial material; the reference 3D model is structurally related to the initial 3D model; A data processing unit is used to preprocess the mesh data of the initial 3D model to obtain initial noisy voxels; input the initial noisy voxels and the initial reference image into the diffusion model to be trained to obtain a first sparse voxel estimation feature that represents the geometric information of the initial 3D model and the material information of the initial material of the reference 3D model surface from the viewpoint corresponding to the initial reference image; obtain a target loss value based on the difference information between the sparse voxel reference feature and the first sparse voxel estimation feature; and train the diffusion model to be trained using the target loss value so that the trained target diffusion model can output sparse voxel features that represent geometric and material information.

[0007] Fourthly, this disclosure provides a data generation apparatus for a three-dimensional model with materials, comprising: The second acquisition unit is used to acquire the mesh data of the target 3D model and the target reference image; the target reference image is a 2D image of the reference 3D model with the target material; the reference 3D model is structurally related to the target 3D model. A data generation unit is used to preprocess the mesh data of the target 3D model to obtain target noisy voxels; input the target noisy voxels and the target reference image into a target diffusion model to obtain sparse voxel prediction features; wherein the target diffusion model is trained using the above-mentioned model training method; using the obtained sparse voxel prediction features, target sparse voxels are obtained; based on the target sparse voxels, at least one of the following is obtained: mesh data of the target 3D model with target material, and a rendering effect image of the target 3D model with target material.

[0008] Fifthly, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0009] In a sixth aspect, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of the present disclosure.

[0010] In a seventh aspect, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of the present disclosure.

[0011] The beneficial effects of the technical solution provided in this disclosure include at least the following: Thus, this disclosed solution, using sparse voxels as the carriers of 3D attributes, trains the diffusion model to be trained using an initial reference image, sparse voxel reference features, and initial noisy voxels. This enables the trained diffusion model to output sparse voxel features representing both geometric and material information. This allows the model to more accurately understand and reconstruct material details. Furthermore, by using sparse voxels as a unified intermediate representation, this disclosed solution effectively avoids detail blurring and flickering caused by relying on multi-view intermediate representations, thereby improving the consistency of material generation across viewpoints and providing favorable support for the subsequent efficient reconstruction of 3D model materials.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments provided according to this disclosure and should not be construed as limiting the scope of this disclosure.

[0014] Figure 1 This is an illustrative flowchart of a generative model training method according to an embodiment of this application. Figure 1 ; Figure 2 This is an illustrative flowchart of a generative model training method according to an embodiment of this application. Figure 2 ; Figure 3 This is a schematic flowchart of a generative model training method according to an embodiment of this application in one example; Figure 4 This is a training illustration of an embodiment of the "encoding module-decoding module" according to this application. Figure 1 ; Figure 5 This is a training illustration of an embodiment of the "encoding module-decoding module" according to this application. Figure 2 ; Figure 6 This is a training illustration of an embodiment of the "encoding module-decoding module" according to this application. Figure 3 ; Figure 7 This is a schematic diagram of the first training phase of a diffusion model and an "encoding module-decoding module" according to an embodiment of this application; Figure 8 This is a schematic diagram of the second training phase of the diffusion model and the "encoding module-decoding module" according to an embodiment of this application; Figure 9 This is a schematic flowchart illustrating a method for generating data of a three-dimensional model with materials according to an embodiment of this application; Figure 10 This is a schematic diagram illustrating the process of generating data for a three-dimensional model with materials according to an embodiment of this application in one example; Figure 11 This is a schematic diagram of the structure of a model training device for a generative model according to an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a data generation device having a three-dimensional model of materials according to an embodiment of this application; Figure 13 This is a block diagram of an electronic device used to implement a generative model training method or a data generation method for a three-dimensional model with materials according to embodiments of the present disclosure. Detailed Implementation

[0015] The present disclosure will now be described in further detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0016] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0017] Solution A: Implicit volume optimization solution, the core logic of which is: Taking text or an image as input, this method optimizes the implicit volume (i.e., neural radiance fields (NeRF) or signed distance functions (SDF)) through score distillation to fit a 3D scene, and then obtains the target view through volume rendering. This allows for the generation of 3D models from natural language or single images, but this approach has high requirements for computational resources and training time.

[0018] Option B: A fast explicit representation scheme, whose core logic is as follows: This approach enables efficient reconstruction and real-time rendering by rapidly fitting lightweight structures (such as point clouds or Gaussian volumes) through multi-view supervision. Prioritizing speed and scalability, it is suitable for low-latency scenarios, such as real-time augmented reality (AR) or virtual reality (VR) modeling.

[0019] However, the above-mentioned 3D model reconstruction scheme still has significant limitations and cannot meet the high-quality requirements of 3D model reconstruction. Specifically, these limitations are as follows: (a) Long model training and inference time: When iterating and optimizing a single object, the 3D generation scheme based on score distillation sampling (SDS) has a long optimization time and is difficult to generate quickly.

[0020] (b) Difficulty in material generation consistency: Using multiple views as an intermediate representation can easily lead to accumulated errors during the mapping process, resulting in blurred details, flickering, and inconsistency across viewpoints in the reconstructed materials.

[0021] (c) Semantic and style alignment is unstable: under textual or weak image conditions, detail style is prone to deviate from structural semantics.

[0022] (d) High implementation cost: The generated 3D model lacks the complete Level of Detail (LOD), U-axis-V-axis mapping and material information required for cross-engine or pipeline docking, resulting in high implementation cost.

[0023] Based on this, the present disclosure provides a model training method. This method uses sparse voxels (SVs) as the 3D attribute carriers and utilizes an initial reference image, sparse voxel reference features, and initial noisy voxels of the initial 3D model to train the diffusion model to be trained. This enables the trained diffusion model to output sparse voxel features representing geometric and material information. This effectively solves the problem of accumulated errors caused by relying on intermediate representations from multiple views during material generation, thus fundamentally avoiding the problems of detail blurring and flickering, and improving the consistency of material generation across different viewpoints. Simultaneously, the present disclosure significantly reduces the difficulty of decoupling complex materials from the reference 3D model through implicit learning of sparse voxel features, enabling the model to transfer decoupled complex materials to the initial 3D model, thereby improving the model's generalization ability and laying the foundation for subsequent high-resolution generation of 3D models with materials.

[0024] Specifically, Figure 1 This is an illustrative flowchart of a generative model training method according to an embodiment of this application. Figure 1 This method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0025] Furthermore, the method includes at least a portion of the following: For example... Figure 1 As shown, it includes: Step S101: Obtain the target training sample.

[0026] Here, the target training samples include: the mesh data of the initial 3D model (also known as the mesh of the initial 3D model), the initial reference image, and the sparse voxel reference features.

[0027] Furthermore, the sparse voxel reference feature is obtained by encoding sparse voxels carrying information related to the geometry of the initial 3D model and information related to the initial material.

[0028] Here, the sparse voxel reference features specifically refer to the reference data of sparse voxel features. In other words, under the premise that sparse voxels are used as the carriers of three-dimensional attributes, the present disclosure uses the reference data of the sparse voxel features corresponding to the initial three-dimensional model with initial material as the label data when training the diffusion model, which facilitates the subsequent calculation of the training loss of the diffusion model.

[0029] Furthermore, the initial reference image is a two-dimensional image of a reference 3D model having the initial material, and the structure of the reference 3D model in the two-dimensional image is related to that of the initial 3D model. For example, in one example, the structure of the reference 3D model in the initial reference image is similar to that of the initial 3D model, or, further, the structure of the reference 3D model in the initial reference image is the same as that of the initial 3D model. In this way, prior structural information of the 3D model is provided for the subsequent material diffusion of the model, thereby fully learning the distribution of the material on the initial 3D model.

[0030] Furthermore, the target training samples of this disclosed solution, in addition to including the initial reference image of the reference 3D model with the initial material in a single view, may also include the initial reference images of the reference 3D model with the initial material in each of N (N is an integer greater than 1) viewpoints. In this way, the direction of model generation can be more clearly constrained, and the consistency between the generated results and user expectations can be improved.

[0031] Furthermore, in one example, the target training sample may also include initial prompt text; furthermore, the initial prompt text is used to describe the geometry and / or material of the initial 3D model, such as describing the appearance, structure, and material of the initial 3D model, thus providing sufficient prior information for the subsequent guidance model to generate the required materials.

[0032] Step S102: Preprocess the mesh data of the initial 3D model to obtain the initial noisy voxels.

[0033] Here, the initial noisy voxel can specifically be a sparse voxel carrying noise.

[0034] For example, in a specific instance, the preprocessing of the mesh data of the initial 3D model described above to obtain initial noisy voxels may specifically include: Step S102-1: Perform voxelization on the mesh data of the initial three-dimensional model to obtain sparse voxels of the initial three-dimensional model.

[0035] Step S102-2: Add noise to the sparse voxels of the initial three-dimensional model to obtain the initial noisy voxels.

[0036] In other words, after obtaining the mesh data of the initial 3D model, the mesh data of the initial 3D model is first processed into voxels to obtain sparse voxels of the initial 3D model; then the sparse voxels of the initial 3D model are processed with noise, for example, random noise is used to process the sparse voxels of the initial 3D model to obtain noisy sparse voxels, that is, the initial noisy voxels are obtained.

[0037] It should be noted that, in this example, the sparse voxels of the initial 3D model refer to the non-empty set of voxels in the voxel representation of the initial 3D model, which consists of all voxel meshes containing geometric or attribute information.

[0038] Thus, this disclosed solution, through sparse voxelization, can effectively solve the problems of high resource consumption and low computational efficiency caused by irregular grid data and complex topology, thereby providing favorable support for improving the training efficiency of the diffusion model. Furthermore, by adding noise to sparse voxels, a progressive noise perturbation that conforms to the principle of the diffusion model is constructed, which lays the foundation for the subsequent model to learn the stable reconstruction process from noisy voxels to clear 3D structures and improve the reconstruction capability of the 3D model.

[0039] Step S103: Input the initial noisy voxel and the initial reference image into the diffusion model to be trained to obtain the first sparse voxel estimation features that represent the geometric information of the initial three-dimensional model and the material information of the initial material of the reference three-dimensional model surface from the viewpoint corresponding to the initial reference image.

[0040] It should be noted that, under the training paradigm of the diffusion model, the diffusion model takes the initial noisy voxels and the initial reference image as input to estimate the added noise (which can be called estimated noise). Then, the estimated noise is used to denoise the initial noisy voxels. Under the constraints of the geometric information of the reference 3D model in the viewpoint corresponding to the initial reference image and the material information of the initial material in the corresponding viewpoint, the first sparse voxel estimation feature corresponding to the initial 3D model is obtained by denoising.

[0041] Furthermore, the first sparse voxel estimated feature specifically refers to the estimated value of the first sparse voxel feature. That is, under the premise that the sparse voxel is used as the three-dimensional attribute carrier, the present invention uses the diffusion model to be trained to perform denoising, and obtains the estimated value of the feature attribute corresponding to the denoised sparse voxel, which is the estimated value of the first sparse voxel feature.

[0042] Step S104: Based on the difference information between the sparse voxel reference features and the first sparse voxel estimated features, obtain the target loss value.

[0043] In other words, the sparse voxel reference features are used as the label data, and the obtained first sparse voxel estimated features are used as the estimated values ​​output by the model. Then, the target loss value is obtained based on the difference information between the two.

[0044] Step S105: Use the target loss value to train the diffusion model to be trained so that the trained target diffusion model can output sparse voxel features that represent geometric and material information.

[0045] Here, the trained target diffusion model can output sparse voxel features that represent geometric and material information. For example, in one example, the output is a sparse voxel feature that integrates geometric and material features.

[0046] In other words, the present invention uses an initial reference image and initial noisy voxels of an initial 3D model as inputs into the diffusion model to be trained to obtain a first sparse voxel estimation feature that characterizes the geometric information of the initial 3D model and the material information of the initial material of the reference 3D model surface from the viewpoint corresponding to the initial reference image. At this time, the sparse voxel reference feature is used as the label data when training the diffusion model. Then, based on the difference between the sparse voxel reference feature and the first sparse voxel estimation feature, the target loss value is obtained. Then, the obtained target loss value is used to fine-tune the parameters of the diffusion model to be trained to obtain the trained diffusion model.

[0047] Thus, this disclosed solution, using sparse voxels as the carriers of 3D attributes, trains the diffusion model to be trained using an initial reference image, sparse voxel reference features, and initial noisy voxels. This enables the trained diffusion model to output sparse voxel features representing both geometric and material information. This allows the model to more accurately understand and reconstruct material details. Furthermore, by using sparse voxels as a unified intermediate representation, this disclosed solution effectively avoids detail blurring and flickering caused by relying on multi-view intermediate representations, thereby improving the consistency of material generation across viewpoints and providing favorable support for the subsequent efficient reconstruction of 3D model materials.

[0048] Moreover, the proposed solution can accept not only single-view reference images as input during the learning process, but also multi-view reference images. The input of multi-view reference images can supplement geometric and material information from different viewing angles. This can more clearly constrain the direction of model generation, thereby improving the controllability of reconstructing a 3D model with materials, making the generated 3D model more complete and stable, and thus improving the consistency between the generated results and user expectations.

[0049] Furthermore, the disclosed scheme can utilize the difference information between the sparse voxel reference features and the first sparse voxel estimated features to obtain the target loss value, which is then used to train the diffusion model to be trained. Under the premise of using sparse voxels as a unified intermediate representation, the above training scheme significantly reduces the complexity of model training, thereby reducing the time and computing resources required for model training. In this way, the training cost of the model is greatly saved, while improving the processing efficiency of the model. This provides favorable support for the subsequent efficient 3D reconstruction using the trained diffusion model.

[0050] Figure 2This is an illustrative flowchart of a generative model training method according to an embodiment of this application. Figure 2 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that... Figure 1 The methods shown can also be applied to this example, and the related content will not be elaborated further in this example.

[0051] Furthermore, the method includes at least a portion of the following: For example... Figure 2 As shown, it includes: Step S201: Obtain the first sparse voxel.

[0052] Here, the first sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material, such as geometric information of the initial 3D model and material information of the initial material.

[0053] Furthermore, in one example, the first sparse voxel can be obtained in the following manner; specifically, the above-described method of obtaining the first sparse voxel can specifically include one of the following methods: Method 1: Using the mesh data of the initial 3D model, perform inverse rendering on N initial model images to obtain the first sparse voxel.

[0054] Here, the initial model image is a two-dimensional image of the initial three-dimensional model with the initial material; the initial three-dimensional model has different perspectives in different initial model images.

[0055] Furthermore, the N initial model images can be specifically the model images of the initial 3D model from each of the N viewpoints, such as the rendered images of the initial 3D model with initial materials.

[0056] It should be noted that the "inverse rendering" described in this disclosure is the reverse process of traditional "rendering" technology. Specifically, traditional "rendering" aims to generate a two-dimensional image based on a known three-dimensional scene, while inverse rendering reverses and recovers the inherent physical properties of a three-dimensional scene from one or more two-dimensional images. These inherent physical properties include: the geometry (such as meshes and depth), materials (such as textures), and ambient lighting conditions (such as shadows) of objects in the three-dimensional scene. Therefore, the core objective of "inverse rendering" is to decouple and separate the geometric and material information coupled to image pixels, thereby extracting highly usable digital three-dimensional assets for subsequent editing and physical simulation.

[0057] At this point, the sparse voxels obtained by inverse rendering the model images from N perspectives carry information about the geometry of the initial 3D model and information about the initial material from each of the N perspectives, such as the geometric information of the geometry and the material information of the initial material.

[0058] In this way, the disclosed solution uses inverse rendering to extract the relevant information of the geometry and the relevant information of the initial 3D model from each viewpoint from N initial model images, and represents them in the form of sparse voxels. This effectively reduces the dependence on the intermediate representation of multiple views, and provides favorable support for improving the consistency of material generation across viewpoints when using diffusion models to reconstruct 3D models.

[0059] Method 2: Sampling point clouds is performed on each triangular face represented by the mesh data of the initial 3D model carrying the initial material to obtain multiple material point clouds. Spatial feature processing is performed using the multiple material point clouds to obtain the first sparse voxel.

[0060] For example, in one example, point cloud sampling is first performed on each triangular face represented by the mesh data of the initial 3D model carrying the initial material to obtain multiple material point clouds; then, the multiple material point clouds are mapped to the 3D voxel mesh space, and the first sparse voxel is obtained according to the voxel mesh with material point clouds falling in the 3D voxel mesh space.

[0061] Here, in the example above, when there is a voxel grid in the three-dimensional voxel grid space where multiple material point clouds fall, the multiple material point clouds falling in the voxel grid can be aggregated (e.g., averaged) to obtain the voxel grid after material point cloud aggregation. At this time, the first sparse voxel is obtained based on the voxel grid after material point cloud aggregation.

[0062] Thus, this disclosed solution provides two different voxel transformation schemes to extract sparse voxels from different datasets. Specifically, for public datasets where material information cannot be directly obtained (such as N initial model images), inverse rendering can be used to optimize and obtain the first sparse voxel. For special datasets where material information can be directly obtained (such as mesh data of initial 3D models carrying initial materials), point cloud sampling is used to obtain the first sparse voxel. This effectively breaks the limitations of a single data source for model training, achieves the extraction of unified feature representations from heterogeneous datasets, and obtains a unified sparse voxel representation of geometric and material information. This provides high-quality and diverse sample data for subsequent diffusion model training, thereby improving the stability and robustness of model training.

[0063] Furthermore, since the present invention can extract voxel data (such as the first sparse voxel) from the mesh data of the initial 3D model and then convert the dataset into a unified voxel intermediate representation, the model training process can be carried out in the native voxel space, thereby reducing the loss caused by data construction and providing favorable support for improving the stability and robustness of model training.

[0064] Step S202: Using the trained encoding module, perform feature processing on the first sparse voxel to obtain sparse voxel reference features.

[0065] It should be noted that, using the trained encoding module, the high-resolution, low-dimensional first sparse voxel can be encoded into a low-resolution, high-dimensional sparse voxel reference feature.

[0066] Step S203: Obtain the target training sample.

[0067] Here, the target training samples include: the grid data of the initial 3D model, the initial reference image, and the sparse voxel reference features.

[0068] Furthermore, the sparse voxel reference feature is obtained by encoding sparse voxels carrying information related to the geometry of the initial 3D model and information related to the initial material; the initial reference image is a 2D image of the reference 3D model with the initial material; the reference 3D model is structurally related to the initial 3D model.

[0069] It should be noted that the relevant explanations regarding sparse voxel reference features and initial reference images can be found in the examples above, and will not be repeated here.

[0070] Step S204: Preprocess the mesh data of the initial 3D model to obtain the initial noisy voxels.

[0071] It should be noted that the relevant content of step S204 can be referred to the relevant example of step S102 above, and will not be repeated here.

[0072] Step S205: Input the initial noisy voxel and the initial reference image into the diffusion model to be trained to obtain the first sparse voxel estimation features that represent the geometric information of the initial three-dimensional model and the material information of the initial material of the reference three-dimensional model surface from the viewpoint corresponding to the initial reference image.

[0073] Here, in one example, the diffusion model may specifically be a sparse flow transformer model, or it may be a diffusion transformer (DiT), etc. This disclosure does not impose any specific limitations on this.

[0074] Step S206: Based on the difference information between the sparse voxel reference features and the first sparse voxel estimated features, obtain the target loss value.

[0075] Step S207: Use the target loss value to train the diffusion model to be trained so that the trained target diffusion model can output sparse voxel features that represent geometric and material information.

[0076] That is to say, in one example, such as Figure 3 As shown, before constructing the target training samples, the first sparse voxel corresponding to the initial 3D model is obtained, and the first sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material. Further, the first sparse voxel is input into the trained encoding module to obtain sparse voxel reference features, which are used as label data when training the diffusion model. In other words, the sparse voxel reference features, the mesh data of the initial 3D model, and the initial reference image can be used as target training samples for model training. Further, the mesh of the initial 3D model in the target training samples is voxelized and noise-added to obtain initial noisy voxels, and the obtained initial noisy voxels and the initial reference image are input into the sparse flow matching model to be trained to obtain the first sparse voxel estimated features. At this time, the target loss value is obtained by using the difference information between the sparse voxel reference features and the first sparse voxel estimated features, and the obtained target loss value is used to fine-tune the parameters of the sparse flow matching model to be trained to obtain the trained sparse flow matching model.

[0077] In this way, the disclosed solution can obtain sparse voxel reference features by using a first sparse voxel carrying information about the geometry of the initial 3D model and information about the initial material as input to the encoder. The sparse voxel reference features are then used as label data for training the diffusion model. This provides a clear optimization direction for the training of the diffusion model, thereby significantly improving the training stability and efficiency of the diffusion model in reconstructing 3D model materials, and enhancing the realism and controllability of the materials generated on the 3D model. At the same time, it effectively reduces the dependence on high-density voxels or complete sample data, thus significantly reducing the training cost of the diffusion model and further enhancing its adaptability in complex scenes.

[0078] Furthermore, the disclosed solution uses sparse voxel reference features and the first sparse voxel estimation features output by the diffusion model to calculate the target loss value, and uses the target loss value to optimize the diffusion model. In this way, geometric information and material information can be effectively integrated, thereby improving the accuracy and robustness of the diffusion model in expressing sparse voxel features. This provides favorable support for the subsequent efficient material reconstruction of the 3D model and the improvement of the fidelity and detail of the generated material.

[0079] In addition, when applying the solution disclosed herein, the diffusion model used can specifically adopt a sparse flow matching model. In this way, through the inverse integration capability of the sparse flow matching model, high-quality 3D content can be generated efficiently, thereby further improving the high fidelity of 3D models with materials.

[0080] Furthermore, in a specific example, the trained encoding module can be obtained in the following manner; for example, in one example, the method further includes: Step 1: Obtain the second sparse voxel.

[0081] Here, the second sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material, such as geometric information of the initial 3D model and material information of the initial material.

[0082] It should be noted that, in one example, the second sparse voxel and the first sparse voxel can be the same sparse voxel. That is, the second sparse voxel used by the training encoding module of this disclosure can also be used to train the diffusion model, for example, to obtain the label data (i.e., sparse voxel reference features) required for training the diffusion model.

[0083] Alternatively, in another example, the second sparse voxel and the first sparse voxel can be two different sparse voxels. For example, two sparse voxels sampled from different datasets (such as public datasets and special datasets) for an initial 3D model with initial materials.

[0084] Here, the method for obtaining the second sparse voxel is similar to the method for obtaining the first sparse voxel described above. For the method of obtaining this part, please refer to the example above, and it will not be repeated here.

[0085] Step 2: Input the second sparse voxel into the encoding module to be trained to encode the first sparse voxel encoding feature.

[0086] Here, the first sparse voxel encoding feature represents the combined information of the geometric information of the initial 3D model and the material information of the initial material.

[0087] Step 3: Input the first sparse voxel encoding feature into the decoding module to decode and obtain the estimated sparse voxel carrying the relevant information (or geometric information) of the geometry of the initial 3D model and the relevant information (or material information) of the initial material.

[0088] Step 4: Based on the estimated sparse voxels, obtain the first loss value.

[0089] In other words, the reconstructed sparse voxel corresponding to the second sparse voxel is used as the label data, and the obtained estimated sparse voxel is used as the estimated value output by the editing module. Then, the first loss value is obtained based on the difference information between the two.

[0090] Step 5: Using the first loss value, train the model on the encoding module to be trained to obtain the trained encoding module.

[0091] That is to say, in one example, such as Figure 4 As shown, the high-resolution (e.g., 512) low-dimensional second sparse voxels are first input into the encoding module to be trained to obtain the low-resolution (e.g., 64) high-dimensional first sparse voxel encoding features. Then, the first sparse voxel encoding features are input into the decoding module to obtain the high-resolution (e.g., 512) low-dimensional estimated sparse voxels. Using a loss function, such as the mean squared error loss function, and combining the obtained estimated sparse voxels with the reconstructed high-resolution (e.g., 512) low-dimensional sparse voxels, the first loss value is calculated. Finally, the obtained first loss value is used to fine-tune the parameters of the encoding module to be trained to obtain the trained encoding module.

[0092] In this way, the proposed solution constructs a closed-loop self-supervised learning circuit of "encoding module-decoding module". It uses the estimated sparse voxel and second sparse voxel label data output by the decoding module to calculate the loss and optimize the parameters of the encoding module in reverse. This enables the encoding module to extract high-dimensional sparse voxel features that contain both geometric and material information, effectively solving the problem of insufficient sparse voxel feature representation. This provides a clear optimization direction for the subsequent training of the diffusion model and thus provides favorable support for improving the realism and detail of the generated materials.

[0093] Furthermore, in a specific example, model training can be performed as follows; specifically, the above-described method of using the first loss value to train the encoding module to obtain the trained encoding module (e.g., step 5) can specifically include: Using the first loss value, the decoding module and the encoding module to be trained are trained to obtain the trained encoding module and the trained decoding module.

[0094] Continue with Figure 4 Taking the model training shown as an example, as Figure 5 As shown, after obtaining the first loss value, the parameters of the decoding module and the encoding module to be trained can be fine-tuned according to the first loss value to obtain the trained encoding module and the trained decoding module.

[0095] In one example, the "encoding module-decoding module" in this disclosure can be specifically a variational autoencoder (VAE), or it can be other autoencoders, etc. This disclosure does not impose any specific limitations on this.

[0096] In this way, the disclosed scheme can use the same loss value to fine-tune the parameters of the encoder and decoder end-to-end, thereby effectively improving the modeling ability of the "encoding module-decoding module" for geometric and material information. Moreover, the above training process not only enhances the ability of the encoding module to extract highly discriminative sparse voxel features, which provides more reliable and physically consistent label data for subsequent training of the diffusion model, but also improves the accuracy of the decoding module in restoring material details, reduces reconstruction errors and artifacts, and provides favorable support for improving the realism and detail of the generated materials in the future.

[0097] Furthermore, in a specific example, the encoding module and the decoding module are structurally symmetrical isomorphic networks.

[0098] Here, the "isomorphic network" referred to in this example means a neural network that is similar or identical in topology, such as the type and number of network layers, the arrangement of neurons, and the parameter configuration. In short, it means a network with a similar or identical skeleton.

[0099] Furthermore, in one example, the encoding module includes a sparse convolutional encoding layer and a first sparse spatial attention layer. The sparse convolutional encoding layer is used to extract geometric information of the geometric shape and material information of the initial material from the input sparse voxels (such as the second sparse voxel) to obtain high-dimensional features representing the geometric and material information. The first sparse spatial attention layer is used to enhance the key information in the high-dimensional features while suppressing redundant information in the high-dimensional features to encode sparse voxel features (such as the first sparse voxel encoded features). In this way, high-resolution voxel data can be effectively encoded and processed through the above network layers.

[0100] Furthermore, in another example, the decoding module includes a second sparse spatial attention layer and a sparse convolutional decoding layer. The second sparse spatial attention layer is used to reconstruct the material information in the input sparse voxel features (such as the first sparse voxel encoded features) to be globally distributed in three-dimensional space, so as to obtain the reconstructed sparse voxel features in the high-dimensional latent space. The sparse convolutional decoding layer is used to restore the sparse voxel features in the high-dimensional latent space and obtain low-dimensional features that characterize the geometric information of the geometric shape and the material information of the initial material, so as to estimate the sparse voxels (such as estimating sparse voxels).

[0101] It should be noted that, in one example, the encoding module in the "encoding module and decoding module" of this disclosure may specifically adopt the example structure of the encoding module including the sparse convolutional coding layer and the first sparse spatial attention layer, and the decoding module may specifically be other modules with decoding capabilities; or, in another example, the encoding module in the "encoding module and decoding module" of this disclosure may specifically be other modules with encoding capabilities, and the decoding module may specifically adopt the example structure of the decoding module including the second sparse spatial attention layer and the sparse convolutional decoding layer; or, in yet another example, the encoding module in the "encoding module and decoding module" of this disclosure may specifically adopt the example structure of the encoding module including the sparse convolutional coding layer and the first sparse spatial attention layer, and the decoding module may specifically adopt the example structure of the decoding module including the second sparse spatial attention layer and the sparse convolutional decoding layer.

[0102] Continue with Figure 5 Taking the training of the encoding module shown as an example, as Figure 6 As shown, the encoding module includes a sparse convolutional encoding layer and a first sparse spatial attention layer connected in series, and the decoding module includes a second sparse spatial attention layer and a sparse convolutional decoding layer connected in series. The encoding module and the decoding module are structurally symmetrical isomorphic networks. For example, the sparse convolutional encoding layer and the sparse convolutional decoding layer are symmetrically arranged, and the first sparse spatial attention layer and the second sparse spatial attention layer are symmetrically arranged. The above-mentioned symmetrical arrangement ensures that the encoder and decoder maintain consistency in structural expressive power, thereby effectively improving the compression, reconstruction and generation quality of voxel data.

[0103] Furthermore, in one example, such as Figure 6As shown, a high-resolution (e.g., 512) low-dimensional second sparse voxel is input into the sparse convolutional coding layer in the coding module to be trained, so as to extract the geometric information of the geometry and the material information of the initial material in the second sparse voxel, thereby obtaining low-resolution high-dimensional features representing the geometric and material information; further, the obtained low-resolution high-dimensional features are input into the first sparse spatial attention layer in the coding module to be trained, so as to enhance the key information in the low-resolution high-dimensional features and suppress the redundant information in the high-dimensional features, thereby obtaining low-resolution (e.g., 64) high-dimensional first sparse voxel coding features.

[0104] Furthermore, the low-resolution, high-dimensional first sparse voxel encoded features are input into the second sparse space attention layer in the decoding module to reconstruct the material information in the first sparse voxel encoded features as a global distribution in three-dimensional space, thereby obtaining the reconstructed sparse voxel features in the high-dimensional latent space. Furthermore, the reconstructed sparse voxel features in the high-dimensional latent space are input into the sparse convolution decoding layer in the decoding module to restore the sparse voxel features in the high-dimensional latent space and obtain high-resolution (e.g., 512) low-dimensional features that represent geometric information of geometric shape and material information of initial material, so as to output high-resolution (e.g., 512) low-dimensional estimated sparse voxels.

[0105] It should be noted that in practical applications, given the large amount of GPU memory consumed by the "encoding-decoding module" when processing high-resolution 3D data (such as second sparse voxels), this disclosed solution employs a voxel grid pruning technique during the training phase. Specifically, the high-resolution second sparse voxels are randomly pruned to obtain multiple equal-sized voxel blocks. One equal-sized voxel block is then randomly selected from these blocks to train the "encoding-decoding module," and model training is completed by traversing all equal-sized voxel blocks. This limits the voxel size during a single training iteration, reducing GPU memory usage and overcoming the memory bottleneck caused by the cubic increase in 3D data resolution. This allows for model training using higher-resolution 3D voxels. Furthermore, the aforementioned equal-sized voxel blocks enhance the model's spatial data, effectively improving its generalization ability to local geometry and material structures, as well as its reasoning ability from local features to the overall structure.

[0106] Thus, the "encoding module-decoding module" in this disclosed solution adopts a symmetrical three-dimensional sparse convolutional structure, which can compress sparse voxel data into a low-dimensional latent space and reconstruct the original data, thereby realizing the compression and reconstruction of geometric and material information. This improves the processing efficiency of the encoding module for high-resolution three-dimensional content, thus providing favorable support for improving the training efficiency and training stability of the diffusion module. At the same time, this disclosed solution also effectively improves the accuracy and efficiency of the decoding module in restoring material details, providing favorable support for subsequent high-precision material reconstruction of three-dimensional models using the trained diffusion and decoding modules.

[0107] Furthermore, since the present invention introduces sparse convolutional structures (such as sparse convolutional coding layers and sparse convolutional decoding layers) into the "encoding module-decoding module", it can effectively handle the distribution of high-resolution voxel data in sparse space, thereby reducing the amount of computation and memory usage during processing. This enables the model to train and generate higher-resolution voxel data. At the same time, it also enables the training and generation of high-resolution voxel data to run efficiently on resource-constrained devices, thereby optimizing computational efficiency and computational resources.

[0108] Figure 7 This is a schematic diagram of the diffusion model and the two-stage training of the "encoding module-decoding module" in this disclosed scheme. For example... Figure 7 As shown, in the "encoding module-decoding module" of this disclosure, the encoding module includes a sparse convolutional encoding layer and a first sparse spatial attention layer, and the decoding module includes a second sparse spatial attention layer and a sparse convolutional decoding layer. The sparse convolutional encoding layer and the sparse convolutional decoding layer are symmetrically arranged, as are the first sparse spatial attention layer and the second sparse spatial attention layer.

[0109] Furthermore, this disclosed scheme employs a two-stage training method, training the "encoding module-decoding module" and the diffusion module separately; specifically, as follows... Figure 7As shown, in the first training phase, the second sparse voxels are first obtained. Here, for public datasets where materials cannot be directly obtained (such as model images from N perspectives), the mesh data of the initial 3D model can be used to perform inverse rendering on the multi-view model images to obtain the second sparse voxels. For special datasets where materials can be directly obtained (containing the mesh of the initial 3D model with initial materials), point cloud sampling can be performed on the mesh of the initial 3D model with initial materials to obtain multiple material point clouds. Then, the multiple material point clouds are mapped to the 3D voxel mesh space, and the high-resolution, low-dimensional second sparse voxels are obtained based on the voxel meshes containing material point clouds falling in the 3D voxel mesh space. Secondly, the high-resolution, low-dimensional second sparse voxels are input into the sparse convolutional coding layer of the encoding module to be trained, to obtain high-dimensional features representing geometric and material information. These high-dimensional features are then input into the first sparse spatial attention layer of the encoding module to be trained, to obtain the first sparse voxel encoding features. Next, the first sparse voxel encoding features are input into the second sparse spatial attention layer of the decoding module to be trained, to obtain the reconstructed sparse voxel features in the high-dimensional latent space. These reconstructed sparse voxel features are then input into the sparse convolutional decoding layer of the decoding module to be trained, to decode and obtain the high-resolution, low-dimensional estimated sparse voxels. Finally, a loss is calculated based on the estimated sparse voxels and the reconstructed sparse voxels (i.e., the label data corresponding to the second sparse voxels), to obtain the first loss value. The parameters of the encoding module and the decoding module to be trained are then fine-tuned using the first loss value to obtain the trained encoding module and the trained decoding module.

[0110] Furthermore, having obtained the trained encoding module, a second training phase is performed to train the sparse flow matching model (Sparse flow transformer), specifically, as follows: Figure 8 As shown, firstly, the first sparse voxel is obtained, and the method for obtaining the first sparse voxel can refer to the method for obtaining the second sparse voxel described above, which will not be repeated here. At this time, the first sparse voxel can be input into the trained encoding module to obtain low-resolution, high-dimensional sparse voxel reference features, which are used as label data for training the sparse flow matching model. Secondly, the mesh of the initial 3D model is voxelized and noise-added to obtain the initial noisy voxel, and it is input together with the initial reference image of the initial 3D model with the initial material into the sparse flow matching model to be trained to obtain the first sparse voxel estimation features in the latent space. Finally, the first sparse voxel features and the sparse voxel reference features are used to calculate the loss to obtain the target loss value, and the target loss value is used to fine-tune the parameters of the sparse flow matching model to be trained to obtain the trained sparse flow matching model.

[0111] Figure 9 This is a schematic flowchart illustrating a method for generating data of a three-dimensional model with materials according to an embodiment of this application. This method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0112] Furthermore, the method includes at least a portion of the following: For example... Figure 9 As shown, it includes: Step S901: Obtain the mesh data of the target 3D model and the target reference image.

[0113] Here, the target reference image is a two-dimensional image of a reference three-dimensional model with the target material; the reference three-dimensional model is structurally related to the target three-dimensional model.

[0114] It should be noted that the relevant explanations regarding the target reference image can be found in the examples above, and will not be repeated here.

[0115] Step S902: Preprocess the mesh data of the target 3D model to obtain target noisy voxels.

[0116] It should be noted that the specific processing logic for preprocessing can be found in the example above, and will not be repeated here.

[0117] Step S903: Input the noisy target voxel and the target reference image into the target diffusion model to obtain sparse voxel prediction features.

[0118] Here, the target diffusion model is trained using the model training method of any of the above embodiments.

[0119] Furthermore, the obtained sparse voxel prediction features are the predicted values ​​of sparse voxel features, and the feature information used to characterize the geometric information of the target 3D model and the material information of the target material on the surface of the reference 3D model from the viewpoint corresponding to the target reference image.

[0120] Step S904: Use the obtained sparse voxel prediction features to obtain the target sparse voxel.

[0121] Here, the target sparse voxel carries geometric information of the target 3D model's geometry and material information of the target material.

[0122] Step S905: Based on the target sparse voxel, obtain at least one of the following: Mesh data of a target 3D model with the target material. A rendered image of a target 3D model with the target material.

[0123] It should be noted that the solution disclosed herein can be combined with the user's output requirements. For example, in one example, if the user's output requirement is to output the mesh data of a 3D model with materials, the mesh data of the target 3D model with the target material can be obtained based on the target sparse voxels; or, in another example, if the user's output requirement is to output the rendered image of a 3D model with materials, the rendered image of the target 3D model with the target material can be obtained based on the target sparse voxels.

[0124] In this way, the disclosed solution utilizes a target diffusion model and combines the target noisy voxels and the target reference image to obtain low-resolution, high-dimensional sparse voxel prediction features. Based on the obtained sparse voxel prediction features, high-resolution, low-dimensional target sparse voxels are obtained, thereby obtaining material data for the 3D model. Thus, with sparse voxels as the carrier of 3D attributes, the model can more accurately understand and reconstruct material details, thereby avoiding detail blurring and flickering caused by relying on multi-view intermediate representations. This improves the controllability and consistency of material generation, further ensures the resolution of textures during material generation, and thus improves the fidelity of the material in the reconstructed 3D model.

[0125] Furthermore, in a specific example, the target sparse voxel can be obtained in the following manner; specifically, the above-described method of using the obtained sparse voxel prediction features to obtain the target sparse voxel (e.g., step S904) can specifically include: The obtained sparse voxel prediction features are input into the decoding module to decode the target sparse voxels carrying the geometric information of the target 3D model and the material information of the target material.

[0126] In other words, after obtaining the sparse voxel prediction features using the target diffusion model, the obtained sparse voxel prediction features are decoded using a decoding module (such as a trained decoding module) to obtain the target sparse voxel carrying the geometric information of the geometric shape of the target 3D model and the material information of the target material.

[0127] In this way, the disclosed solution uses a "target diffusion model + trained decoding module". For example, the diffusion model is used to obtain sparse voxel features in the latent space (i.e., sparse voxel prediction features), and the decoding module is used to decode the sparse voxel features to obtain an explicit sparse voxel representation with material (such as the target sparse voxel). This allows for a more accurate understanding and reconstruction of material details, providing favorable support for the subsequent accurate acquisition of 3D data with material. Moreover, the disclosed solution effectively solves the problem of accumulated errors caused by relying on intermediate representations of multiple views through sparse voxel representations, thereby avoiding the problems of detail blurring and flickering caused by it from the root, thus improving the fidelity and detail reproduction of material generation on the 3D model.

[0128] Furthermore, in a specific example, the data of the 3D model with materials can be obtained through post-processing methods, specifically including at least one of the following: Method 1: That is, based on the target sparse voxels described above, obtaining the rendering effect of the target 3D model with the target material (e.g., step S905) can specifically include: The rendering engine is invoked, and the target sparse voxels, the mesh data of the target 3D model, and the camera data from the preset viewpoint are combined to perform rendering processing, so as to obtain the rendering effect of the target 3D model with the target material under the preset viewpoint.

[0129] In other words, when the user's output requirement is to output a rendered image of a 3D model with materials, the rendering engine is invoked, and the rendering process is performed based on the explicit sparse voxels with materials (i.e., the target sparse voxels) and the preset viewpoint specified by the user (such as the camera data of the preset viewpoint input), to obtain a rendered image of the target 3D model with the target materials under the preset viewpoint.

[0130] In this way, the disclosed solution can process the target sparse voxels through post-processing to obtain a rendered image of the target 3D model with the target material. This greatly improves the realism and fidelity of the distribution of the target material on the target 3D model, and at the same time, enhances the user experience.

[0131] Method 2: That is, obtaining the mesh data of the target 3D model with the target material based on the target sparse voxels as described above (e.g., step S905), can specifically include: The target sparse voxels are mapped onto the mesh data of the target 3D model to obtain the mesh data of the target 3D model with the target material.

[0132] In other words, when the user's output requirement is to output the mesh data of a 3D model with material, the target sparse voxels are mapped to the mesh data of the target 3D model, for example, mapped to the mesh occupied by the target 3D model in 3D space, to obtain the mesh data of the target 3D model with the target material.

[0133] In this way, the disclosed solution can use post-processing to convert the target sparse voxels into commonly used mesh data, thereby effectively outputting 3D assets that meet the requirements of 3D design, which facilitates subsequent rendering, editing and display of the mesh data of the 3D model with materials.

[0134] Figure 10This is a schematic diagram illustrating how the disclosed solution reconstructs a 3D model of a target with target material using a target diffusion module and a decoding module. For example... Figure 10 As shown, the mesh data of the target 3D model is voxelized and noise-added to obtain noisy target voxels. These noisy voxels, along with a target reference image of the target 3D model with the target material, are then input into a trained sparse flow matching model (corresponding to the target diffusion model mentioned above) to obtain low-resolution (e.g., 64) high-dimensional sparse voxel prediction features in the latent space. Further, these sparse voxel prediction features are input into the second sparse space attention layer of the trained decoding module to obtain reconstructed sparse voxel features in the high-dimensional latent space. This process then converts the high-dimensional latent space into a more coherent and accurate representation of the target material. The reconstructed sparse voxel features in space are input into the sparse convolutional decoding layer in the trained decoding module to decode and obtain the target sparse voxels. At this point, the output can be in a specified format according to the user's output requirements. For example, the rendering engine can be invoked, and rendering processing can be performed using the target sparse voxels, the mesh of the target 3D model, and camera data from a specified viewpoint to obtain a rendered image of the target 3D model with the target material from the user's specified viewpoint. Alternatively, the target sparse voxels can be mapped onto the mesh of the target 3D model to obtain a mesh of the target 3D model with the target material. This not only improves the reconstruction quality of the 3D model with materials but also enhances the user experience through the open choice of output format.

[0135] Thus, this disclosed solution utilizes a "target diffusion model + trained decoding module" to obtain an explicit sparse voxel representation with target material (such as target sparse voxels), thereby effectively improving the fidelity and detail of the reconstructed 3D model with material. Furthermore, this disclosed solution forms a complete data transformation chain through a preprocessing process of extracting voxels from the mesh (such as the mesh data of the target 3D model) and a postprocessing process of extracting the mesh (mesh data of the target 3D model with target material) from the voxels (such as target sparse voxels). Since sparse voxels are always used as intermediate representations during model processing, this helps reduce geometric detail loss and data representation errors during training data construction while improving the reconstruction efficiency of the 3D model.

[0136] This disclosure provides a model training device for generative models, such as... Figure 11 As shown, it includes: The first acquisition unit 1101 is used to acquire target training samples; the target training samples include: mesh data of an initial three-dimensional model, an initial reference image, and sparse voxel reference features; wherein, the sparse voxel reference features are obtained by encoding sparse voxels carrying information related to the geometry of the initial three-dimensional model and information related to the initial material; the initial reference image is a two-dimensional image of a reference three-dimensional model with the initial material; the reference three-dimensional model is structurally related to the initial three-dimensional model; The data processing unit 1102 is used to preprocess the mesh data of the initial 3D model to obtain initial noisy voxels; input the initial noisy voxels and the initial reference image into the diffusion model to be trained to obtain a first sparse voxel estimation feature that represents the geometric information of the initial 3D model and the material information of the initial material of the reference 3D model surface from the viewpoint corresponding to the initial reference image; obtain a target loss value based on the difference information between the sparse voxel reference feature and the first sparse voxel estimation feature; and train the diffusion model to be trained using the target loss value so that the trained target diffusion model can output sparse voxel features that represent geometric information and material information.

[0137] In a specific example of the disclosed solution, the first acquisition unit is further configured to: Obtain the first sparse voxel; wherein the first sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material. The first sparse voxel is processed using the trained encoding module to obtain the sparse voxel reference features.

[0138] In a specific example of the disclosed solution, the data processing unit is specifically used for: The mesh data of the initial three-dimensional model is voxelized to obtain sparse voxels of the initial three-dimensional model; The sparse voxels of the initial 3D model are noise-added to obtain the initial noisy voxels.

[0139] In a specific example of the scheme disclosed herein, the first acquisition unit is further configured to acquire a second sparse voxel; wherein the second sparse voxel carries information related to the geometry of the initial three-dimensional model and information related to the initial material. The data processing unit is further configured to input the second sparse voxel into the encoding module to be trained, so as to encode a first sparse voxel encoding feature; wherein, the first sparse voxel encoding feature represents the combined information of the geometric information of the initial three-dimensional model and the material information of the initial material; input the first sparse voxel encoding feature into the decoding module, so as to decode an estimated sparse voxel carrying the relevant information of the geometric shape of the initial three-dimensional model and the relevant information of the initial material; obtain a first loss value based on the estimated sparse voxel; and use the first loss value to train the encoding module to be trained to obtain the trained encoding module.

[0140] In a specific example of the disclosed solution, the data processing unit is specifically used for: Using the first loss value, the decoding module and the encoding module to be trained are trained to obtain the trained encoding module and the trained decoding module.

[0141] In a specific example of the scheme disclosed herein, the encoding module includes a sparse convolutional encoding layer and a first sparse spatial attention layer; wherein, the sparse convolutional encoding layer is used to extract the geometric information of the geometric shape and the material information of the initial material in the input sparse voxels to obtain high-dimensional features representing the geometric information and material information; the first sparse spatial attention layer is used to enhance the key information in the high-dimensional features while suppressing the redundant information in the high-dimensional features to encode the sparse voxel features; And / or, The decoding module includes a second sparse spatial attention layer and a sparse convolutional decoding layer. The second sparse spatial attention layer is used to reconstruct the material information in the input sparse voxel features and globally distribute it in three-dimensional space to obtain the reconstructed sparse voxel features in the high-dimensional latent space. The sparse convolutional decoding layer is used to restore the sparse voxel features in the high-dimensional latent space and obtain low-dimensional features that represent geometric information with geometric shape and material information of the initial material to estimate the sparse voxels.

[0142] In a specific example of the disclosed solution, the data processing unit is further configured to: Using the mesh data of the initial 3D model, inverse rendering is performed on N initial model images to obtain the first sparse voxel; wherein, the initial model image is a 2D image of the initial 3D model with the initial material; the initial 3D model has a different viewpoint in different initial model images; or, Point cloud sampling is performed on each triangular face represented by the mesh data of the initial 3D model carrying the initial material to obtain multiple material point clouds. The multiple material point clouds are then used for spatial feature processing to obtain the first sparse voxel.

[0143] This disclosure also provides a data generation device for a 3D model with materials, such as... Figure 12 As shown, it includes: The second acquisition unit 1201 is used to acquire the mesh data of the target 3D model and the target reference image; the target reference image is a 2D image of the reference 3D model with the target material; the reference 3D model is structurally related to the target 3D model; The data generation unit 1202 is used to preprocess the mesh data of the target 3D model to obtain target noisy voxels; input the target noisy voxels and the target reference image into the target diffusion model to obtain sparse voxel prediction features; wherein, the target diffusion model is trained using any of the model training methods described above; using the obtained sparse voxel prediction features, target sparse voxels are obtained; based on the target sparse voxels, at least one of the following is obtained: mesh data of the target 3D model with target material, and a rendering effect image of the target 3D model with target material.

[0144] In a specific example of the disclosed solution, the data generation unit is specifically used for: The obtained sparse voxel prediction features are input into the decoding module to decode the target sparse voxels carrying the geometric information of the target 3D model and the material information of the target material.

[0145] In a specific example of the disclosed solution, the data generation unit is specifically used for: The rendering engine is invoked, and the target sparse voxels, the mesh data of the target 3D model, and the camera data from the preset viewpoint are combined to perform rendering processing, so as to obtain the rendering effect of the target 3D model with the target material under the preset viewpoint.

[0146] In a specific example of the disclosed solution, the data generation unit is specifically used for: The target sparse voxels are mapped onto the mesh data of the target 3D model to obtain the mesh data of the target 3D model with the target material.

[0147] For a description of the specific functions and examples of each unit of the apparatus in this disclosure embodiment, please refer to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be repeated here.

[0148] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0149] Figure 13 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 13As shown, the electronic device includes a memory 1310 and a processor 1320. The memory 1310 stores a computer program that can run on the processor 1320. The number of memories 1310 and processors 1320 can be one or more. The memory 1310 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the methods provided in the above-described method embodiments. The electronic device may also include a communication interface 1330 for communicating with external devices and performing data exchange and transmission.

[0150] If the memory 1310, processor 1320, and communication interface 1330 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0151] Optionally, in a specific implementation, if the memory 1310, processor 1320 and communication interface 1330 are integrated on a single chip, the memory 1310, processor 1320 and communication interface 1330 can communicate with each other through an internal interface.

[0152] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0153] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

[0154] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)). It is worth noting that the computer-readable storage media mentioned in this disclosure can be non-volatile storage media; in other words, it can be non-transient storage media.

[0155] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0156] In the description of the embodiments of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0157] In the description of the embodiments disclosed herein, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0158] In the description of embodiments of this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0159] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for training a generative model, comprising: Enter the first stage of training: train the encoding module based on the second sparse voxel carrying the relevant information of the geometry and the relevant information of the initial 3D model, so that the trained encoding module can output the encoding features that represent the comprehensive information of geometric and material information. After the first stage of training is completed, the acquired first sparse voxel is processed using the trained encoding module to obtain the sparse voxel reference feature after encoding the first sparse voxel; wherein, the first sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material. Entering the second stage of training: acquiring target training samples; the target training samples include: mesh data of the initial 3D model, an initial reference image, and sparse voxel reference features; wherein, the initial reference image is a 2D image of the reference 3D model with the initial material; the reference 3D model is structurally related to the initial 3D model; The mesh data of the initial 3D model is preprocessed to obtain initial noisy voxels; The initial noisy voxel and the initial reference image are input into the diffusion model to be trained to obtain the first sparse voxel estimation features that represent the geometric information of the initial 3D model and the material information of the initial material of the reference 3D model surface under the viewpoint corresponding to the initial reference image. Based on the difference information between the sparse voxel reference features and the first sparse voxel estimation features, the target loss value is obtained. The target loss value is used to train the diffusion model to be trained so that the trained target diffusion model can output sparse voxel features that represent geometric and material information.

2. The method according to claim 1, wherein, The preprocessing of the mesh data of the initial 3D model to obtain initial noisy voxels includes: The mesh data of the initial three-dimensional model is voxelized to obtain sparse voxels of the initial three-dimensional model; The sparse voxels of the initial 3D model are noise-added to obtain the initial noisy voxels.

3. The method according to claim 1, wherein, The trained encoding module is obtained in the following way: Obtain a second sparse voxel; wherein the second sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material. The second sparse voxel is input into the encoding module to be trained to encode the first sparse voxel encoding feature; wherein, the first sparse voxel encoding feature represents the combined information of the geometric information of the initial 3D model and the material information of the initial material; The first sparse voxel encoding feature is input to the decoding module to decode and obtain the estimated sparse voxel carrying the relevant information of the geometry of the initial three-dimensional model and the relevant information of the initial material. Based on the estimated sparse voxels, the first loss value is obtained; Using the first loss value, the encoding module to be trained is used to train the model to obtain the trained encoding module.

4. The method according to claim 3, wherein, The step of using the first loss value to train the encoding module to be trained to obtain the trained encoding module includes: Using the first loss value, the decoding module and the encoding module to be trained are trained to obtain the trained encoding module and the trained decoding module.

5. The method according to claim 3, wherein, The encoding module includes a sparse convolutional encoding layer and a first sparse spatial attention layer. The sparse convolutional encoding layer is used to extract the geometric information of the geometric shape and the material information of the initial material in the input sparse voxels to obtain high-dimensional features that represent the geometric and material information. The first sparse spatial attention layer is used to enhance the key information in the high-dimensional features while suppressing the redundant information in the high-dimensional features to encode the sparse voxel features. And / or, The decoding module includes a second sparse spatial attention layer and a sparse convolutional decoding layer. The second sparse spatial attention layer is used to reconstruct the material information in the input sparse voxel features and globally distribute it in three-dimensional space to obtain the reconstructed sparse voxel features in the high-dimensional latent space. The sparse convolutional decoding layer is used to restore the sparse voxel features in the high-dimensional latent space and obtain low-dimensional features that represent geometric information with geometric shape and material information of the initial material to estimate the sparse voxels.

6. The method according to any one of claims 1-5, further comprising: Using the mesh data of the initial 3D model, inverse rendering is performed on N initial model images to obtain the first sparse voxel; wherein, the initial model image is a 2D image of the initial 3D model with the initial material; the initial 3D model has a different viewpoint in different initial model images; or, Point cloud sampling is performed on each triangular face represented by the mesh data of the initial 3D model carrying the initial material to obtain multiple material point clouds. The multiple material point clouds are then used for spatial feature processing to obtain the first sparse voxel.

7. A method for generating data for a 3D model with materials, comprising: Acquire the mesh data of the target 3D model and the target reference image; The target reference image is a two-dimensional image of a reference three-dimensional model with the target material; The reference 3D model is structurally related to the target 3D model; The mesh data of the target 3D model is preprocessed to obtain noisy target voxels; The target noisy voxel and the target reference image are input into the target diffusion model to obtain sparse voxel prediction features; wherein, the target diffusion model is trained using the model training method described in any one of claims 1 to 6; The target sparse voxel is obtained by using the predicted features of the sparse voxel. Based on the target sparse voxel, at least one of the following is obtained: Mesh data of a target 3D model with the target material. A rendered image of a target 3D model with the target material.

8. The method according to claim 7, wherein, The process of obtaining the target sparse voxel by utilizing the predicted features of the obtained sparse voxels includes: The obtained sparse voxel prediction features are input into the decoding module to decode the target sparse voxels carrying the geometric information of the target 3D model and the material information of the target material.

9. The method according to claim 7 or 8, wherein, Based on the target sparse voxels, a rendered image of the target 3D model with the target material is obtained, including: The rendering engine is invoked, and the target sparse voxels, the mesh data of the target 3D model, and the camera data from the preset viewpoint are combined to perform rendering processing, so as to obtain the rendering effect of the target 3D model with the target material under the preset viewpoint.

10. The method according to claim 7 or 8, wherein, Based on the target sparse voxels, mesh data of the target 3D model with the target material is obtained, including: The target sparse voxels are mapped onto the mesh data of the target 3D model to obtain the mesh data of the target 3D model with the target material.

11. A model training device for a generative model, comprising: The data processing unit is used to enter the first training phase to train the encoding module based on the second sparse voxel carrying the relevant information of the geometry of the initial 3D model and the relevant information of the initial material, so that the trained encoding module can output the encoding features that represent the comprehensive information of geometric and material information. It is also used to perform feature processing on the first sparse voxel after the first stage of training is completed, using the trained encoding module to obtain sparse voxel reference features after encoding the first sparse voxel; wherein, the first sparse voxel carries information related to the geometry of the initial 3D model and information related to the initial material. The first acquisition unit is used to enter the second training phase to acquire target training samples; the target training samples include: mesh data of the initial 3D model, an initial reference image, and sparse voxel reference features; wherein, the initial reference image is a 2D image of the reference 3D model with the initial material; the reference 3D model is structurally related to the initial 3D model; The data processing unit is further configured to: preprocess the mesh data of the initial 3D model in the second training phase to obtain initial noisy voxels; input the initial noisy voxels and the initial reference image into the diffusion model to be trained to obtain a first sparse voxel estimation feature representing the geometric information of the initial 3D model and the material information of the initial material on the surface of the reference 3D model from the viewpoint corresponding to the initial reference image; obtain a target loss value based on the difference information between the sparse voxel reference feature and the first sparse voxel estimation feature; and train the diffusion model to be trained using the target loss value so that the trained target diffusion model can output sparse voxel features representing geometric and material information.

12. The apparatus according to claim 11, wherein, The data processing unit is specifically used for: The mesh data of the initial three-dimensional model is voxelized to obtain sparse voxels of the initial three-dimensional model; The sparse voxels of the initial 3D model are noise-added to obtain the initial noisy voxels.

13. The apparatus according to claim 11, wherein, The first acquisition unit is further configured to acquire a second sparse voxel; wherein the second sparse voxel carries information related to the geometry of the initial three-dimensional model and information related to the initial material. The data processing unit is further configured to input the second sparse voxel into the encoding module to be trained, so as to encode a first sparse voxel encoding feature; wherein, the first sparse voxel encoding feature represents the combined information of the geometric information of the initial three-dimensional model and the material information of the initial material; input the first sparse voxel encoding feature into the decoding module, so as to decode an estimated sparse voxel carrying the relevant information of the geometric shape of the initial three-dimensional model and the relevant information of the initial material; obtain a first loss value based on the estimated sparse voxel; and use the first loss value to train the encoding module to be trained to obtain the trained encoding module.

14. The apparatus according to claim 13, wherein, The data processing unit is specifically used for: Using the first loss value, the decoding module and the encoding module to be trained are trained to obtain the trained encoding module and the trained decoding module.

15. The apparatus according to claim 13, wherein, The encoding module includes a sparse convolutional encoding layer and a first sparse spatial attention layer. The sparse convolutional encoding layer is used to extract the geometric information of the geometric shape and the material information of the initial material in the input sparse voxels to obtain high-dimensional features that represent the geometric and material information. The first sparse spatial attention layer is used to enhance the key information in the high-dimensional features while suppressing the redundant information in the high-dimensional features to encode the sparse voxel features. And / or, The decoding module includes a second sparse spatial attention layer and a sparse convolutional decoding layer. The second sparse spatial attention layer is used to reconstruct the material information in the input sparse voxel features and globally distribute it in three-dimensional space to obtain the reconstructed sparse voxel features in the high-dimensional latent space. The sparse convolutional decoding layer is used to restore the sparse voxel features in the high-dimensional latent space and obtain low-dimensional features that represent geometric information with geometric shape and material information of the initial material to estimate the sparse voxels.

16. The apparatus according to any one of claims 11-15, wherein, The data processing unit is further configured to: Using the mesh data of the initial 3D model, inverse rendering is performed on N initial model images to obtain the first sparse voxel; wherein, the initial model image is a 2D image of the initial 3D model with the initial material; the initial 3D model has a different viewpoint in different initial model images; or, Point cloud sampling is performed on each triangular face represented by the mesh data of the initial 3D model carrying the initial material to obtain multiple material point clouds. The multiple material point clouds are then used for spatial feature processing to obtain the first sparse voxel.

17. A data generation device for a three-dimensional model with materials, comprising: The second acquisition unit is used to acquire the mesh data of the target 3D model and the target reference image; The target reference image is a two-dimensional image of a reference three-dimensional model with the target material; The reference 3D model is structurally related to the target 3D model; A data generation unit is used to preprocess the mesh data of the target 3D model to obtain target noisy voxels; input the target noisy voxels and the target reference image into a target diffusion model to obtain sparse voxel prediction features; wherein, the target diffusion model is trained using the model training method described in any one of claims 1 to 6; and the obtained sparse voxel prediction features are used to obtain target sparse voxels. Based on the target sparse voxels, at least one of the following is obtained: mesh data of the target 3D model with the target material, and a rendering of the target 3D model with the target material.

18. The apparatus according to claim 17, wherein, The data generation unit is specifically used for: The obtained sparse voxel prediction features are input into the decoding module to decode the target sparse voxels carrying the geometric information of the target 3D model and the material information of the target material.

19. The apparatus according to claim 17 or 18, wherein, The data generation unit is specifically used for: The rendering engine is invoked, and the target sparse voxels, the mesh data of the target 3D model, and the camera data from the preset viewpoint are combined to perform rendering processing, so as to obtain the rendering effect of the target 3D model with the target material under the preset viewpoint.

20. The apparatus according to claim 17 or 18, wherein, The data generation unit is specifically used for: The target sparse voxels are mapped onto the mesh data of the target 3D model to obtain the mesh data of the target 3D model with the target material.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-6 or 7-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6 or 7-10.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6 or 7-10.

Images