3D model generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a 3D model generation method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing a sparse voxel feature field for a 3D model of a preset category, the sparse voxel feature field comprising initial geometric features and initial color features of voxel vertexes; a joint reconstruction model is trained according to the sparse voxel feature field, a trained joint reconstruction model is obtained, the joint reconstruction model comprises a first sub-model and a second sub-model, the first sub-model is used for generating a joint representation vector related to a preset category according to input geometric features and color features, and the second sub-model is used for generating a second representation vector related to the preset category; the second sub-model is used for reconstructing the joint representation vector into geometric features and color features; inputting a target joint representation vector related to a target 3D model of a preset category into the second sub-model to generate a target geometric feature; and generating a target 3D model according to the target geometric features and the sparse voxel feature field. Color features and model structure point positions are bound, dynamic deformation of the 3D model can be adapted, and the high-quality 3D model is automatically generated.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence application technology, and in particular to a 3D model generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In modern video game and digital film production, there is usually a high requirement for visual realism, which necessitates the design of a large number of high-quality three-dimensional (3D) models. Currently, artificial intelligence (AI) is generally used to automatically generate these 3D models, but this approach often suffers from the following problems:

[0003] First, existing AI can usually only generate "white models" of objects, that is, only the shape without color, which requires users to add color. This process is not only cumbersome and inefficient, but also requires a high amount of manpower and time when there are a large number of 3D models.

[0004] Secondly, existing AI-generated 3D models are prone to color and geometric detail misalignment during dynamic deformation, resulting in "texture stretching" or color overflow, which seriously affects the final art quality.

[0005] In conclusion, current AI technologies struggle to achieve automated generation of high-quality 3D models. Summary of the Invention

[0006] This invention provides a 3D model generation method to solve the problem that existing AI technologies struggle to automatically generate high-quality 3D models.

[0007] In a first aspect, the present invention provides a 3D model generation method, comprising:

[0008] A sparse voxel feature field is constructed for a 3D model of a preset category. The sparse voxel feature field includes multiple voxels and voxel feature data. The feature data includes the initial geometric features and initial color features of the voxel vertices.

[0009] The joint reconstruction model is trained based on the sparse voxel feature field to obtain the trained joint reconstruction model. The joint reconstruction model includes a first sub-model and a second sub-model. The first sub-model is used to generate a joint representation vector related to a preset category based on the input geometric features and color features. The second sub-model is used to reconstruct the joint representation vector into geometric features and color features.

[0010] Input the joint representation vector of the target associated with the target 3D model of the preset category into the trained second sub-model to generate the target geometric features associated with the target 3D model;

[0011] The target 3D model is generated based on the target's geometric features and the sparse voxel feature field.

[0012] In a second aspect, the present invention provides a 3D model generation apparatus, comprising:

[0013] The feature field construction module is used to construct a sparse voxel feature field for a 3D model of a preset category. The sparse voxel feature field includes multiple voxels and feature data of the voxels. The feature data includes the initial geometric features and initial color features of the voxel vertices.

[0014] The model training module is used to train a joint reconstruction model based on the sparse voxel feature field to obtain the trained joint reconstruction model. The joint reconstruction model includes a first sub-model and a second sub-model. The first sub-model is used to generate a joint representation vector related to a preset category based on the input geometric features and color features. The second sub-model is used to reconstruct the joint representation vector into geometric features and color features.

[0015] The target geometric feature acquisition module is used to input the joint representation vector of the target 3D model related to the preset category into the trained second sub-model to generate target geometric features related to the target 3D model.

[0016] The model generation module is used to generate a 3D model of the target based on the target's geometric features and the sparse voxel feature field.

[0017] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the 3D model generation method described in the first aspect of the present invention.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the 3D model generation method described in the first aspect of the present invention.

[0022] The 3D model generation method provided in this embodiment of the invention has the following beneficial effects:

[0023] A sparse voxel feature field is constructed for 3D models of preset categories, where the initial geometric and color features of each voxel vertex are correlated. When training the joint reconstruction model based on the sparse voxel feature field, the first sub-model learns the joint representation of geometric and color features related to the preset category, obtaining a joint representation vector related to the preset category. After training the joint reconstruction model, the first sub-model can be used to generate different joint representation vectors as needed. The joint representation vectors enable high coupling and alignment between geometric and color features. When using joint representation vectors to construct 3D models, color features can be bound to fixed points on the model body, better adapting to various dynamic deformations of the 3D model and avoiding the seams and distortion problems caused by stretching traditional textures, thus ensuring visual quality under dynamic conditions.

[0024] Secondly, once the joint reconstruction model is trained, it is only necessary to obtain the joint representation vector of the target 3D model related to the preset category. The joint representation vector can then be input into the second sub-model of the joint reconstruction model to generate target geometric features for constructing the target 3D model of the preset category. Based on the generated target geometric features and sparse voxel feature field, the target 3D model is constructed. Users can obtain or generate different joint representation vectors to automatically generate high-quality 3D models with integrated shape and color without the need for secondary coloring. This greatly improves the efficiency of 3D model production and reduces manpower and time costs.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a 3D model generation method provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the data preprocessing, representation, and model training stages provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the model reasoning and generation stage provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of different 3D models of a preset category provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of a 3D model generation device provided in an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] First, some of the terms involved in this invention will be explained:

[0035] Sparse voxel fields: Sparse voxel fields are a data structure and model used for efficiently representing and manipulating three-dimensional spatial data (especially data with empty regions). A voxel is an abbreviation for "volume pixel," which is the basic unit for discretizing this 3D field.

[0036] A dense voxel grid can be imagined as a uniform, massive array of three-dimensional pixels. A sparse voxel field is a voxel grid with sparsity; for example, for most 3D models, the effective data is that the surface of the 3D model occupies only a small portion of the entire space, with the majority being "empty." Instead of using a large, uniform 3D array, a sparse voxel field employs an efficient data structure to store and manage only non-empty voxels.

[0037] Figure 1 This is a flowchart illustrating a 3D model generation method provided in an embodiment of the present invention. This embodiment is applicable to 3D model generation. The method can be executed by a 3D model generation device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the 3D model generation method includes:

[0038] S101. Construct a sparse voxel feature field for a 3D model of a preset category. The sparse voxel feature field includes multiple voxels and voxel feature data. The feature data includes the initial geometric features and initial color features of the voxel vertices.

[0039] A preset category represents a model style. For example, a preset category can be a bronze style, a marble style, a grass style, or a building style. 3D models of different preset categories usually have differences in texture, shape, pattern, and feel. The preset category can be set according to actual needs, and this invention does not limit it.

[0040] 3D models are typically polygonal meshes, such as irregular meshes. The first step is to acquire the 3D model data, which can be point clouds, parametric surfaces, or directed distance function values. Preprocessing steps for the 3D model include normalization, scaling and translating the 3D model to a preset standard coordinate system, and correcting topological errors and removing redundant faces. This step converts the continuous 3D model into discrete voxels.

[0041] It should be noted that the feature data corresponding to the 3D model structure is the training data for the joint reconstruction model. To ensure that the joint reconstruction model has a sufficient quantity and variety of training data, the number of 3D models should be set to multiple. For example, if a user wants to generate a large number of new tree 3D models, and has not previously trained a joint reconstruction model on the 3D models with the preset category of "trees", then the existing large number of tree 3D model data can be used to extract feature data as the training data for the joint reconstruction model in S102.

[0042] It should also be noted that since the accuracy of the training data affects the training effect of the joint reconstruction model, the 3D model used in this step needs to be a 3D model that meets the user's needs. That is, the geometric structure of the model and the combination of geometric structure and color meet the user's needs, so that the geometric structure and the combination of geometric structure and color corresponding to the data predicted by the subsequently trained joint reconstruction model can also meet the user's needs.

[0043] Optionally, a sparse voxel feature field is constructed for 3D models of a preset category, including: determining voxelization parameters and a stereo space surrounding the 3D models of the preset category; performing voxelization processing in the stereo space according to the voxelization parameters to obtain multiple voxels; selecting surface voxels within a preset range from the surface of the 3D model; performing voxel feature sampling on the selected surface voxels to obtain feature data of multiple surface voxels; and binding the surface voxels with their corresponding feature data to obtain a sparse voxel feature field.

[0044] The voxelization parameter can be the target resolution or a preset voxel size. When a axially aligned solid space that can completely surround the 3D model is determined, the solid space is divided into X, Y, and Z axes. Then, based on the voxelization parameter, the solid space is divided into multiple uniform cubic units along the X, Y, and Z axes, and each unit defines a voxel.

[0045] Placing a 3D model into a huge space composed of countless tiny cubes (voxels), such as a 1024×1024×1024 voxel grid, would consume a lot of computing power and time if all voxels were processed, reducing the efficiency and quality of the job. Since most voxels are in an unoccupied state, in order to improve storage and access efficiency, this embodiment uses a sparse data structure to store only surface voxels within a preset range from the surface of the 3D model, thereby obtaining sparse voxels. Then, voxel feature sampling can be performed on the sparse voxels to obtain feature data.

[0046] Voxel feature sampling involves acquiring feature data for each voxel vertex (a voxel cube has 8 voxel vertices). The feature data includes geometric features and color features. Typically, two key geometric values ​​need to be calculated: the Signed Distance Function (SDF) value and the Deformation Vector.

[0047] Directed Distance Function (SDF) value: Calculates the distance from this vertex to the nearest surface of the 3D model. This distance has a positive or negative sign: positive if the vertex is outside the model, and negative if it is inside the model. A point with an SDF value of 0 is exactly on the model surface. The directed distance function perfectly describes the internal and external space and surface contours of an object.

[0048] Deformation Vector: This is a 3D vector that can be initially set to zero. In more advanced implementations, it can be learned through optimization and used to fine-tune the voxel mesh to better fit very sharp or minute details on the model, much like fine-tuning clay with your fingers.

[0049] The directed distance function value and the deformation vector together constitute a precise mathematical description of the model's geometry.

[0050] Color features: For the same voxel vertex, find its nearest point on the surface of the original textured model, and read the RGB color value of that point as the RGB value of the voxel vertex. Based on this RGB value, obtain the color features (i.e., initial color features) of the voxel vertex. In this step, the color information is extracted and bound to a specific voxel vertex in space.

[0051] Associating the geometric features (SDF value, deformation vector) and color features (color code) obtained in the above steps with the corresponding voxel vertices, each activated sparse voxel vertex carries a rich data packet, such as {SDF: -0.05, Deformation: [0, 0, 0], Color Code: [0.1, -0.4, ..., 0.9]}.

[0052] The data set of all these vertices constitutes the multimodal sparse voxel feature field.

[0053] In an optional embodiment, voxel feature sampling is performed on the sparse voxel feature field to obtain feature data of multiple surface voxels, including:

[0054] Voxel feature sampling is performed on the sparse voxel feature field to obtain the directed distance function value, deformation vector, and RGB vector of the voxel vertices of all surface voxels. For each voxel vertex, the corresponding directed distance function value and deformation vector are used as initial geometric features. The corresponding RGB vector is converted into initial color features, where the number of data points in the RGB vector is less than the number of data points in the initial color features. For each surface voxel, the initial geometric features and initial color features of the voxel vertex are correlated and used as the feature data of the surface voxel.

[0055] The initial geometric features include the directed distance function value and deformation vector corresponding to the voxel vertices, and the initial color features are the RGB vectors of the voxel vertices. There are multiple sets of feature data, each set including at least one pair of initial geometric features and initial color features. Specifically, when the number of feature data is the same as the number of voxel vertices, a set of feature data includes one pair of initial geometric features and initial color features; when the number of feature data is the same as the number of surface voxels, and there are N voxel vertices, a set of feature data includes N pairs of initial geometric features and initial color features.

[0056] The number of data points in the RGB vector is less than the number of data points in the initial color feature, which means that the initial color feature has more data points and can represent richer, more detailed and more layered color features.

[0057] In this embodiment, the corresponding RGB vector can be converted into initial color features using a preset color conversion model. The training process of the color conversion model is as follows:

[0058] 1. Data collection: Randomly sample millions of pixel RGB vectors (color values) from a large number of images or model textures to form a huge color database.

[0059] 2. Model building: Build a small neural network consisting of an encoder and a decoder.

[0060] The encoder's task is to compress a 3-dimensional RGB vector into a lower-dimensional color feature code (i.e., the initial color feature), which is an abstract and condensed mathematical expression of the RGB vector.

[0061] The decoder's task is to receive this 8-dimensional feature code and try its best to restore it to the original 3-dimensional RGB color value. For example, the encoder compresses a 3-dimensional RGB vector into an 8-dimensional color feature code; the decoder's task is to receive this 8-dimensional color feature code and try its best to restore it to the original 3-dimensional RGB vector. It should be noted that the dimension of the encoded result only needs to be greater than 3 dimensions of the RGB vector. The "8-dimensional" described in this invention is only an example and is not intended to limit the invention.

[0062] 3. Training process: By continuously inputting RGB vectors, the color conversion model compresses and decompresses them, and compares the difference between the decompressed color and the original RGB vector. The parameters of the color conversion model are adjusted with the goal of minimizing this difference, and a trained color conversion model is obtained.

[0063] After training, this color conversion model becomes an efficient "color compression / decompression tool." Its advantage lies in the fact that similar colors (such as deep red and scarlet) will have mathematically very similar feature codes in an 8-dimensional feature space. This creates a smooth, continuous "color map," on which AI can naturally generate smooth color gradients rather than harsh color blocks when learning and interpolating.

[0064] S102. Train the joint reconstruction model based on the sparse voxel feature field to obtain the trained joint reconstruction model, which includes the first sub-model and the second sub-model.

[0065] The first sub-model is used to generate a joint representation vector related to a preset category based on the input geometric and color features, and the second sub-model is used to reconstruct the joint representation vector into geometric and color features.

[0066] The feature data consists of multiple sets. In each set, the initial geometric and color features are associated with the corresponding voxel vertices, thus the initial geometric and color features are also correlated. When training the joint reconstruction model, all feature data from a single 3D model can be input into the first sub-model of the joint reconstruction model. The first sub-model generates a predicted joint representation vector, which is then input into the second sub-model. The second sub-model further predicts the joint representation vector to obtain the reconstructed geometric and color features. The reconstructed geometric and color features are compared with the initial geometric and color features in the feature data. When the difference is minimized, the joint reconstruction model training is considered complete. During this training process, the first sub-model is required to generate accurate joint representation vectors related to the preset category, and the second sub-model is required to understand the structural data (i.e. feature data) of the 3D model represented by the joint representation vector. After training, the first sub-model can learn to understand the relationship between geometric features and color features, that is, learn how color is attached to the geometric structure in this type of 3D model; the second sub-model can learn how to accurately reconstruct the joint representation vector.

[0067] When generating joint representation vectors, these vectors can be stored in the vector space of the joint reconstruction model. Each vector space can be used to store the joint representation vectors corresponding to a preset category; that is, each vector space corresponds to a preset category. Taking a 3D model with a bronze ware style as an example, the joint representation vector is z. One z vector represents a complete and specific 3D bronze ware model instance, and one joint representation vector is a point in the vector space. After training the joint reconstruction model with a large amount of bronze ware 3D model data, the resulting vector space stores a large number of joint representation vectors related to the bronze ware style (color and structure), thus becoming a space representing the bronze ware style. In this space, any z vector corresponds to a unique bronze ware 3D model.

[0068] S103. Input the joint representation vector of the target related to the target 3D model of the preset category into the trained second sub-model to generate the target geometric features related to the target 3D model.

[0069] The target joint representation vector can be obtained directly or by sampling features from an existing 3D model of a preset category. Specifically, the target joint representation vector can be obtained through sampling or input.

[0070] Input: A specific joint representation vector is provided by the user. For example, a joint representation vector z is obtained by encoding a known 3D model, then slightly modified (such as changing the dimension value representing "size"), and then input back to generate a similar but different variant of the joint representation vector z as input.

[0071] Sampling: Randomly draw a joint representation vector z from a vector space that follows its probability distribution (usually a normal distribution). This is equivalent to "creating" a new 3D object concept that has appeared in the training data but conforms to its pattern.

[0072] In an optional embodiment, the target joint representation vector is generated by: obtaining a joint representation vector from a vector space corresponding to a preset category to obtain a spare vector; and generating the target joint representation vector based on the spare vector.

[0073] By obtaining joint representation vectors from the vector space of preset categories, multiple joint representation vectors with the same preset category can be obtained. Users can select or randomly choose at least one joint representation vector as a backup vector. When one backup vector is selected, it is directly used as the target joint representation vector, and the newly generated target 3D model of the preset category is the same as a historical 3D model. Alternatively, the backup vector can be fine-tuned to obtain a new joint representation vector, which can generate a 3D model that is different from the historical 3D model and has the preset category. When multiple identical backup vectors are selected, a joint representation vector that is related to the target style and already exists can be used as a backup vector; linear interpolation is performed on the backup vector to obtain the target joint representation vector. Optionally, when adjusting the backup vector, only the geometric features in the backup vector can be adjusted.

[0074] The vector space containing the joint representation vector of the current target style is continuous, but there are differences between any two joint representation vectors. Optionally, two or more different alternative vectors can be selected, and linear interpolation can be performed between the multiple different alternative vectors to obtain the target joint representation vector.

[0075] For example, when the target style is "artificial mountain," the joint reconstruction model should first be trained based on the feature data of the fake 3D model to obtain joint representation vectors related to artificial mountains. When the 3D model of the preset category to be generated is "artificial mountain," at least one joint representation vector can be selected from the existing joint representation vectors related to artificial mountains. The existing joint representation vectors related to the artificial mountain style are used as backup vectors. When one backup vector is selected, it is directly used as the target joint representation vector, and the newly generated 3D model of the preset category is the same as a historical artificial mountain 3D model. When two backup vectors are selected, linear interpolation can be performed on the backup vectors to obtain the target joint representation vector. For example, if backup vector 1 has a spherical shape and backup vector 2 has a triangular pyramidal shape, linear interpolation of the two backup vectors can yield an artificial mountain 3D model with a shape between a triangular pyramid and a sphere. The generated artificial mountain 3D model is a reasonable model, not meaningless noise. Interpolation of the two backup vectors can obtain a sequence of smooth transitions between the two images.

[0076] As demonstrated in this embodiment, after training the joint reconstruction model based on 3D model data of a certain preset category, a vector space corresponding to that preset category can be obtained. Existing joint representation vectors can then be directly obtained from this vector space. By modifying or directly using the obtained joint representation vectors as needed, a target joint representation vector can be obtained for use. After the joint reconstruction model is completed, a large number of new joint representation vectors can be obtained by modifying existing joint representation vectors, improving the generation speed and efficiency of joint representation vectors. This can be used to generate a large number of 3D models of the same preset category (with similar styles) but with differences, significantly improving the efficiency of users creating 3D models of the same category.

[0077] In another optional embodiment, the method for generating the joint representation vector of the target includes: obtaining an initial 3D model corresponding to a target 3D model of a preset category; extracting the feature data corresponding to the initial 3D model in a sparse voxel feature field to obtain target feature data; and inputting the target feature data into a trained first sub-model to generate the joint representation vector of the target.

[0078] The initial 3D model is the initial model of the target 3D model for a preset category. That is, it is a preliminary, unmodified, or poorly modified 3D model. Specifically, the initial 3D model can be a 3D model with only geometric structure and no color, or a 3D model that is initially completed but whose geometric structure does not meet the standards corresponding to the preset category. This initial 3D model can be mapped to corresponding feature data in a sparse voxel feature field, and the target feature data corresponding to the initial 3D model in the sparse voxel feature field can be extracted. After training, the first sub-model can generate a joint representation vector that accurately represents the structure of the original 3D model. Therefore, the target feature data corresponding to the initial 3D model can also be input into the first sub-model to generate the target joint representation vector.

[0079] In this embodiment, the target joint representation vector is obtained based on the initial 3D model provided by the user. This initial 3D model is a preliminary 3D model that has not been modified in terms of geometry and / or color, or the modification is not up to standard. Since the first sub-model has learned the relationship between geometry and color in the preset category, that is, it has learned how color is attached to geometry in the preset category, including the adjustment of geometry itself, the first sub-model can "modify" the target feature data of the initial 3D model to obtain a target joint representation vector that is highly related to the preset category. This can improve the rationality of the geometric structure of the generated target 3D model. By randomly sampling a target joint representation vector, the second sub-model can generate a brand new and reasonable 3D model.

[0080] As described in step S102, the second sub-model in the trained joint reconstruction model can learn how to perform high-precision reconstruction of the joint representation vector. Therefore, when the target joint representation vector related to the 3D model of the preset category is input into the trained second sub-model, it can reconstruct multiple related target geometric features.

[0081] Furthermore, a joint representation model of the target can be generated based on the relevant textual description of the target 3D model. This invention does not limit the method of obtaining the joint representation model of the target.

[0082] S104. Generate a 3D model of the target based on the target's geometric features and sparse voxel feature field.

[0083] Each target geometric feature corresponds to a voxel vertex. By mapping the target geometric features to a sparse voxel feature field, multiple grid points on the surface of the target 3D model are obtained. The initial color features of the voxel vertices in the sparse voxel feature field can be used to construct the color features of each grid point on the surface of the target 3D model, thereby realizing the construction of the geometric structure of the 3D model of the preset category and coloring the surface of the geometric structure to obtain the 3D model of the preset category.

[0084] In an optional embodiment, the target geometric features also include directed distance function values. When the voxel resolution is high (the number of voxels is sufficient), voxel vertices with a directed distance function value of 0 can be selected as reference vertices of the target 3D model of a preset category. These reference vertices are grid points that occupy the surface of the target 3D model. The color features of other grid points between the reference vertices are obtained by interpolation using the initial color features of the reference vertices. This enables accurate coloring of the entire target 3D model of the preset category, and the colors of adjacent grid points can transition smoothly, resulting in a target 3D model of the preset category with uniform and accurate colors.

[0085] The 3D model generation method provided in this invention performs sparse voxel feature sampling on a 3D model of a preset category in a sparse voxel feature field. The initial geometric features and initial color features of each voxel vertex are interrelated. When training the joint reconstruction model based on the sparse voxel feature field, the first sub-model can learn the joint representation of geometric and color features related to the preset category, obtaining a joint representation vector related to the preset category. After training the joint reconstruction model, the first sub-model in the joint reconstruction model can be used to generate different joint representation vectors as needed. The joint representation vector enables high coupling and alignment of geometric and color features. When using the joint representation vector to construct a 3D model, color features can be bound to fixed points on the model body, better adapting to various dynamic deformations of the 3D model and avoiding the seams and distortion problems caused by stretching traditional textures, thus ensuring visual quality under dynamic performance. Secondly, once the joint reconstruction model is trained, it is only necessary to obtain the joint representation vector of the target 3D model related to the preset category. The joint representation vector can then be input into the second sub-model of the joint reconstruction model to generate target geometric features for constructing the target 3D model of the preset category. Based on the generated target geometric features and sparse voxel feature field, the target 3D model is constructed. Users can obtain or generate different joint representation vectors to automatically generate high-quality 3D models with integrated shape and color without the need for secondary coloring. This greatly improves the efficiency of 3D model production and reduces manpower and time costs.

[0086] In an optional embodiment, a joint reconstruction model is trained based on all feature data to obtain the trained joint reconstruction model, including:

[0087] The initial geometric and color features from all feature data are input into the first sub-model of the joint reconstruction model to obtain the joint representation vector. The joint representation vector is then input into the second sub-model of the joint reconstruction model to obtain the reconstructed predicted geometric and color features. The joint loss is calculated based on the differences between the predicted geometric features and the initial geometric features, and the differences between the predicted color features and the initial color features. The parameters of the first and second sub-models are optimized to ensure that the joint loss is minimized, thus obtaining the trained joint reconstruction model.

[0088] Minimizing the joint loss typically means that the gradient change of the joint loss is less than a preset change magnitude, and / or that the joint loss is less than a preset loss threshold.

[0089] By training the joint reconstruction model, the first sub-model is required to generate accurate joint representation vectors related to the preset category, and the second sub-model is required to understand the structural data (i.e. feature data) of the 3D model represented by the joint representation vector. After training, the first sub-model can learn to understand the relationship between geometric features and color features, that is, learn how color is attached to the geometric structure in this type of 3D model; the second sub-model can learn how to reconstruct the joint representation vector with high accuracy.

[0090] The joint loss is calculated based on the differences between the predicted geometric features and the initial geometric features, and the differences between the predicted color features and the initial color features. This includes: determining the geometric reconstruction loss based on the differences between the predicted geometric features and the initial geometric features; determining the color reconstruction loss based on the differences between the predicted color features and the initial color features; and calculating the joint loss based on the geometric reconstruction loss, the color reconstruction loss, and preset weights.

[0091] The learning of a joint reconstruction model requires a "scoring criterion," which is the loss function (i.e., joint loss). The geometric reconstruction loss measures the difference between the predicted geometric features (predicted geometric features) and the true geometric features (initial geometric features). The color reconstruction loss measures the difference between the predicted color features (predicted color features) and the true color features (initial color features).

[0092] In an optional embodiment, the first sub-model is an encoder and the second sub-model is a decoder; the encoder is used to compress feature data into the latent space to obtain a joint representation vector; the decoder is used to decompress the joint representation vector to obtain predicted geometric features and color features.

[0093] The joint reconstruction model in this embodiment is a variational autoencoder (VAE), a generative model characterized by its continuous and structured latent space. After training, the VAE learns to map input data (such as a 3D model) to a smooth latent space (encoding) and can sample from any point in this space to generate a reasonable new data (decoding).

[0094] The training and use of the model includes an encoding process (i.e., a compression process) and a decoding process (i.e., a decompression process).

[0095] Encoding Process: A sparse voxel feature field corresponding to a model is taken as input and fed into the encoder part of the variational autoencoder. The encoder consists of a series of sparse convolutional layers. Convolution is a common operation used in AI for feature extraction, and sparse convolution is a variant specifically designed for handling sparse data. It only performs calculations on those activated voxels, maintaining efficiency. Through layers of convolution, the original complex data field is continuously condensed and refined, ultimately compressed into a fixed-length (dimensional) "latent vector z," which is the joint representation vector described in this paper. This vector z can be seen as the "gene" or "soul" of the 3D model, encoding all the geometric and color information of the model in a highly abstract way. The joint representation vector is a low-dimensional vector (e.g., 512 numbers), which is a highly abstract and compressed representation of the original 3D data. It captures the core features of the object, such as category, pose, size, and scale, but is no longer a specific point cloud or mesh.

[0096] Decoding process: The joint representation vector z is fed into the decoder. The decoder's task is to reverse the process, restoring this condensed "gene" into a complete feature field. The decoder structure is typically symmetrical to the encoder, using "sparse deconvolution layers" to progressively amplify features and restore details.

[0097] The purpose of encoding before decoding is:

[0098] The encoding process forces the model to ignore redundant details and noise, seeking and extracting the most essential and discriminative features from the data. For example, for a picture of a cat, the encoder learns high-level abstract features such as "has ears," "has whiskers," and "fur texture," rather than the precise value of a specific pixel. The compressed data is no longer an image, but a low-dimensional, dense feature vector, i.e., a joint representation vector, where all data is mapped to a point in the vector space.

[0099] The decoding process forces the model to reconstruct an output that is as close as possible to the original data based on the learned essential features. In this process, it learns how these high-level features combine to form the patterns and relationships in the specific image we see.

[0100] In this context, the vector space is the latent space, which is a semantic space, not a pixel space. In this space, semantically similar samples are grouped close together. For example, the joint representation vectors of all "cars" will cluster in one region, while the latent joint representation vectors of all "cats" will cluster in another region.

[0101] In an optional embodiment, the geometric features include a directed distance function value and a deformation vector. The decoder includes multiple output heads, including an occupancy head, a first geometric head, a second geometric head, and a color head. The occupancy head is used to define whether a voxel position belongs to a 3D model entity. The first geometric head is used to predict the directed distance function value of the voxel vertex. The second geometric head is used to predict the deformation vector of the voxel vertex, and the deformation vector is used to adjust the position of the corresponding mesh point when calculating the directed distance function value. The color head is used to predict the color features of the voxel vertex.

[0102] In the final layer of the decoder, multiple output heads are used. Through a splitting structure, the final feature map output by the decoder is not used for a single task, but is distributed to multiple parallel, small "prediction heads." Each head is one or more simple network layers, performing a single function. This multi-head design allows the model to learn and reconstruct information from different dimensions in a decoupled and more focused manner, avoiding mutual interference between different tasks. This is key to achieving high-quality joint generation of geometry and color.

[0103] It should be noted that when performing voxelization to obtain multiple voxels, the spatial relationship between each voxel and the surface of the 3D model can be determined by a preset voxelization algorithm. The spatial relationship is either occupied or unoccupied. The spatial relationship of voxels containing points on the surface of the 3D model is occupied, while the spatial relationship of the remaining voxels is unoccupied. In this step, the occupancy header can also use this information to define whether the voxel position belongs to the 3D model entity.

[0104] In an optional embodiment, the joint loss is expressed as:

[0105] L_total = λ_geom * L_geom + λ_color * L_color + β * L_KL

[0106] Where L_total is the joint loss, L_geom and λ_geom are the geometric reconstruction loss and their corresponding hyperparameter weights, L_color and λ_color are the color reconstruction loss and their corresponding hyperparameter weights, and L_KL and β are the information divergence loss and their corresponding hyperparameter weights. The information divergence loss is used to regularize the latent space where the joint representation vector is located.

[0107] KL Divergence Loss (L_KL): This is a unique part of the Variational Autoencoder (VAE) model. It ensures that the joint representation vectors in the vector space are regularly distributed (usually a standard normal distribution), which allows us to easily sample joint representation vectors from this space to create new models later. Joint Optimization: We sum the three parts of the loss in a weighted manner to obtain a total "error score". Then, using the backpropagation algorithm and gradient descent optimizer (such as Adam), we continuously adjust all the parameters inside the VAE model, with the goal of making this total score smaller and smaller. Through joint optimization, the joint reconstruction model can learn to understand the complex relationship between geometry and color. For example, the joint reconstruction model can learn deep-level model-making rules such as "the color may change due to wear at the sharp edges of objects".

[0108] In an optional embodiment, the second color feature is a color feature code, and the generation of the target 3D model based on the target geometric features and sparse voxel feature field includes:

[0109] Construct a geometric mesh for the target 3D model based on all target geometric features; for each mesh point on the geometric mesh, map the mesh point to a sparse voxel feature field; for each mesh point, use the voxel surrounding the mesh point as the target voxel; determine the color features of the mesh point based on the initial color features of all voxel vertices in the target voxel; when all mesh points in the geometric mesh have been traversed, use the geometric mesh as the target 3D model.

[0110] The target geometric features represent the geometric structure of the target 3D model, and thus the geometric mesh of the target 3D model can be constructed based on all the target geometric features.

[0111] Since a sparse voxel feature field has already been constructed, grid points on the geometric mesh can be mapped to the sparse voxel feature field. The color features of the grid points are then set based on the initial color features of the neighboring voxel vertices in the sparse voxel feature field. The voxels surrounding the grid points are used as target voxels, and the grid points are colored based on the initial color features of all voxel vertices in the target voxels. Once all grid points are colored, the target 3D model is obtained.

[0112] In an optional embodiment, the target geometric features include directed distance function values. Constructing a geometric mesh of the target 3D model based on all the target geometric features includes: identifying target geometric features with directed distance function values ​​of 0 as solid geometric features; and constructing a geometric mesh of the target 3D model based on the solid geometric features.

[0113] This embodiment describes the process of constructing a geometric mesh for a 3D model. In this embodiment, only the geometric information (SDF and deformation field) is considered, which can be achieved using the Marching Cubes algorithm. This algorithm can very efficiently detect isosurfaces with an SDF value of 0 and convert them into a continuous, smooth 3D geometric mesh composed of a large number of tiny triangular facets. Thus, a pure, high-precision "white model" is created. It is guaranteed to be watertight, without any holes, which is crucial for subsequent applications such as game engines.

[0114] In an optional embodiment, the initial color features and the color features of the grid points are of the type of color feature codes. After determining the color features of the grid points based on the initial color features of all voxel vertices in the target voxel, the method further includes: converting the color features of the grid points into RGB vectors.

[0115] The color features of the grid points are converted into RGB vectors using a color conversion model, which can be found in the description of the color conversion model in S101. After converting the color features of the grid points into RGB vectors, the grid points can be colored based on these RGB vectors.

[0116] In an optional embodiment, determining the color features of a grid point based on the initial color features of all voxel vertices in the target voxel includes: constructing a normalized cubic coordinate system based on the target voxel for each grid point; determining the relative coordinates of the grid point in the cubic coordinate system; determining the weight of each voxel vertex in the target voxel based on the relative coordinates; and interpolating the initial color features of each voxel vertex based on the weights to obtain the color features of the grid point.

[0117] This embodiment describes the process of coloring a 3D model. For each grid point, we obtain its 3D coordinates and map them to a pre-generated sparse color feature code field. This point may not fall exactly on a voxel vertex, but rather inside a voxel cube. Therefore, we find the eight voxel vertices surrounding it and read their respective color feature codes. Then, we use the trilinear interpolation algorithm to smoothly blend these eight color feature codes based on the point's relative position within the cube, calculating a precise color feature code specific to that grid point. Trilinear interpolation is crucial for ensuring a smooth color transition on the model surface. It should be noted that calculating color feature codes using trilinear interpolation is merely an example and does not constitute a limitation on the color feature code calculation method in this invention.

[0118] The "build geometry first, then interpolate and color" process in this embodiment cleverly solves the alignment problem between color and geometry. Because the color of the mesh points is calculated directly on the final high-precision geometric surface, it can perfectly fit every bump and detail, without the stretching and misalignment problems of traditional texture mapping.

[0119] In an optional embodiment, determining the weight of each voxel vertex in the target voxel based on relative coordinates includes:

[0120] Obtain the relative coordinates of the grid points in the local coordinate system of the target voxel; based on the spatial geometric relationship between the relative coordinates and each vertex of the target voxel, determine the weight of the grid point relative to each voxel vertex, where the weight is negatively correlated with the distance from the grid point to each voxel vertex. That is, the farther the distance, the smaller the weight, and the greater the distance, the greater the weight. For example, if the relative coordinates of the grid point in the normalized cubic coordinate system are (u, v, w), then its weight with respect to the 8 vertices can be determined by standard trilinear interpolation weight formulas such as (1-u)(1-v)(1-w) and u(1-v)(1-w).

[0121] In an optional embodiment, the initial color features of each voxel vertex are interpolated based on weights to obtain the color features of the grid points. This includes: weighted summation of the initial color features of each voxel vertex based on its weight to obtain the color features of the grid points. This ensures that the color features of each grid point can obtain a color (the RGB color corresponding to the color feature) that transitions smoothly with the surrounding colors based on its relative position in the voxel.

[0122] To clearly illustrate the 3D model generation method and its beneficial effects of the present invention, the following examples are provided.

[0123] 3D model generation methods consist of three stages:

[0124] Phase 1: Data preprocessing and representation.

[0125] Phase Two: Model Training.

[0126] Phase 3: Model Reasoning and Generation.

[0127] Figure 2 This is a flowchart illustrating the data preprocessing, representation, and model training stages provided in an embodiment of the present invention. Figure 2 (a) in the diagram is a flowchart of stage one, as shown below. Figure 2 As shown in (a), in stage one, the original 3D model (with texture / vertex color) is first obtained, the original 3D model is sparsed voxelized, and SDF, deformation vector and RGB vector are extracted. Among them, SDF and deformation vector are geometric features, and RGB vector is converted by color autoencoder to obtain color features. Then, the geometric features and color features are associated and merged to obtain the sparse voxel feature field.

[0128] Figure 2 (b) in the diagram is a flowchart of stage two, as shown below. Figure 2 As shown in (b), in stage two, the data in the sparse voxel feature field is input into the encoder for compression to obtain a joint representation vector. Then, the decoder decompresses the joint representation vector to obtain the SDF, deformed vector, and RGB vector, thus obtaining the reconstructed sparse voxel feature field. The reconstructed feature field is compared with the initial feature field to obtain the joint loss, and the parameters of the encoder and decoder are optimized based on the joint loss.

[0129] Figure 3 This is a flowchart illustrating the reasoning and generation stages of a model, such as... Figure 3 As shown, in stage three, a joint representation vector is obtained through sampling or input. The joint representation vector is then decoded by a decoder to obtain the predicted geometric features. A geometric network is generated based on the geometric features. Vertex colors are applied based on the existing color feature codes and the geometric network. Specifically, this includes color interpolation of the grid points. The color features obtained by interpolation are then converted into RGB by a color decoder to obtain a 3D model with vertex colors.

[0130] After completing all the above steps, a brand new, geometrically detailed, naturally colored, and perfectly integrated high-quality 3D model will be successfully generated.

[0131] When faced with the "shape-color mismatch" problem using the 3D model generation method described in this example, those skilled in the art would typically consider improving the UV unwrapping algorithm, optimizing the texture mapping function, or making corrections at the rendering level. This invention takes a different approach, addressing the problem at the most fundamental data representation layer by designing a multimodal "sparse voxel feature field" to fundamentally ensure coupling. This approach of solving the problem at its source is not immediately obvious to those skilled in the art.

[0132] Furthermore, the synergistic effect of the technical elements in the 3D model generation method of this embodiment produced unexpected technical effects: sparse voxels (high efficiency) + color feature codes (smooth and continuous) + multi-head decoding (decoupling and focus) + trilinear interpolation (precise mapping). The organic combination of these technical elements together achieved the effect of "generating seamless, smooth, and aligned vertex colors while maintaining extremely high geometric detail accuracy." This effect cannot be predicted by simply adding up the individual technical points, but is the result of "1+1>2" produced by their synergistic work.

[0133] Compared to the traditional "geometry first, texture later" workflow, this invention significantly improves production efficiency and fundamentally avoids texture stretching and seam problems under dynamic deformation. Compared to other joint generation methods, this invention, through its unique workflow, achieves significant progress in the accuracy of geometry-color alignment, enabling the generation of higher-fidelity 3D assets. Furthermore, this invention overcomes the long-standing industry bias that "high-quality textures must rely on 2D mapping and UVs," proving that vertex color-based generation paths can achieve results comparable to traditional methods in specific scenarios (especially stylized assets and assets requiring extensive dynamic deformation).

[0134] In a specific application, the 3D model generation method of this invention can be applied to Chinese-style games to quickly generate scene props or weapon equipment with a unified style and rich details. Suppose the art team needs to design a series of "Bronze Age" themed weapons for a new version, including swords, halberds, and tripods. These weapons have similar bronze materials and rust-covered color styles, but their geometric shapes differ. For details, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of different 3D models of a preset category provided in an embodiment of the present invention. Here, the preset category is bronze material.

[0135] Implementation process:

[0136] 1. Data preparation stage (corresponding to stage one):

[0137] First, a color autoencoder is trained. A large number of bronze artifact photographs are collected, and the colors (from bright bronze to dark green rust) are extracted. These RGB values ​​are used to train the color autoencoder, resulting in a latent space that can represent the "bronze color spectrum". Then, a batch of existing textured 3D models of traditional Chinese weapons are prepared as the training set.

[0138] For each model, a data preprocessing pipeline is run: sparse voxel sampling is performed near the model surface. At each grid point, the SDF value is calculated. At each grid point, its original texture color is sampled and converted into a Color_feature_code (e.g., an 8-dimensional vector) using a trained color encoder. Finally, each weapon model is transformed into a multimodal sparse voxel feature field.

[0139] 2. Model training phase (corresponding to phase two):

[0140] We input the feature field data of all weapons into our designed Sparconv-VAE model for training.

[0141] The model learns the common geometric paradigms (such as sharp edges and rustic outlines) and common color paradigms (the distribution patterns of bronze color and rust) of these weapons simultaneously through a joint loss function.

[0142] 3. Application Generation Phase (corresponding to Phase Three):

[0143] Rapidly generating new weapons: Now, an artist wants to create a new bronze dagger. Instead of starting from scratch, he can sample from the latent space of our trained model, or input a simple sketch of a dagger concept to guide the generation of a joint representation vector z.

[0144] Decoding and Generation: When z is fed into the decoder, the model immediately generates a predicted sparse feature field containing geometric and color information.

[0145] Geometric Construction: The Marcin cube algorithm extracts a finely shaped "white model" of a dagger from the geometric field. The outline, blood groove, handle, and other structures of this dagger are already formed.

[0146] Coloring: The system automatically executes the vertex color application process. For each vertex of the dagger mesh, its color feature code is obtained from the color field through trilinear interpolation, and then converted into RGB values ​​using a color decoder. As a result, the blade of the dagger exhibits a brighter metallic color, while the grooves and edges naturally generate dark rust, with the color distribution perfectly matching the geometric details.

[0147] Delivery and Iteration: The artist receives a high-quality dagger model with its own base vertex color. They can then make minor tweaks and detail adjustments, or use it directly in the game scene, significantly shortening the production cycle from concept to final asset.

[0148] By applying 3D model generation, this invention transforms the previously time-consuming modeling and texture painting process into a rapid human-computer collaborative generation workflow, greatly improving the efficiency and creativity of game content production.

[0149] Figure 5 This is a schematic diagram of a 3D model generation device provided in an embodiment of the present invention. Figure 5 As shown, the 3D model generation device includes:

[0150] The feature field construction module 100 is used to construct a sparse voxel feature field for a 3D model of a preset category. The sparse voxel feature field includes multiple voxels and feature data of the voxels. The feature data includes the initial geometric features and initial color features of the voxel vertices.

[0151] The model training module 200 is used to train a joint reconstruction model based on the sparse voxel feature field to obtain a trained joint reconstruction model. The joint reconstruction model includes a first sub-model and a second sub-model. The first sub-model is used to generate a joint representation vector related to a preset category based on the input geometric features and color features. The second sub-model is used to reconstruct the joint representation vector into geometric features and color features.

[0152] The target geometric feature acquisition module 300 is used to input the joint representation vector of the target 3D model related to the preset category into the trained second sub-model to generate target geometric features related to the target 3D model.

[0153] The model generation module 400 is used to generate the target 3D model based on the target geometric features and the sparse voxel feature field.

[0154] Optionally, the feature field construction module 100 includes:

[0155] The solid space determination submodule is used to determine the voxelization parameters and the solid space surrounding the 3D model of the preset category;

[0156] A voxelization processing submodule is used to perform voxelization processing in the three-dimensional space according to the voxelization parameters to obtain multiple voxels;

[0157] The surface voxel filtering submodule is used to filter out surface voxels within a preset range from the surface of the 3D model;

[0158] The sampling submodule is used to sample the voxel features of the selected surface voxels to obtain feature data of multiple surface voxels.

[0159] The sparse voxel feature field determination submodule is used to bind surface voxels to their corresponding feature data to obtain a sparse voxel feature field.

[0160] Optionally, the sampling submodule is used to perform the following steps:

[0161] Voxel feature sampling is performed on the selected surface voxels to obtain the directed distance function value, deformation vector and RGB vector of the voxel vertices of all surface voxels;

[0162] For each voxel vertex, the corresponding directed distance function value and deformation vector are used as initial geometric features; the corresponding RGB vector is converted into initial color features, wherein the number of data points in the RGB vector is less than the number of data points in the initial color features;

[0163] For each surface voxel, the initial geometric features and the initial color features of the voxel vertices are associated and used as feature data for the surface voxel.

[0164] Optionally, the model training module 200 includes:

[0165] The joint representation vector construction submodule is used to input the initial geometric features and the initial color features in the sparse voxel feature field into the first sub-model of the joint reconstruction model to obtain the joint representation vector;

[0166] The prediction submodule is used to input the joint representation vector into the second sub-model of the joint reconstruction model to obtain the reconstructed predicted geometric features and predicted color features;

[0167] The loss calculation submodule is used to calculate the joint loss based on the difference between the predicted geometric features and the initial geometric features, and the difference between the predicted color features and the initial color features;

[0168] The optimization submodule is used to optimize the parameters of the first sub-model and the second sub-model to ensure that the joint loss reaches the minimum value, thereby obtaining the trained joint reconstruction model.

[0169] Optionally, the loss calculation submodule is used to perform the following steps:

[0170] The geometric reconstruction loss is determined based on the difference between the predicted geometric features and the initial geometric features;

[0171] The color reconstruction loss is determined based on the difference between the predicted color features and the initial color features;

[0172] The joint loss is calculated based on the geometric reconstruction loss, the color reconstruction loss, and the preset weights.

[0173] Optionally, the first sub-model is an encoder, and the second sub-model is a decoder;

[0174] The encoder is used to compress the feature data into the latent space to obtain a joint representation vector;

[0175] The decoder is used to decompress the joint representation vector to obtain the predicted geometric and color features.

[0176] Optionally, the geometric features include a directed distance function value and a deformation vector, and the decoder includes at least one of a plurality of output heads, the output heads including an occupancy head, a first geometric head, a second geometric head, and a color head;

[0177] The occupancy header is used to define whether a voxel location belongs to a 3D model entity;

[0178] The first geometric head is used to predict the directed distance function value of the voxel vertex;

[0179] The second geometric head is used to predict the deformation vector of the voxel vertex, and the deformation vector is used to adjust the position of the corresponding grid point when calculating the directed distance function value;

[0180] The color head is used to predict the color features of voxel vertices.

[0181] Optionally, the device further includes:

[0182] The storage module is used to store the generated joint representation vectors in the vector space of the joint reconstruction model, and each vector space corresponds to a preset category.

[0183] Optionally, the method for generating the joint representation vector of the target includes:

[0184] Obtain a joint representation vector from the vector space corresponding to the preset category to obtain a backup vector;

[0185] Generate a joint representation vector of the target based on the backup vector.

[0186] Optionally, generating the target joint representation vector based on the backup vector includes:

[0187] Linear interpolation is performed on the backup vector to obtain the target joint representation vector.

[0188] Optionally, the method for generating the joint representation vector of the target may further include:

[0189] Obtain the initial 3D model corresponding to the target 3D model of the preset category;

[0190] Extract the feature data corresponding to the initial 3D model in the sparse voxel feature field to obtain the target feature data;

[0191] The target feature data is input into the trained first sub-model to generate a joint target representation vector.

[0192] Optionally, the model generation module 400 includes:

[0193] The geometric mesh construction submodule is used to construct the geometric mesh of the target 3D model based on all the target geometric features;

[0194] The grid point mapping submodule is used to map each grid point on the geometric grid to the sparse voxel feature field.

[0195] The target voxel determination submodule is used to identify the voxel surrounding the grid point as the target voxel for each grid point.

[0196] The color feature determination submodule is used to determine the color features of the grid points based on the initial color features of all voxel vertices in the target voxel;

[0197] The target 3D model determination submodule is used to determine the geometric mesh as the target 3D model after all the mesh points in the geometric mesh have been traversed.

[0198] Optionally, the target geometric features include directed distance function values, and the geometric mesh construction submodule is used to perform the following steps:

[0199] The target geometric features with a directed distance function value of 0 are determined as entity geometric features;

[0200] The geometric mesh of the target 3D model is constructed based on the geometric features of the entity.

[0201] Optionally, the initial color features and the color features of the grid points are of the type of color feature codes, and the model generation module 400 further includes:

[0202] The color feature conversion submodule is used to convert the color features of the grid points into RGB vectors.

[0203] Optionally, the color feature determination submodule includes:

[0204] A coordinate system construction unit is used to construct a normalized cubic coordinate system based on the target voxel for each of the grid points;

[0205] A relative coordinate determination unit is used to determine the relative coordinates of the grid points in the cubic coordinate system;

[0206] A weight determination unit is used to determine the weight of each voxel vertex in the target voxel based on the relative coordinates;

[0207] A color feature determination unit is used to interpolate the initial color features of each voxel vertex based on the weights to obtain the color features of the grid points.

[0208] Optionally, the weight determination unit is used to perform the following steps:

[0209] Obtain the relative coordinates of the grid points in the local coordinate system of the target voxel;

[0210] Based on the spatial geometric relationship between the relative coordinates and each vertex of the target voxel, the weight of the grid point relative to each voxel vertex is determined, wherein the weight is negatively correlated with the distance from the grid point to each voxel vertex.

[0211] Optionally, the color feature determination unit is used to perform the following steps:

[0212] The initial color features of each voxel vertex are weighted and summed based on the weight of each voxel vertex to obtain the color features of the grid points.

[0213] The 3D model generation apparatus provided in this embodiment of the invention can execute the 3D model generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0214] Figure 6 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0215] like Figure 6As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0216] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0217] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as 3D model generation methods.

[0218] In some embodiments, the 3D model generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the 3D model generation method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the 3D model generation method by any other suitable means (e.g., by means of firmware).

[0219] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0220] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0221] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0222] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a monitor with a cathode ray tube or liquid crystal display); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0223] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0224] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0225] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0226] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating 3D models, characterized in that, include: A sparse voxel feature field is constructed for a 3D model of a preset category. The sparse voxel feature field includes multiple voxels and voxel feature data. The feature data includes the initial geometric features and initial color features of the voxel vertices. The joint reconstruction model is trained based on the sparse voxel feature field to obtain the trained joint reconstruction model. The joint reconstruction model includes a first sub-model and a second sub-model. The first sub-model is used to generate a joint representation vector related to a preset category based on the input geometric features and color features. The second sub-model is used to reconstruct the joint representation vector into geometric features and color features. Input the joint representation vector of the target associated with the target 3D model of the preset category into the trained second sub-model to generate the target geometric features associated with the target 3D model; The target 3D model is generated based on the target's geometric features and the sparse voxel feature field.

2. The method as described in claim 1, characterized in that, The construction of a sparse voxel feature field for a 3D model of a preset category, and the acquisition of feature data for all voxels in the sparse voxel feature field, includes: Determine the voxelization parameters and the solid space surrounding the 3D model of the preset category; Voxelization is performed in the three-dimensional space according to the voxelization parameters to obtain multiple voxels; Filter out surface voxels within a preset range from the surface of the 3D model; Voxel feature sampling is performed on the selected surface voxels to obtain feature data of multiple surface voxels; By binding the surface voxels to their corresponding feature data, a sparse voxel feature field is obtained.

3. The method as described in claim 2, characterized in that, The process of sampling voxel features from the selected surface voxels yields feature data for multiple surface voxels, including: Voxel feature sampling is performed on the selected surface voxels to obtain the directed distance function value, deformation vector and RGB vector of the voxel vertices of all surface voxels; For each voxel vertex, the corresponding directed distance function value and deformation vector are used as initial geometric features; the corresponding RGB vector is converted into initial color features, wherein the number of data points in the RGB vector is less than the number of data points in the initial color features; For each surface voxel, the initial geometric features and the initial color features of the voxel vertices are associated and used as feature data for the surface voxel.

4. The method as described in claim 1, characterized in that, The step of training a joint reconstruction model based on the sparse voxel feature field to obtain the trained joint reconstruction model includes: The initial geometric features and the initial color features in the sparse voxel feature field are input into the first sub-model of the joint reconstruction model to obtain the joint representation vector; The joint representation vector is input into the second sub-model of the joint reconstruction model to obtain the reconstructed predicted geometric features and predicted color features; Calculate the joint loss based on the difference between the predicted geometric features and the initial geometric features, and the difference between the predicted color features and the initial color features; The parameters of the first sub-model and the second sub-model are optimized to ensure that the joint loss is minimized, thereby obtaining the trained joint reconstruction model.

5. The method as described in claim 4, characterized in that, The calculation of the joint loss based on the difference between the predicted geometric features and the initial geometric features, and the difference between the predicted color features and the initial color features, includes: The geometric reconstruction loss is determined based on the difference between the predicted geometric features and the initial geometric features; The color reconstruction loss is determined based on the difference between the predicted color features and the initial color features; The joint loss is calculated based on the geometric reconstruction loss, the color reconstruction loss, and the preset weights.

6. The method as described in claim 1, characterized in that, The first sub-model is an encoder, and the second sub-model is a decoder; The encoder is used to compress the feature data into the latent space to obtain a joint representation vector; The decoder is used to decompress the joint representation vector to obtain the predicted geometric and color features.

7. The method as described in claim 6, characterized in that, The geometric features include a directed distance function value and a deformation vector, and the decoder includes at least one of a plurality of output heads, the output heads including an occupancy head, a first geometric head, a second geometric head, and a color head; The occupancy header is used to define whether a voxel location belongs to a 3D model entity; The first geometric head is used to predict the directed distance function value of the voxel vertex; The second geometric head is used to predict the deformation vector of the voxel vertex, and the deformation vector is used to adjust the position of the corresponding grid point when calculating the directed distance function value; The color head is used to predict the color features of voxel vertices.

8. The method according to any one of claims 1-7, characterized in that, Also includes: The generated joint representation vectors are stored in the vector space of the joint reconstruction model, and each vector space corresponds to a preset category.

9. The method as described in claim 8, characterized in that, The methods for generating the joint representation vector of the target include: Obtain a joint representation vector from the vector space corresponding to the preset category to obtain a backup vector; Generate the target joint representation vector based on the backup vector.

10. The method as described in claim 9, characterized in that, The step of generating the target joint representation vector based on the backup vector includes: Linear interpolation is performed on the backup vector to obtain the target joint representation vector.

11. The method according to any one of claims 1-7, characterized in that, The method for generating the joint representation vector of the target also includes: Obtain the initial 3D model corresponding to the target 3D model of the preset category; Extract the feature data corresponding to the initial 3D model in the sparse voxel feature field to obtain the target feature data; The target feature data is input into the trained first sub-model to generate a joint target representation vector.

12. The method according to any one of claims 1-7, characterized in that, The step of generating the target 3D model based on the target's geometric features and sparse voxel feature field includes: Construct the geometric mesh of the target 3D model based on all the target geometric features; For each grid point on the geometric grid, the grid point is mapped to the sparse voxel feature field; For each of the grid points, the voxel surrounding the grid point is taken as the target voxel; The color features of the grid points are determined based on the initial color features of all voxel vertices in the target voxel; Once all the grid points in the geometric grid have been traversed, the geometric grid is used as the target 3D model.

13. The method as described in claim 12, characterized in that, The target geometric features include directed distance function values, and the construction of the geometric mesh of the target 3D model based on all the target geometric features includes: The target geometric features with a directed distance function value of 0 are determined as entity geometric features; The geometric mesh of the target 3D model is constructed based on the geometric features of the entity.

14. The method as described in claim 12, characterized in that, The initial color feature and the color feature of the grid point are of the type of color feature code. After determining the color feature of the grid point based on the initial color feature of all voxel vertices in the target voxel, the method further includes: The color features of the grid points are converted into RGB vectors.

15. The method as described in claim 12, characterized in that, Determining the color features of the grid points based on the initial color features of all voxel vertices in the target voxel includes: For each of the grid points, a normalized cubic coordinate system is constructed based on the target voxel; Determine the relative coordinates of the grid points in the cube coordinate system; The weight of each voxel vertex in the target voxel is determined based on the relative coordinates. The initial color features of each voxel vertex are interpolated based on the weights to obtain the color features of the grid points.

16. The method as described in claim 15, characterized in that, The step of determining the weight of each voxel vertex in the target voxel based on the relative coordinates includes: Obtain the relative coordinates of the grid points in the local coordinate system of the target voxel; Based on the spatial geometric relationship between the relative coordinates and each vertex of the target voxel, the weight of the grid point relative to each voxel vertex is determined, wherein the weight is negatively correlated with the distance from the grid point to each voxel vertex.

17. The method as described in claim 15, characterized in that, The process of interpolating the initial color features of each voxel vertex based on the weights to obtain the color features of the grid points includes: The initial color features of each voxel vertex are weighted and summed based on the weight of each voxel vertex to obtain the color features of the grid points.

18. A 3D model generation device, characterized in that, include: The feature field construction module is used to construct a sparse voxel feature field for a 3D model of a preset category. The sparse voxel feature field includes multiple voxels and feature data of the voxels. The feature data includes the initial geometric features and initial color features of the voxel vertices. The model training module is used to train a joint reconstruction model based on the sparse voxel feature field to obtain the trained joint reconstruction model. The joint reconstruction model includes a first sub-model and a second sub-model. The first sub-model is used to generate a joint representation vector related to a preset category based on the input geometric features and color features. The second sub-model is used to reconstruct the joint representation vector into geometric features and color features. The target geometric feature acquisition module is used to input the joint representation vector of the target 3D model related to the preset category into the trained second sub-model to generate target geometric features related to the target 3D model. The model generation module is used to generate the target 3D model based on the target geometric features and the sparse voxel feature field.

19. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the 3D model generation method according to any one of claims 1-17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the 3D model generation method according to any one of claims 1-17.

Images