Text-to-three-dimensional object generation method based on dual color consistency matching

Abstract

The invention discloses a text-to-three-dimensional object (3D) generation method based on dual color consistency matching, and aims to solve the problem of 3D asset quality reduction caused by multi-view color inconsistency in the prior art. According to the method, in each training iteration, a group of multi-view images are firstly rendered from a 3D model, and a high-fidelity forward view is specified as a color reference; subsequently, optimization is carried out through a double matching mechanism: 1, matching among images: aligning colors of other views to the reference view by using depth semantic features, thereby breaking circulation of error accumulation; 2, image internal matching: in a single image, color and geometric smoothness in the same semantic region are standardized by improving total variation loss; through the dual mechanism, accumulation and propagation of unnatural color distortion are effectively prevented, color inconsistency is reduced, and the visual fidelity and color consistency of the generated 3D model are remarkably improved.

Description

Technical Field

[0001] This invention relates to a text-to-3D object generation method for high-fidelity 3D content generation, specifically a text-to-3D object generation method based on dual color consistency matching. Background Technology

[0002] 3D object modeling is widely used in the game and film visual effects industries, and related professional software such as Unreal Engine and Maya are also widely used in these industries, greatly improving the efficiency of 3D object modeling. With the development of virtual reality technology and metaverse technology, the market demand for 3D object modeling is further increasing. Traditional modeling processes have a high learning curve and require a significant amount of time and effort from artists. Therefore, designing an auxiliary tool that can lower the barrier to entry and further assist technical personnel in improving productivity is particularly important.

[0003] In recent years, the emergence of large-scale text-to-image diffusion models, such as the Stable Diffusion (SD) model series, has greatly improved the efficiency of 2D content generation and lowered the learning threshold. This has further driven a paradigm shift in the field of 3D content creation, making it possible to generate 3D assets directly from text descriptions. Techniques such as Score Distillation Sampling (SDS) reduce reliance on large-scale 3D datasets by distilling knowledge from pre-trained 2D text-to-image models, paving a new path for high-fidelity text-to-3D generation.

[0004] Despite significant advancements in existing technologies, a persistent bottleneck severely hinders the generation of high-quality 3D content: color distortion caused by multi-view inconsistency. This problem manifests as unnatural color distributions across different views of a 3D asset; for example, an object may have the expected color on its front, while its sides or back may exhibit unexpected color shifts or blotches, greatly reducing the visual realism and coherence of the generated results.

[0005] Research has revealed that the root cause of this color inconsistency lies in the inherent "error accumulation reinforcement loop" in the SDS optimization process. In each iteration, SDS-based methods generate a pseudo-ground value (pseudo-GT) for the rendered image of a single view to guide the optimization. Once a small color error appears in a view (typically a poorly generated non-frontal view, such as a rear view), the diffusion model generates an equally flawed pseudo-ground value based on this flawed input, even amplifying the error. This misguided gradient is used to optimize the 3D model, causing the error to solidify and propagate. In subsequent iterations, this error is continuously amplified, eventually leading to severe, global color distortion.

[0006] Therefore, how to effectively break this error accumulation cycle and systematically improve the color consistency of multiple views is a key technical problem that urgently needs to be solved in the current field of text-to-3D generation. Summary of the Invention

[0007] In view of this, the present invention discloses a text-to-3D object generation method based on dual color consistency matching to solve the problem of inconsistent colors in multiple views caused by error accumulation in existing text-to-3D generation technologies. This method uses dual consistency color matching to systematically improve the color consistency of multiple views and generate 3D assets with highly consistent colors and high visual fidelity.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] This invention provides a method for generating 3D objects from text based on dual color consistency matching, the specific scheme of which is as follows:

[0010] Step 1: Render multi-view images from the 3D model, including a frontal view and multiple non-frontal views;

[0011] Step 2: Use a visual feature extraction model to extract features and segment semantic regions in the rendered image;

[0012] Step 3: Using the frontal view as a color reference, perform color consistency matching for similar semantic regions between images from non-frontal views;

[0013] Step 4: Perform intra-image color consistency matching for each rendered image to maintain color smoothness within the same semantic region;

[0014] Step 5: Combine the inter-image color consistency loss, intra-image color consistency loss, and fractional distillation sampling loss to jointly optimize the 3D model.

[0015] As a further optimization of the present invention, in step 1, the rendering operation of the 3D model includes the following steps:

[0016] (1) 3DGS is used as the three-dimensional representation of the 3D model to take advantage of its efficient rendering capability and high-fidelity representation capability; based on the text description of the 3D model features input by the user, a pre-trained text-to-point cloud generation model is called to generate a preliminary three-dimensional point cloud distribution that conforms to the text description. The point cloud distribution contains the initial geometric skeleton and basic color information of the object in three-dimensional space; the generated point cloud distribution is used as the basis to initialize the learnable parameters of the 3DGS model.

[0017] (2) During the optimization process, a set of multi-view images is rendered based on the 3D model. The set of multi-view images includes at least one reference view image as a color reference benchmark and at least one non-reference view image. The reference view is preset to generate a view with more stable quality and higher fidelity (such as a frontal view) as an anchor image for color consistency matching. The other views are randomly selected by random sampling.

[0018] As a further optimization of the present invention, step 2, the feature extraction and region segmentation of the rendered image includes the following steps:

[0019] (1) For each rendered image in the multi-view image set The rendering image is subjected to forward propagation inference using a pre-trained Transformer-based visual feature extraction model. The model divides the image into a series of image patches and extracts a high-dimensional, semantically rich visual feature vector for each patch. These visual feature vectors constitute a feature map corresponding to the rendered image. ,

[0020] (2) Regarding the feature map All feature vectors in the image are subjected to nonparametric clustering using the mean-shift clustering algorithm. This clustering process is based on distance or similarity in the feature space, aiming to group nearby feature vectors into the same category. The clustering results segment the rendered image into... Each semantically consistent region with clearly defined boundaries is identified, and a corresponding semantic label graph is generated. .

[0021] As a further optimization of the present invention, in step 3, cross-image color consistency matching includes the following steps:

[0022] (1) In each iteration of optimization, color consistency matching is performed on the rendering images of non-frontal view. First, some patches are randomly selected from the rendering image to be matched, and the patch most similar to it in the anchor image is found. The color distribution difference loss is calculated. ,in This indicates that a patch in the image to be matched is currently undergoing color matching. Indicates color anchoring in the image and The patch with the highest similarity

[0023] (2) In the calculation of color distribution difference loss, the color distribution characteristics of each patch are represented by a discretized LAB color space vector. First, the RGB image is converted into a LAB image, and the L, A, and B channels of the LAB color space are divided into intervals based on experimental experience. By statistically analyzing the LAB color values ​​of each pixel in the patch into the pre-divided intervals, the color frequency distribution of the patch in each interval of the LAB color space is obtained. The discrete LAB color representation constructed in this way is used to calculate the color distribution difference loss across images. ,in This represents the transformation of the RGB color features of a given patch into a discrete LAB color frequency distribution representation, where i represents the corresponding rendered image, and j represents the patch corresponding to the i-th rendered image.

[0024] (3) Introduce reciprocity detection, that is, obtain Then, in reverse order, search for the image to be matched with... Most similar patch Combining the semantic space partitioning results from step 2, if and If the colors are within the same semantic region, such color consistency matches are considered valid. Further expanding the filter, if the center of the patch to be matched is located in the background region, the current patch is skipped. This process ultimately yields the final filter. ,in Indicates the background area. Indicating reciprocity check The semantic region to which it belongs

[0025] (4) Combining the above steps, the final cross-image color consistency matching function is obtained, which includes color matching validity check and LAB color discrete representation: This optimization scheme significantly improves the 3D color consistency and fidelity of generated 3D objects by introducing cross-image color consistency matching and using discretized LAB space vectors that conform to human perception to represent color distribution. Simultaneously, the use of filters not only enhances the effectiveness of cross-image color consistency matching but also accelerates the computation process.

[0026] As a further optimization of the present invention, in step 4, the intra-image color consistency matching includes the following steps:

[0027] (1) Using the total variational loss function, combined with the semantic region division obtained in step 2, perform pixel-level color smoothing on patches of the same semantic region in a single rendered image.

[0028] (2) Further render the depth map obtained from the current viewpoint, and combine it with the semantic region segmentation obtained in step 2 to extend consistency matching to the depth information.

[0029] (3) Combining the above operations with color consistency and depth consistency matching, the final image intra-consistency matching is obtained: ,in and This represents the parameter weights for the corresponding loss calculation. This represents the current rendered image. Represents a semantic graph. A depth map representing the current viewpoint. This indicates that the image-intra-image consistency matching combines corresponding image information and semantic graph information. By introducing the information features extracted in step 2, it improves the anti-color distortion loss function widely used in current technologies, and adjusts the loss calculation according to... By imposing range restrictions, the boundary changes between different semantic regions are preserved while ensuring distortion prevention, thus improving the fidelity of the generated 3D model. At the same time, the loss function is extended to the depth information, further improving the generation quality.

[0030] As a further optimization of the present invention, the 3D model optimization in step 5 includes the following steps:

[0031] (1) Using fractional distillation sampling loss, Gaussian noise is added to the current rendered image to a certain extent according to a preset scheme. The noisy image and the text description provided by the user are input into the text-to-image diffusion model to predict the noise result required for the inversion process. The difference between the predicted noise result and the actual added noise is calculated to obtain the fractional distillation sampling loss.

[0032] (2) Combining the cross-image color consistency matching loss obtained in step 3, the intra-image color consistency matching loss obtained in step 4, and the fractional distillation sampling loss, the three are weighted and used as the total loss. This is then used for backpropagation to optimize the parameters of the 3D representation 3DGS, thereby updating the 3D model. The fractional distillation sampling optimization process is a self-optimizing loop process, but it may also have a cyclic accumulation effect on color errors that occur in the loop, resulting in severe global color inconsistency in the generated 3D model. This invention introduces color consistency matching loss, introducing a correction amount outside the loop process, breaking the error accumulation loop problem in the fractional distillation sampling optimization process, greatly alleviating the color inconsistency problem existing in the current text generation 3D model technology, and improving the fidelity of the final generated 3D model.

[0033] The present invention adopts the above technical solution, and its beneficial effects are as follows: by using the reliability of the forward view as an anchor point, the error accumulation cycle problem in the existing text generation 3D model technology - fractional distillation sampling - is solved, and the color distortion problem of the 3D model is significantly reduced; the introduction of a dual matching mechanism works together to ensure that the color of the 3D model is consistent under different viewpoints, and the transition of color and depth structure is natural in a single viewpoint, which greatly improves the visual realism.

[0034] Mainstream 3D text generation models employ fractional distillation sampling, requiring loss calculations based on pre-trained 2D text-to-image models. This optimized approach can be implemented using current mainstream pre-trained 2D text-to-image models. Compared to some existing improvement techniques, it eliminates the need for training or fine-tuning the pre-trained image generation model based on 3D consistency enhancement. Training and fine-tuning these pre-trained models with 3D data, given the current vastly smaller amount of 3D data compared to pre-trained 2D data, not only increases dataset creation costs but also significantly reduces the diversity of 3D-aware text-to-image models generated. Therefore, this approach maintains a lower deployment cost while preserving the diversity of generated 3D models.

[0035] Meanwhile, all the data required for optimization of the proposed solution can be obtained from the mainstream pipeline of current text-to-3D model generation technology. There is no need to introduce additional user input (such as reference images) as control conditions. It can ensure the fidelity of the generated 3D model without raising the user's threshold for use. It has extremely high plug-and-playability and is easy to integrate into the current technology pipeline.

[0036] This solution effectively solves the color problem in 3D generation, enabling the generation of high-fidelity 3D models with richer details and more realistic visual effects, providing a powerful auxiliary tool for art modelers. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the overall process of the dual color consistency matching method proposed in this invention. Detailed Implementation

[0038] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0039] The text-to-3D generation method based on dual color consistency matching proposed in this invention includes a color consistency matching part and a fractional distillation sampling part. The color matching part, as a pluggable module, can be adapted to all fractional distillation sampling techniques using the same approach. This method can be based on open-source models to achieve training-free, direct optimization-based 3D model generation.

[0040] Example: A schematic diagram of the text-to-3D object generation method based on dual color consistency matching proposed in this invention is shown below. Figure 1 As shown in the figure, this diagram illustrates the general process of directly optimizing and generating a 3D model, mainly consisting of a color consistency matching part and a fractional distillation sampling part. The steps of this invention are described in detail below.

[0041] Experimental conditions: A computer was selected for training, equipped with an Intel(R) processor (2.4GHz) and 128GB of RAM, running Ubuntu 20.04.6, and an NVIDIA RTX 5880Ada (48GB) graphics card; the software environment consisted of the deep learning framework PyTorch 2.0.1.

[0042] Transformers 4.30.2.

[0043] This invention provides a method for generating 3D objects from text based on dual color consistency matching, the method comprising the following steps:

[0044] Step 1: Render multi-view images from the 3D model, including a frontal view and multiple non-frontal views.

[0045] (1) Using 3DGS as the three-dimensional representation of the 3D model, the user-input text description is processed by a pre-trained language-image pre-training (CLIP) model and then provided to the pre-trained generative model Point-E to generate a point cloud distribution that conforms to the text description, thereby initializing the relevant attributes of 3DGS.

[0046] (2) During the optimization process, each specified rendering Images from different perspectives, pre-defined to ensure that at least one frontal view serves as the baseline. The remaining perspectives were randomly selected using random sampling.

[0047] Step 2: Use a visual feature extraction model to extract features and segment semantic regions in the rendered image.

[0048] (1) For each rendered image, the pre-trained DINOv2 model is used to divide the rendered image into non-overlapping 17*17 pixel patches to obtain the visual features of the corresponding patches.

[0049] (2) For all patches in a single rendered image, based on their visual feature vectors, the mean-shift clustering algorithm is used to cluster each patch, dividing it into groups. The semantic label map is obtained by dividing the region into three distinct semantic regions and assigning a semantic label to each pixel. .

[0050] Step 3: Using the frontal view as a color reference, perform color consistency matching for similar semantic regions between images from non-frontal views.

[0051] (1) In each iteration of optimization, color consistency matching is performed on the rendering images of non-frontal views. First, a rendering image to be matched is selected. In the process, establish the features and frontal view anchoring map of all its patches. The similarity matrix of the patch features, and then randomly selected For some patches in the image, find the patch most similar to it in the anchor map and calculate the color distribution difference loss. ,in express A color matching patch is being implemented. Indicates color anchoring in the image and The patch with the highest similarity

[0052] (2) In the calculation of color distribution difference loss, a discretized LAB color space vector is used to represent the color distribution characteristics of each patch. First, the RGB image is converted into a LAB image, and the L, A, and B channels of the LAB color space are divided into 64, 64, and 128 intervals, respectively. For the color distribution representation of the entire patch, frequency is used instead of probability for statistical analysis. By statistically analyzing the LAB color value of each pixel in the patch into the pre-divided intervals, the frequency distribution of color occurrence in each interval of the LAB color space of the patch is obtained. The discrete LAB color representation constructed in this way is used to calculate the color distribution difference loss across images. ,in This represents converting the RGB color representation of a given patch into a discrete LAB color frequency representation, where i represents the corresponding render image, and j represents the patch corresponding to the i-th render image.

[0053] (3) Introduce a reciprocity check, i.e., have obtained In reverse, search for the image to be matched with... Most similar patch Combined with the semantic space partitioning performed in step 2, if and If the colors are within the same semantic region, such color consistency matching pairs are considered valid. Further expanding the filter, if the center of the patch to be matched is located in a blank area, the current patch is skipped. This ultimately yields the final filter. ,in Indicates the background area. Indicating reciprocity check The semantic region to which it belongs

[0054] (4) Combining the above steps, the final cross-image color consistency matching function can be obtained, which includes color matching validity check and LAB color discrete representation: .

[0055] Step 4: Perform intra-image color consistency matching for each rendered image to maintain color smoothness within the same semantic region.

[0056] (1) Using the total variational loss function, combined with the semantic region division obtained in step 2, perform pixel-level color smoothing on patches of the same semantic region in a single rendered image.

[0057] (2) Further render the depth map obtained from the current viewpoint, and combine it with the semantic region segmentation obtained in step 2 to extend consistency matching to the depth information.

[0058] (3) Combining the above operations with color consistency and depth consistency matching, the final intra-image consistency matching is obtained: ,in and The weights of the corresponding loss calculation parameters are all set to 1. This represents the current rendered image. Represents a semantic graph. A depth map representing the current viewpoint. This indicates intra-image consistency matching that combines corresponding image information and semantic graph information.

[0059] Step 5: Combine the inter-image color consistency loss, intra-image color consistency loss, and fractional distillation sampling loss to jointly optimize the 3D model.

[0060] (1) Using fractional distillation sampling loss, Gaussian noise is added to the current rendered image to a certain extent according to the design scheme. The noisy image and the text description provided by the user are input into the text-to-image diffusion model Stable Diffusion2.1 to obtain the predicted noise result. The difference between the predicted noise result and the actual added noise is calculated to obtain the fractional distillation sampling loss.

[0061] (2) Combining the cross-image color consistency matching loss obtained in step 3, the intra-image color consistency matching loss obtained in step 4, and the fractional distillation sampling loss, the three are weighted in a ratio of 1:1:1 and used as the total loss. After backpropagation, the parameters of the 3D representation 3DGS (such as position, color, covariance, etc.) are optimized. After 5000 iterations of optimization, the final 3D model is obtained.

[0062] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating 3D objects from text based on dual color consistency matching, characterized in that, Includes the following steps: Step 1: Render multi-view images from the 3D model, including a frontal view and multiple non-frontal views; Step 2: Use a visual feature extraction model to extract features and segment semantic regions in the rendered image; Step 3: Using the frontal view as a color reference, perform color consistency matching for similar semantic regions between images from non-frontal views; Step 4: Perform intra-image color consistency matching for each rendered image to maintain color smoothness within the same semantic region; Step 5: Combine the inter-image color consistency loss, intra-image color consistency loss, and fractional distillation sampling loss to jointly optimize the 3D model.

2. The method for generating 3D objects from text based on dual color consistency matching according to claim 1, characterized in that, The 3D model rendering process in step 1 is as follows: (1) 3D Gaussian Splatting (3DGS) is used as the 3D representation of the 3D model to leverage its efficient rendering capabilities and high-fidelity representation capabilities; based on the user-input text description (Prompt) of the 3D model features, a pre-trained text-to-point cloud generation model is invoked to generate a preliminary 3D point cloud distribution that conforms to the text description. The point cloud distribution contains the initial geometric skeleton and basic color information of the object in 3D space; the generated point cloud distribution is used as the basis to initialize the optimizable parameters of the 3DGS model. (2) During the optimization process, a set of multi-view images are rendered based on the 3D model. The set of multi-view images includes at least one reference view image as a color reference benchmark and at least one non-reference view image. The reference view is preset to generate a view with more stable quality and higher fidelity, which is used as the anchor image for color consistency matching. The other views are randomly selected by random sampling.

3. The method for generating 3D objects from text based on dual color consistency matching according to claim 1, characterized in that, The feature extraction and region segmentation in step 2 are specifically performed as follows: (1) For each rendered image in the multi-view image set The system utilizes a pre-trained Transformer-based visual feature extraction model to perform forward propagation inference on the rendered image. The image is divided into a series of image patches, and a high-dimensional, semantically rich visual feature vector is extracted for each patch. These visual feature vectors constitute the feature map corresponding to the rendered image. , (2) Regarding the feature map All feature vectors in the image are grouped into nonparametric clusters using the Mean Shift clustering algorithm. The clustering process is based on distance or similarity in the feature space, aiming to group nearby feature vectors into the same category. The clustering results segment the rendered image into... Each semantically consistent region with clearly defined boundaries is identified, and a corresponding semantic label graph is generated. .

4. The method for generating 3D objects from text based on dual color consistency matching according to claim 1, characterized in that, The specific steps for color consistency matching in step 3 are as follows: (1) In each iteration of optimization, color consistency matching is performed on the rendering images of non-frontal view. First, some patches are randomly selected from the rendering image to be matched, and the patch most similar to it in the anchor image is found. The color distribution difference loss is calculated. ,in This indicates that a patch in the image to be matched is currently undergoing color matching. Indicates color anchoring in the image and The patch with the highest similarity (2) In the calculation of color distribution difference loss, the color distribution characteristics of each patch are represented by a discretized LAB color space vector. First, the RGB image is converted into a LAB image, and the L, A, and B channels of the LAB color space are divided into intervals based on experimental experience. By statistically analyzing the LAB color values ​​of each pixel in the patch into the pre-divided intervals, the color frequency distribution of the patch in each interval of the LAB color space is obtained. The discrete LAB color representation constructed in this way is used to calculate the color distribution difference loss across images. ,in This represents the transformation of the RGB color features of a given patch into a discrete LAB color frequency distribution representation, where i represents the corresponding rendered image, and j represents the patch corresponding to the i-th rendered image. (3) Introduce reciprocity detection, that is, obtain Then, in reverse order, search for the image to be matched with... Most similar patch Combining the semantic space partitioning results from step 2, if and If the colors are within the same semantic region, such color consistency matches are considered valid. The filter is then further expanded; if the center of the patch to be matched is located in the background region, the current patch is skipped, resulting in the final filter. ,in Indicates the background area. Indicating reciprocity check The semantic region to which it belongs (4) Combining the above steps, the final cross-image color consistency matching function is obtained, which includes color matching validity check and LAB color discrete representation: .

5. The method for generating 3D objects from text based on dual color consistency matching according to claim 1, characterized in that, Step 4, intra-image color consistency matching, specifically involves the following steps: (1) Using the Total Variation Loss (TV Loss) function, combined with the semantic region segmentation obtained in step 2, the patches of the same semantic region in a single rendered image are subjected to pixel-level color smoothing. (2) Further render the depth map obtained from the current viewpoint, and combine it with the semantic region segmentation obtained in step 2 to extend consistency matching to the depth information. (3) Combining color consistency and depth consistency matching, the final intra-image consistency matching is obtained: ,in and This represents the weights of the parameters used in the corresponding loss calculation. This represents the current rendered image. Represents a semantic graph. A depth map representing the current viewpoint. This indicates intra-image consistency matching that combines corresponding image information and semantic graph information.

6. The method for generating 3D objects from text based on dual color consistency matching according to claim 1, characterized in that, Step 5 involves optimizing the 3D model, specifically as follows: (1) Calculate the loss using fractional distillation sampling (SDS). Add Gaussian noise to the current rendered image to a certain level according to a preset scheme. Input the noisy image and the user-provided text description into the text-to-image diffusion model to obtain the predicted noise result. Calculate the difference between the predicted noise result and the actual added noise to obtain the fractional distillation sampling loss. (2) Combining the cross-image color consistency matching loss obtained in step 3, the intra-image color consistency matching loss obtained in step 4, and the fractional distillation sampling loss, the three are weighted and used as the total loss. After backpropagation, the parameters of the 3D representation 3DGS are optimized to update the 3D model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text-to-3D object generation method based on dual color consistency matching as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the method for generating three-dimensional objects from text based on dual color consistency matching as described in any one of claims 1-6.

Images