3D printing model construction method based on multi-modal condition guidance

Abstract

The invention relates to the technical field of three-dimensional computer graphics and artificial intelligence, in particular to a multi-modal condition guidance-based 3D printing model construction method, which specifically comprises the following steps of: S1, constructing a basic model by taking an open source model as a backbone network architecture; s2, constructing a submerged space diffusion model constrained by view angle parameters; s3, constructing a pairing data set; s4, training the submerged space diffusion model based on the paired data set; s5, receiving a text prompt input by a user, and defining a group of standard view angle parameter sequences required by the 3D printing model; s6, generating a corresponding 2D image for each visual angle parameter in the sequence; and S7, outputting a 3D printing model. The method is realized through a two-stage method: firstly, generating a multi-view image with geometric consistency through an injection view angle condition; and then introducing ray confidence regularization loss in three-dimensional reconstruction to eliminate geometric artifacts and form a clear surface boundary.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional computer graphics and artificial intelligence, and particularly relates to a 3D printing model construction method based on multi-modal condition guidance. BACKGROUND

[0002] Traditional 3D model creation processes, such as computer-aided design, require users to have deep professional knowledge and software operation skills, and are time-consuming and have high thresholds.

[0003] In recent years, with the development of AIGC (artificial intelligence generated content) technology, technologies for directly generating 3D models from text or images have emerged. However, the existing technologies have many defects: in terms of the method for generating 3D models from text, the generation process involves the generation of multi-view images, but the images of each view of the current generation model cannot satisfy complete geometric consistency, resulting in low precision of the generated 3D model and the presence of holes and non-manifold edges, which cannot be applied to subsequent 3D printing slicing. SUMMARY

[0004] The present application aims to provide a 3D printing model construction method based on multi-modal condition guidance to improve the surface state of the generated 3D printing model, so that the AI-generated model can be applied to 3D printing.

[0005] To achieve the above-mentioned purpose, the present application provides a 3D printing model construction method based on multi-modal condition guidance, comprising the following steps: S1: constructing a basic model using an open-source model as a backbone network architecture; S2: adjusting the basic model, using low-rank adaptation technology to adaptively adjust the parameters of the neural network structure of the basic model to introduce view angle parameters, and constructing a latent space diffusion model constrained by the view angle parameters; S3: constructing a paired data set, rendering each view angle of each 3D model in the 3D model library under a controlled rendering environment through an automated rendering pipeline to generate 2D images corresponding to different view angle parameters, and labeling a text description for each 2D image to form a paired data set with a text description, corresponding view angle parameters and corresponding rendering images; S4: training the latent space diffusion model based on the paired data set; S5: receiving a user input text prompt, defining a standard view angle parameter sequence required for a group of 3D printing models; S6: for each view angle parameter in the sequence, inputting the corresponding text prompt and view angle parameter into the latent space diffusion model simultaneously to generate a corresponding 2D image; S7: Optimize the neural radiance field using photometric loss driving, and introduce a specific loss function into the optimized neural radiance field to output the 3D printing model in combination with the use of the confidence map.

[0006] Before training using the Stable Diffusion model, hereinafter referred to as (SD model), a paired data set is first constructed, which contains a large number of 3D models from Objaverse, ABO and self-built library. For these 3D models, on the one hand, through an automated rendering pipeline, 2D images are generated by rendering the 3D models in a standardized virtual photo studio environment with multiple preset, deterministic pose, and standardized view angles. On the other hand, the 3D models are annotated by a large language model (LLM) to generate a paired data set, which contains annotated text descriptions, view parameters, and rendered 2D images corresponding to each record in the data set. The paired data set can be used for training of the SD model. In the SD model, a low-rank adaptation technology (LORA) structure is introduced to inject view parameters, so that the SD model can be constrained by view parameters. After training, a view parameter-constrained latent diffusion model (VC-LDM) is obtained, which can be used to generate 3D printing models. After the user inputs a text prompt, a standard view parameter sequence is defined. The VC-LDM can generate a set of geometrically self-consistent image sequences based on the standard view parameter sequence. Then, a 3D printing model is outputted by using a specific loss function in combination with the use of the confidence map.

[0007] The definition of a standard view parameter sequence required for the 3D printing model includes: if the user input text prompt contains view parameters, the user input view parameters form the standard view parameter sequence; if the input text prompt does not contain view parameters, the model generates a standard view parameter sequence.

[0008] Further, the adjustment of the base model in step S2 includes adaptive adjustment of the parameters of the neural network structure of the base model using low-rank adaptation technology to introduce view parameters and construct a view parameter-constrained latent diffusion model. S21: Use a view encoder to encode discrete view parameters into a high-dimensional vector, S21: Use a text encoder to encode semantic conditions; S21: Modify the model attention calculation formula to construct a latent diffusion model that accepts semantic conditions and high-dimensional vectors in combination with low-rank adaptation.

[0009] In the process of adaptively adjusting the parameters of the neural network structure using the low-rank adaptation technology in the latent space diffusion model constrained by the view angle parameter, after the view angle and the text are encoded by the view angle encoder and the text encoder respectively, a composite guidance condition is formed to enable the latent space diffusion model to accept the constraint of the view angle parameter.

[0010] Further, in the step S6, after generating the corresponding 2D image, consistency checking can also be performed. Figure 1 Specifically, the consistency checking process is a consistency checking module based on CLIP image similarity, which calculates the feature similarity between all generated images. Figure 1 If the deviation of any view from other views exceeds the threshold, the view is redrawn using a different random seed to ensure the consistency of the image.

[0011] Further, Attention(Q, K, V) = softmax(Q · (K_text T + K_view T ) / sqrt(d_k)) · V , wherein, Q represents the image features being processed in the neural network, K represents the label information, V represents the semantic content adapted to the label information, K_text T means the representation of the key generated by the semantic condition after transposition, K_view T means a high-dimensional vector encoded by the view angle parameter, K_view after the key is generated by processing inside the model, K_ text T + K_view T means that the text content label and the view angle parameter label are combined into a composite label, sqrt(d_ k) is a scaling factor, softmax(..) is a normalization function.

[0012] The attention calculation formula of the cross-attention layer of the latent space diffusion model is modified, in this way, during the training process, the cross-attention layer of the latent space diffusion model not only receives the text prompt from the text encoder, but also receives the view angle parameter from the view angle encoder, and the low-rank adaptation weight learns how to adjust the generation process according to the view angle parameter, so that the model can accurately generate the object form under the specific view angle parameter when receiving it. This ensures that when a series of continuous view angle parameters are input in the inference stage, the model can generate a set of geometrically completely self-consistent image sequences.

[0013] Further, in step S7, the specific loss function is L_total = L'_rgb + λ_rcr · L_rcr , wherein L'_rgb the meaning of the modified low-rank adaptive loss function is λ_rcr the meaning of the hyper-parameter balancing photometric accuracy and geometric cleanliness is L_rcr the meaning of the ray confidence regularization loss is; the modified low-rank adaptive loss function L'_rgb is: L'_rgb =∑ w(r) · ||C(r) - C_gt(r) || 2 ， , wherein C(r) the meaning of the rendered pixel color is C_gt(r) the true pixel color in the image.

[0014] By introducing a confidence map for each image, generated by a saliency or segmentation model, to identify the object of interest, pixels corresponding to the object have high confidence while background pixels have low confidence, the specific loss function is provided to force the optimization process to focus on reconstructing the object itself and avoid generating inconsistent background.

[0015] Further, in step S7, the use of the confidence map to output the 3D printing model specifically refers to: projecting a 3D ray from each pixel in the generated 2D image; introducing a confidence map for each 2D image and selecting a confidence threshold; determining that a pixel with a confidence less than the threshold is a background pixel, and determining that a pixel with a confidence greater than or equal to the threshold is an object pixel; for the rays of the background pixels, penalizing the density accumulation along their paths.

[0016] The ray confidence regularization loss L_rcr =∑ (1 - w(r)) ·∫ [σ(r(t))] dt, , wherein ∫ [σ(r(t))] dt σ the meaning of the sum of the densities of all sampling points on a ray is.

[0017] A standard neural radiance field optimizes a neural network to project 3D coordinates X and viewing directions d to a color c and density r(t) = o + t · d For each pixel in the generated image, an exact 3D ray can be projected for the pixel under the condition that the exact viewing angle parameters are known Figure 1 , wherein o is the camera origin, dis the direction vector, the optimization process is driven by photometric loss between rendered pixels and input image pixels, after the confidence of the confidence map is filtered, the path of the ray belonging to the background in the three-dimensional space should be empty, the path being empty is essentially the core physical intuition of the ray confidence regularization loss. It can be imagined that a matrix three-dimensional space content is completed by each ray to build a model, but the ray should not be completely penetrated, because a shape is formed, like digging a shape in a solid object; therefore, any density accumulation on these rays is punished, for a background ray, the cumulative transmittance of the ray from beginning to end should be 1 in the ideal case, and the ray confidence regularization loss is actually the integral of the density of the known empty ray, and is weighted by the confidence that the ray is indeed empty. Minimizing the ray confidence regularization encourages the model to learn a compact, clean geometry, and carve out the redundant floating debris in the three-dimensional space. Figure 2

[0018] The application is a 3D printing model construction method based on multi-modal condition guidance, which is aimed at the problem that the images of each view of the generated model cannot meet the geometric self-consistency requirement, and the precision of the generated 3D model does not meet the requirements. On the one hand, the low-rank adaptation technology is used to introduce the view angle parameter, and the text prompt is also introduced, the paired data set records the text prompts corresponding to each generated image, the view angle parameter, and the deterministic characteristics of the generated view are used to construct a specific loss function, the known empty area is inferred from the segmentation of the view, to actively regularize the geometry, and force the formation of a clearer and more accurate surface boundary. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, below the drawings needed to be used in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 2 is a multi-view image generation process of the application of a 3D printing model construction method based on multi-modal condition guidance. Figure 3 is a 3D model generation process of the application of a 3D printing model construction method based on multi-modal condition guidance.

[0021] Figure 4 is a 3D model generation process of the application of a 3D printing model construction method based on multi-modal condition guidance.

[0022] Figure 5 is a graphical preview generated based on the prompt word.

[0023] Figure 6 ​It is a multi-view 2D image generated based on prompts.

[0024] c_text It is a 3D model generated based on a standard viewpoint parameter sequence.

[0025] c_view This is a flowchart illustrating a 3D printing model construction method based on multimodal condition guidance according to the present invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0027] Example 1 We employ the open-source Stable Diffusion v2.1 model as the backbone network architecture to construct a large-scale, high-quality "3D-MultiView-Text" paired dataset. This dataset contains 1.5 million high-quality 3D models from Objaverse, ABO, and our own libraries. For each model, an automated rendering pipeline renders it in a standardized virtual studio environment from N preset, deterministic poses, including azimuth, elevation, and distance, generating corresponding 2D images. Simultaneously, each model is annotated with a detailed text description optimized by a Large Language Model (LLM), including the main features, materials, and style of the object. Each record in the final dataset is {text description, viewpoint parameter_i, rendered image_i}, where i ranges from 1 to N. A low-rank adaptive structure is used to inject viewpoint parameters into the model's neural network. Specifically, discrete viewpoint parameters are encoded into a high-dimensional vector. During training, the model's cross-attention layers receive semantic conditions not only from the text encoder but also... Attention(Q, K, V) = softmax(Q · (K_text It also receives the viewpoint embedding vector at the same time. + K_view The formula for calculating attention has been modified as follows: ) / sqrt(d_k)) · V T c_view T Figure 1 In this way, LoRA weights specifically learn how to... {V_1, V_ The generation process is adjusted to ensure that the model can accurately generate the object shape from a specific viewpoint when given a series of consecutive viewpoint parameters. This ensures that when a series of consecutive viewpoint parameters are input during the inference phase, the model can generate a geometrically consistent sequence of images.

[0028] like 2,..., V_N}As shown, the reasoning and generation process for generating multi-view images by introducing viewpoint parameters is as follows: User input text prompts; generate a set of standard viewpoint parameter sequences required for 3D reconstruction. V_N I_i For each viewpoint system, text prompts and viewpoint parameters will be provided. Figure 1 Simultaneously, the input is fed into a latent space diffusion model constrained by viewpoint parameters to generate the corresponding 2D image. I_i ;Optionally integrate a CLIP-based image similarity-based view Figure 4 Consistency verification module. This module calculates the consistency of all generated images. f_Θ The similarity of features between the views is used to determine the degree of difference between the views. If the deviation of a view from other views exceeds a threshold, the view will be redrawn using different random seeds to ensure the high consistency of the entire set of images.

[0029] Example 2: Please see σ Standard neural radiation fields are optimized by a neural network. I_i , 3D coordinates x and observation direction d Mapping to a color c and density r(t) = o + t · d The optimization process is driven by the photometric loss between the rendered pixels and the input image pixels. However, when the pose is fully known, a stronger geometric prior is introduced, thus utilizing ray confidence regularization of the loss term. For the generated image L'_rgb Each pixel in the image has its precise viewpoint parameters known. Therefore, a precise 3D ray can be projected onto that pixel. L'_rgb =∑ w(r) · ||C(r) - C_gt(r) || ,in o It is the camera's origin. d It is a direction vector. r(t) Represents a ray. t It is the distance along the ray direction, the standard neural radiation field loss function. C_gt(r) for: L'_rgb 2 in C(r) It refers to the rendered pixel color. L'_rgb =∑ w(r) · ||C(r) - C_gt(r) || is the ground-truth pixel color in the input image. A confidence map W_i is introduced for each input image. This map is generated by a saliency or segmentation model such as SAM to identify target objects. Pixels corresponding to objects have high confidence (w≈1), while background pixels have low confidence (w≈0).

[0030] Modified loss low-rank adaptive loss function L_rcr for: L_rcr =∑ (1 - w(r)) ·∫ [σ(r(t))] dt 2 This forces the optimization process to focus on reconstructing the object itself, rather than generating an inconsistent background.

[0031] Then introduce ray confidence regularization loss L_rcr : L_total = L'_rgb + λ_rcr · L_rcr This term is actually an integral of the density σ along a known empty ray, weighted by the confidence level that the ray is indeed empty. Minimize Figure 3 The model is encouraged to learn a compact, clean geometry. σ(r(t)) represents the density value at the point t on the ray. We will collect 128 discrete points on the ray, calculate the density σ at each point, and finally sum up the density values ​​of all 128 points.

[0032] The final loss function used to optimize the neural radiation field is: Figure 4 A specific loss function is constructed that utilizes negative spatial information inferred from the segmentation of the deterministically generated input view to actively regularize the geometry and force the formation of a clearer and more accurate surface boundary.

[0033] The specific implementation is as follows: Input the prompt text "Optimus Prime", and generate a result like... Figure 4 The image shown allows users to confirm that the image meets their needs before proceeding to the next step. The user can then input the prompt text "Generate Optimus Primethree-view images, 45 degrees, 90 degrees, 180 degrees" to generate images like... Figure 4 The image shows 2D images from different perspectives; alternatively, you can directly input "Optimus Prime Generate Optimus Prime three-view images, 45 degrees, 90 degrees, 180 degrees" as the prompt to generate the image. Figure 5 The image shown is in ​ Based on image generation, it can continue to generate ​ The 3D model shown above is only an illustrative description of the generation process.

[0034] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for constructing 3D printing models based on multimodal condition guidance, characterized in that, Includes the following steps: S1: Use open-source models as the backbone network architecture to build the basic model; S2: Adjust the base model, use low-rank adaptation technique to adaptively adjust the parameters of the neural network structure of the base model to introduce viewpoint parameters, and construct a latent space diffusion model constrained by viewpoint parameters. S3: Construct a paired dataset. In a controlled rendering environment, render each 3D model in the 3D model library from various perspectives using an automated rendering pipeline to generate 2D images corresponding to different perspective parameters. At the same time, annotate each 2D image with a text description to form a paired dataset with text description, corresponding perspective parameters, and corresponding rendered images. S4: Train the latent space diffusion model based on the paired dataset; S5: Receive text prompts from the user and define a set of standard viewpoint parameter sequences required for a 3D printed model; S6: For each viewpoint parameter in the sequence, input the corresponding text prompt and viewpoint parameter into the latent space diffusion model simultaneously to generate the corresponding 2D image; S7: The neural radiation field is optimized using photometric loss. A specific loss function is introduced into the optimized neural radiation field, and a confidence map is used to output the 3D printing model.

2. The 3D printing model construction method based on multimodal condition guidance as described in claim 1, characterized in that, The open-source model refers to the StableDiffusion model.

3. The 3D printing model construction method based on multimodal condition guidance as described in claim 1, characterized in that, Step S2, which involves adjusting the base model, using low-rank adaptation techniques to adaptively adjust the parameters of the neural network structure of the base model to introduce viewpoint parameters, and constructing a latent space diffusion model constrained by viewpoint parameters, specifically includes: S21: Use a view encoder to encode discrete view parameters into high-dimensional vectors. S21: Encode semantic conditions using a text encoder; S21: Modify the model attention calculation formula and combine it with low-rank adaptation to construct a latent space diffusion model that accepts semantic conditions and high-dimensional vectors.

4. The 3D printing model construction method based on multimodal condition guidance as described in claim 3, characterized in that, The viewing parameters include at least two of the following: azimuth, elevation, distance, elevation, orientation, and attitude.

5. The 3D printing model construction method based on multimodal condition guidance as described in claim 3, characterized in that, The revised attention formula is: Attention(Q,K,V)=softmax(Q·(K_text T +K_view T ) / sqrt(d_k))·V In the formula, Q This represents the image features being processed in the neural network. K Indicates label information, V This represents semantic content that matches the tag information. K_ text T It means the transposed representation of the key generated by the semantic conditions. K_view T This means a high-dimensional vector encoded by viewpoint parameters, which is processed within the model to generate keys. K_view The representation after transposing. sqrt(d_k) Scaling factor softmax(..) This is the normalization function.

6. The 3D printing model construction method based on multimodal condition guidance as described in claim 1, characterized in that, The controlled rendering environment refers to a standardized virtual studio environment.

7. The 3D printing model construction method based on multimodal condition guidance as described in claim 1, characterized in that, The 3D model library includes Objaverse, ABO, and self-built libraries.

8. The 3D printing model construction method based on multimodal condition guidance as described in claim 1, characterized in that, In step S7, the specific loss function is: L_total=L'_rgb+λ_rcr·L_rcr In the formula, L'_rgb This refers to the modified low-rank adaptive loss function. λ_rcr The meaning is a hyperparameter that balances photometric accuracy and geometric cleanliness. L_rcr This refers to the loss due to ray confidence regularization.

9. The 3D printing model construction method based on multimodal condition guidance as described in claim 8, characterized in that, The modified low-rank adaptive loss function L'_rgb for: L'_rgb=Σw(r)·||C(r)-C_gt(r)|| 2 In the formula, C(r) The meaning is the rendered pixel color. C_gt(r) It is the true pixel color in the image.

10. The 3D printing model construction method based on multimodal condition guidance as described in claim 1, characterized in that, In step S7, the use of confidence maps to output 3D printed models specifically refers to: A 3D ray is projected from each pixel of the generated 2D image; A confidence map is introduced for each 2D image, and a confidence threshold is selected; Pixels with a confidence level less than the threshold are classified as background pixels, while pixels with a confidence level greater than or equal to the threshold are classified as object pixels. For a ray of a background pixel, a penalty is applied to the density accumulation along its path.

Images