Zero-sample 3D generation method for enhancing view angle consistency

Abstract

The invention provides a zero-sample 3D generation method for enhancing viewing angle consistency, and aims to solve the problem of inconsistency, namely a'multi-surface 'problem, of the same object under different viewing angles in the current 3D generation technology. This problem originates from typical perspective preferences of a pre-trained generative model. Therefore, a visual angle decoupling method VDM is adopted to extract visual angle features to eliminate visual angle prior preferences and enhance visual angle control, meanwhile, similarity partial order loss PSL is introduced to optimize similarity distribution of images between visual angles, and it is ensured that the generated 3D images keep high consistency under different visual angles. In addition, the technology is also combined with a 3D Gaussian splashing technology, so that the rendering effect and detail performance of the model are further enhanced. According to the scheme, the authenticity and consistency of the 3D content are remarkably improved, the method is more suitable for the fields of virtual reality, game design, industrial design and the like, and practicability and development of the zero-sample 3D generation technology are greatly promoted.

Description

Technical Field

[0001] The present invention belongs to the field of 3D generation, and in particular relates to a zero-sample 3D generation method for enhancing viewing angle consistency. Background Art

[0002] 3D generation technology plays a vital role in many fields such as innovative industrial design, game design, and virtual reality. In particular, the progress of zero-sample text-to-3D generation technology has provided more possibilities for innovative design from scratch, and also provided strong support for user interactive experience and simulation, making the transition from concept to reality more efficient and intuitive. However, unlike the small-span dimensionality increase of text-to-image generation tasks, the inherent complexity of real scenes and the scarcity of 3D data make it a huge challenge to generate high-quality 3D content from text. Recently, DreamFusion technology uses a mature text-to-image generation model, the prior knowledge of the stable diffusion model SD, and the advanced fractional distillation sampling technology SDS to upgrade the 2D results from the perspective dimension to the 3D world without relying on 3D datasets, thus realizing unsupervised 3D content generation. In subsequent research, 3D Gaussian splashing technology is introduced to replace the traditional implicit 3D representation method, and this paradigm is optimized in terms of generation quality, generation speed, and geometric structure.

[0003] However, existing text-to-3D generation methods are still largely limited by geometric collapse and perspective inconsistency problems. One of the most notable problems is the "multi-face" problem (also known as the "multi-face" problem), which usually manifests itself as the same 3D object presenting different or even contradictory appearances under different perspectives, seriously affecting the realism of the 3D generation results. This phenomenon occurs because a key factor is ignored: in order to improve training efficiency, most existing text-to-image generation models use normalized perspective images (typical perspective images, images of objects or scenes presented at common and standard angles) as training data. Such data usually selects common perspectives that can maximize the expression of object features, such as frontal perspectives and oblique front perspectives. Therefore, in the absence of specific perspective conditions, pre-trained text-to-image generation models such as the stable diffusion model SD will generate images under normal perspectives based on perspective prior knowledge. Recent text-to-3D generation methods optimize parameterized 3D models by matching the rendered images of the target 3D object at each perspective with the generated results of the stable diffusion model SD. However, when trying to add a specific perspective description to the original prompt word to guide the model to generate images of the corresponding perspective, the perspective prior knowledge inherent in the stable diffusion model SD and the newly added perspective control information may conflict, resulting in the ambiguity of perspective semantics, thus causing multi-viewing. Figure 1 Consistency issues. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a zero-sample 3D generation method for enhancing perspective consistency in view of the shortcomings of the prior art, comprising the following steps:

[0005] Step 1: mathematically analyze the multifaceted problems in the process of generating three-dimensional objects and propose optimization goals;

[0006] Step 2: Get the text prompt word input by the user and select the theme word in the prompt word, connect the theme word to three different perspective description words (the end of the theme word is connected to "front perspective", "side perspective" and "back perspective" respectively), construct three phrases containing the theme word with perspective description, input the three phrases containing the theme word into the pre-trained text encoder CLIP respectively, obtain three encoding vectors, extract the encoding parts corresponding to the theme word from the three encoding vectors respectively, and then use the method of extracting vertical components to extract the features of the front perspective, side perspective and back perspective of the theme word object from the three theme word encodings respectively;

[0007] Step 3: Input the user prompt word into the pre-trained text into the 3D diffusion model PointE to generate rough 3D point cloud data. Meanwhile, the rough 3D point cloud is fitted using the 3D Gaussian splashing technique to generate an explicitly represented 3D Gaussian model. Then, the 3D Gaussian model is rendered using randomly generated camera parameters to obtain renderings from more than two perspectives.

[0008] Step 4: Input the user prompt word into the pre-trained text encoder CLIP to obtain the prompt word encoding vector, and locate the position of the theme word in the encoding vector according to the theme word. According to the camera parameters randomly generated in step 3, the features of the front view, side view and back view extracted in step 2 are respectively injected into the theme word encoding part in the user prompt word encoding, and the prompt word encoding after the view angle control corresponding to all camera parameters is obtained. The new prompt word encoding vector and the rendering of more than two view angles obtained in step 3 are combined. Figure 1 Input the pre-trained stable diffusion model SD;

[0009] Step 5: The stable diffusion model SD generates a corresponding two-dimensional image based on the user prompt word and the rendered image, extracts the unconditional prediction noise generated during the generation process, and obtains the unconditional denoising result. It then explores the similarity distribution of the image and uses the similarity partial order loss to constrain the unconditional denoising result, thereby enhancing the perspective consistency of the three-dimensional Gaussian model.

[0010] In step 1, the “multi-faceted” issues include:

[0011] Existing text-to-3D generation methods are still largely limited by the problems of geometric collapse and perspective inconsistency. One of the most notable problems is the "multi-face" problem (also known as the "multi-face" problem), which usually manifests itself as the same 3D object presenting different or even contradictory appearances under different perspectives, seriously affecting the realism of the 3D generation results. This phenomenon occurs because a key factor is ignored: in order to improve training efficiency, most existing text-to-image generation models use normalized perspective images (typical perspective images, images of objects or scenes presented at common and standard angles) as training data. Such data usually selects common perspectives that can best express the characteristics of the object, such as the front perspective and the oblique front perspective. Therefore, in the absence of specific perspective conditions, pre-trained text-to-image generation models such as the stable diffusion model SD will generate images under normal perspectives based on perspective prior knowledge. Recent text-to-3D generation methods optimize parameterized 3D models by matching the rendered images of the target 3D object at each perspective with the generated results of the stable diffusion model SD. However, when trying to add a specific perspective description to the original prompt word to guide the model to generate images of the corresponding perspective, the perspective prior knowledge inherent in the stable diffusion model SD and the newly added perspective control information may conflict, resulting in the ambiguity of perspective semantics, thus causing multi-viewing. Figure 1 Consistency issues.

[0012] In step 1, the mathematical level analysis includes:

[0013] The diffusion model is a type of generative model used to learn and sample from complex distributions. It originates from random processes in statistical physics. The core idea is to gradually "diffuse" data from its original form to random noise, and then restore the noise to data through an inverse process. The inverse process is generally to reduce the noise samples through a trained neural network. Later, the concept of denoising diffusion probability model DDPM was proposed, and the denoising diffusion probability model DDPM was used to optimize the following simplified objectives:

[0014]

[0015] in, Represents the loss function of the denoising diffusion probability model DDPM, which is used to measure the difference between the samples generated by the model and the actual samples. By minimizing the loss function , the denoising diffusion probability model DDPM can gradually learn how to denoise and generate high-quality samples. Expected value Represents the original sample data ,noise and time step Calculate the expected value, where the original sample data As the input data of the denoising diffusion probability model DDPM, the noise represents random noise that follows a standard normal distribution with a mean of 0 and a variance of 1. During the training process, this noise will be added to the sample. Represents the diffusion time step, which is used to control the degree of noise addition. The time step in the diffusion model ranges from 1 to Increase gradually. Denotes the denoising diffusion probability model DDPM in the model parameters Next, according to the noise perturbation time step The sample after and time step Predicted noise value; added noise and model prediction noise The square of the two norm between Measures the difference between the predicted noise and the actual added noise. During the inference process, the denoising diffusion probability model DDPM is calculated from Starting from the probability density Sampling the previous sample from the normal distribution of . The denoising diffusion probability model DDPM emphasizes the outstanding ability of diffusion models in capturing and simulating real-world image data. This advantage has spawned a series of innovations and improvements, including the stable diffusion model SD. The stable diffusion model SD is based on the latent diffusion model LDM, and highlights two key advantages in design and implementation. First, by performing the diffusion process in a lower-dimensional latent space, the efficiency and speed of image generation are significantly improved, and the computational burden is greatly reduced. Secondly, by introducing a conditional mechanism, the latent diffusion model LDM uses the advanced text encoder CLIP to encode text conditional information into the latent space, so that the model can efficiently generate highly relevant and realistic images based on specific conditional information. The latent diffusion model LDM optimizes the following simplified objectives:

[0016]

[0017] in, is the loss function of the latent diffusion model LDM, and the expected value Represents the original sample data The potential encoding result of , User prompts , random noise that follows a standard normal distribution and time step Perform expectation calculation; conditional information encoding Indicates prompts to the user through the text encoder CLIP The result of encoding is The second norm represents the difference between the added noise and the predicted noise. The method of the present invention is proposed based on the stable diffusion model SD and the latent diffusion model LDM.

[0018] Under the framework of the denoising diffusion probability model DDPM and the latent diffusion model LDM, the generation of the image can be regarded as the process from the final noise state To the noise-free initial clear state The reverse recovery process is controlled by user prompts Specifically, the inverse restoration process involves gradually restoring from a high-noise state to a noise-free original image state, and the following formula is used to express the final noise-free initial clear state: The probability density of :

[0019]

[0020] in, express The maximum time step that can be taken, Represents the probability distribution of the final state of the diffusion process, which is a standard normal distribution. Indicates that given a user prompt and the current potential representation Under the condition, generate the potential representation of the previous step The conditional probability of It is the differential element in the integral, which means that for every possible The state changes slightly, and these differential elements are used to calculate the joint probability distribution of all possible intermediate states from the initial state to the final state. In the task of generating 3D text from zero samples, the model will integrate the denoising diffusion probability model DDPM process from various perspectives through multiple iterative steps. The generated 2D image representation can be used to construct 3D parameters The unnormalized probability density function of , this function is obtained by:

[0021]

[0022] in, Represents the set of all perspectives. This expression aggregates all the perspectives from the perspective set. Each perspective selected The results of generating images through the denoising diffusion probability model DDPM process reveal the contribution of generated images from different perspectives to the estimation of the overall three-dimensional model parameters. Although the denoising diffusion probability model DDPM method performs well in generating high-quality two-dimensional images, the randomness in its inverse process and its sensitivity to input fluctuations may lead to inconsistent image features generated from different perspectives, thereby affecting the quality of the overall three-dimensional model. Specifically, during the training or distillation stage of the model, the pre-trained denoising diffusion probability model DDPM may tend to produce pseudo-true values ​​with poor quality and inconsistent features. This problem is particularly prominent in the multi-view fusion process, resulting in over-smoothed and detail-missing three-dimensional reconstruction results. To address this challenge, many existing works have introduced the denoising diffusion implicit model DDIM method, which significantly reduces the random variation in each iterative step through a more deterministic recursive process. Although the denoising diffusion implicit model DDIM effectively improves the quality and consistency of the pseudo-true values, the "multi-faceted" problem is still significant, so the focus is on the multi-view fitting process outside the diffusion process. The method of the present invention redefines the three-dimensional parameters The density function of For a given series of optimization steps Viewing angle control , User prompts And the 3D Gaussian model projection under the corresponding perspective Under the condition, each iteration step The product of the conditional likelihoods is expressed as:

[0023]

[0024] in, express The maximum number of optimization steps that can be taken, Represents the iteration step In the three-dimensional parameters The state of the 3D model controlled by the user. It should only contain guidance information related to the generated content, and not directly involve perspective information. However, in the study, it was found that the pre-trained diffusion model had a problem parsing user prompts because it was exposed to training data with normalized perspective. Automatically introduce perspective prior knowledge when In order to accurately control and understand this phenomenon in theory, we start from the stable diffusion model’s understanding of user prompts and Divide into content sections and perspective priors , content part and perspective priors are orthogonal. Directly reflects the content input by the user, and Model-based training experience is the typical perspective preference introduced by the model when parsing user input based on pre-training data. The density function of The new expression is:

[0025]

[0026] in Indicates that it is subject to , this condition indicates that the content part and perspective priors No components in any direction are shared in the vector space, ensuring that the content generation is completely independent of the influence of any specific perspective. Further, taking the logarithm of both sides of the equation yields:

[0027]

[0028] Next, using the chain rule, right Gradient It is expressed as:

[0029]

[0030] in, Indicates the status of the 3D model About 3D parameters The partial derivative of Represents the logarithmic probability density About 3D Model Status The partial derivative of

[0031] right Further expansion using Bayes' theorem:

[0032]

[0033] in, Item relative to the 3D model state is regarded as a constant, so the partial derivative is 0 and this term can be deleted. The term intuitively reflects the model's instinctive perception of 3D objects when there is no external perspective information guidance. However, this perception is particularly susceptible to the influence of view prior knowledge, causing the model to rely on the most common or most significant perspective features in past experience when not constrained by external perspective. This dependence will cause the model to deviate when processing new perspectives, that is, when faced with a situation that is significantly different from the perspective of the training data, the generated results deviate from the true 3D representation. In addition, the second term in the expression This can be further extended by using PCMI, which provides a quantitative way to show how the information interaction between different variables in a specific state exceeds the simple combination of their individual behaviors. Further broken down into:

[0034]

[0035] Among them, the viewing angle control and perspective priors Point-wise conditional mutual information , and can be considered a constant in the current optimization step, so it can be further simplified terms and expand using the definition of conditional probability:

[0036]

[0037] This measure is known In the case of and The additional information between them compared to when they exist independently. The view prior part in the pre-trained model Conflict, that is << , then for each For , the PCMI terms of point conditional mutual information are close to 0, then Item and The item will also have an adverse effect on 3D generation. Therefore, and The semantic consistency of perspectives between them is particularly critical to alleviating the “multi-faceted” problem.

[0038] Step 2 includes:

[0039] Get the user input text prompt word and select the theme word in the prompt word. For example, in the prompt word "a photo of a car", the theme word is "car". Input the theme word directly into the pre-trained CLIP encoder to obtain the encoding vector of the theme word without perspective description .

[0040] Connect the keyword to the descriptive words of three different perspectives, that is, connect the end of the keyword to the front perspective, side perspective and back perspective respectively, to construct three phrases containing the keyword with perspective descriptions. For example, the keyword "car" connected to the perspective description phrases becomes: "car, front perspective", "car, side perspective" and "car, back perspective".

[0041] Input the three phrases containing the subject word with the perspective description into the CLIP encoder respectively and locate the position of the subject word code to obtain the subject word code containing the corresponding perspective description ,and The difference is that the subject word coding containing the corresponding perspective description Contains additional contextual information, which provides a way to extract Content-independent view feature encoding , this process uses the method of extracting the vertical component, which is formulated as:

[0042]

[0043] in, , Indicates the subject word code containing the corresponding perspective description Encoded vector describing the topic word without perspective The projection part on the image is obtained by subtracting this projection component to obtain the content-independent viewing feature encoding. Represents the extracted pure perspective difference. This difference can be used in the encoding process of other prompt words to achieve perspective control. When generating 2D images, if there is no perspective-related description in the prompt words, the generated result is often a normalized perspective result. In order to make the generated view more in line with the diverse needs of users, the subject word encoding in the user prompt is first decoupled from the prior perspective features to eliminate typical view preferences, and on this basis, the perspective control information is injected. Taking the "back perspective" as an example, the "back perspective" feature is injected after decoupling the "front perspective" and "side perspective" features, and the formula is:

[0044]

[0045] in, Indicates the need to eliminate the perspective of prior knowledge, including the front and side. The encoding vector representing the topic word without perspective description Content-independent view feature encoding Direction projection, Represents the content-independent back view feature encoding, is a scaling factor that indicates the injection strength of the new perspective.

[0046] Step 3 includes:

[0047] Inputting the user prompt word into the pre-trained text into the three-dimensional diffusion model PointE to generate rough three-dimensional point cloud data, wherein the rough three-dimensional point cloud data contains basic geometric information of the object surface;

[0048] Anisotropic Gaussian is a three-dimensional Gaussian distribution that allows different variances in different directions, that is, the shape of the distribution can be stretched or scaled along all directions. This feature enables it to adapt to the local structure of point cloud data more flexibly. By fitting anisotropic Gaussian volumes, a Gaussian volume that matches the local distribution of the point cloud can be generated in each local area in the point cloud, forming a continuous and smooth representation. This fitting process takes into account the density distribution and local geometric structure of the point cloud to more accurately capture the shape details of the object. Anisotropic Gaussian volumes are used to capture the geometric features of the rough three-dimensional point cloud data generated by the three-dimensional diffusion model PointE, and these Gaussian volumes directly define the geometric structure of the rough three-dimensional point cloud data generated by the three-dimensional diffusion model PointE through information such as position, direction, and size, which is an explicit expression. This process fits an explicitly represented three-dimensional Gaussian model;

[0049] GPU-optimized tiled rasterization is a technique that efficiently renders 3D scenes into 2D images by dividing the image into small blocks and using the parallel processing capabilities of the GPU. The randomly generated camera parameters are used to render the 3D Gaussian model using GPU-optimized tiled rasterization to obtain renderings from multiple perspectives.

[0050] Step 4 includes:

[0051] The user prompt word is input into the pre-trained text encoder CLIP to obtain the prompt word encoding vector, and the position of the topic word in the encoding vector is located according to the topic word.

[0052] Obtain multiple camera parameters randomly generated in step 3. The viewing angle control process can be adaptively implemented through the camera parameters of each optimization step, specifically including: when the azimuth angle in the camera parameters is located at ( )hour:

[0053]

[0054] in, The encoding vector representing the topic word without perspective description Content-independent forward view feature encoding Projection in direction;

[0055] When the azimuth is at ( ) hour:

[0056]

[0057] Among them, the weight adjusted according to the azimuth angle in the camera parameters reflect The injected intensity is the closer the azimuth angle in the camera parameters is to 180° or -180°, that is, the closer the current viewing angle is to the front and back faces, the weight Through this encoding level of perspective control, the stable diffusion model SD can more accurately understand the perspective control specified by the rendering camera. , without being disturbed by the common viewing preference in the stable diffusion model SD, making the view prior part Viewing angle control with camera parameter control More consistent, this step will obtain the new prompt word encoding after perspective control.

[0058] Next, the new prompt word encoding vector and the multiple perspectives obtained in step 3 are rendered Figure 1 Start by inputting the pre-trained stable diffusion model SD.

[0059] Step 5 includes:

[0060] During the training of the stable diffusion model SD, the data used is a pairing of text and images. However, the description of perspective in the text is usually rough and general, lacking the necessary quantification and accuracy. This vague perspective information limits the stable diffusion model SD from establishing a sound perspective perception ability during learning, which in turn makes it face significant difficulties in converting from 2D images to 3D content. To alleviate this problem, in step 4, by improving the perspective semantic clarity of user prompts during image generation, the user prompts and the generated objects are aligned in perspective semantics, thereby effectively alleviating the dependence of 3D generated content on prior perspective knowledge. However, the mathematical analysis of the "multi-faceted" problem in step 1 shows that another gradient term As an unconditional item, it does not interact with user prompts, and its effect on the perspective of the generated content depends only on the view prior knowledge of the stable diffusion model SD. Therefore, this item also potentially injects a prior perspective preference into the generated content. In order to eliminate this preference, the goal is to find a method that can establish a connection between the unconditional guidance item and the perspective control, or to make the unconditional item controlled by the camera parameters and inject perspective perception into the model.

[0061] The contrastive language image pre-training model is a machine learning model that is specifically used to measure the similarity between images and texts and obtain the cosine similarity score between images and texts. Starting from the distribution of cosine similarity scores between perspective description texts (such as "back", "side", etc.) and rendered images, the relationship between perspective control and rendered images is explored. The two groups of results with and without the "multi-faceted" problem are uniformly sampled. The sampling method is to sample 500 uniformly spaced camera parameters from the upper hemisphere of a fixed radius, all of which point to the center of the sphere at the same height, and render 500 images from the 3D scene. The cosine similarity of each image and the perspective description text is calculated through the pre-trained contrastive language image pre-training model. It is found that the distribution of cosine similarity scores of samples without the "multi-faceted" problem changes roughly periodically with the change of angle, and this change is roughly continuous.

[0062] Furthermore, the cosine similarity relationship between images is explored based on the characteristics of the unconditional term, and the distribution law of cosine similarity is obtained; first, a random image is selected from all the renderings as the reference image, and the cosine similarity between other renderings and the reference image is calculated. Considering the random fluctuations of the pitch angle and the field of view size during the optimization process and the flipping operation of the image, it is necessary to obtain more global features. The first layer vector in the pre-trained encoder ViT is an ideal choice, because the first layer vector in the encoder ViT not only retains the detailed information of each small block of the image, but also effectively integrates the overall layout and relationship of the entire image, so that the feature representation has a comprehensive global perspective. Through experiments, it is found that the cosine similarity distribution calculated from all renderings and the reference image is indeed related to the azimuth of the camera parameters. The closer the azimuth is to the reference image, the higher the cosine similarity score between the rendering and the reference image, and the score gradually decreases from the azimuth of the reference image to both sides. However, when the "multi-faceted" problem occurs, this cosine similarity distribution is disturbed, which provides a way to explicitly constrain the unconditional denoising results to stay away from the "multi-faceted" problem. Specifically, in each optimization step, a random Viewing angle control , and control according to the viewing angle The azimuth angle in the above gives these viewing angle controls an order that obeys the above cosine similarity distribution law. A rectangular coordinate system is established, and a unit circle is constructed with the origin as the center. The angles corresponding to the points on the unit circle start from 0° in the negative direction of the vertical axis and gradually increase to 180° in the positive direction of the horizontal axis and decrease to -180° in the negative direction of the horizontal axis. Place them on the unit circle according to the azimuth angle, then randomly select one of the view controls as the reference camera, and construct a secant line parallel to the horizontal axis on the unit circle with the view control as the starting point , calculate the distance secant line of the points corresponding to other perspective controls Distance ,according to Size pair Sort the camera controls.

[0063] Then, through these perspective controls Get the rendered image , Render the image Potential representation of , and rendered images The noisy potential representation of Potential representation of The corresponding unconditional noise prediction results after the stable diffusion model SD , The stable diffusion model SD is estimated The noise component after noise addition.

[0064] Then, Remove from the noised latent representation to get the unconditional denoising result , and calculate the unconditional denoising result The partial order loss between them, the loss function is :

[0065]

[0066] in, Indicates the viewing angle control in each iteration The number of express The index position in the perspective control, Representative Rendered images Corresponding unconditional denoising results The first layer encoding result of ViT encoder is express and The cosine similarity between . The term will penalize the unconditional denoising results that do not satisfy the cosine similarity distribution, making them closer to the cosine similarity distribution without the "multi-faceted" problem. This partial order loss can repair multi-faceted objects in the early stage of iterative optimization, thereby effectively alleviating the "multi-faceted" problem.

[0067] The present invention also provides a zero-sample 3D generation device for enhancing viewing angle consistency based on the method, comprising:

[0068] A text input module is used to obtain text prompt words input by the user;

[0069] Point cloud generation module, used to generate rough 3D point clouds based on text prompt words;

[0070] 3D fitting module, used to fit the rough point cloud using 3D Gaussian splashing technology to generate an explicit 3D Gaussian model;

[0071] A multi-view rendering module is used to render the 3D Gaussian model from multiple perspectives to obtain a multi-view rendering image;

[0072] View encoding control module, used to eliminate typical view preferences and inject view control to improve the view semantic consistency from text to 2D and then to 3D;

[0073] The text-generated image module is used to add noise to multi-view rendered images, predict noise based on user prompts and multi-view rendered images, and update 3D Gaussian model parameters based on the prediction results;

[0074] The similarity partial order loss constraint module is used to establish the perspective consistency between multi-view images and improve the perspective perception ability of the 3D model.

[0075] Progress tracking module, which monitors the training progress and records and displays various indicators during the training process;

[0076] The model checkpoint management module is responsible for saving and loading the model state to facilitate the continuity and recovery of model training;

[0077] The network interface module handles network communications, receives external commands and sends rendered images;

[0078] The video generation module generates a video file based on the rendered image.

[0079] The encoding control module comprises:

[0080] A view feature decoupling unit, used to extract view feature differences from text prompts;

[0081] A view control unit, which is used to eliminate typical view preferences and inject new view control by using the view feature difference after feature decoupling to enhance the view semantic clarity;

[0082] The text-to-image module comprises:

[0083] An image encoding unit, used for encoding multi-view images to obtain potential representations;

[0084] An image denoising unit, for denoising the latent representation of the multi-view image;

[0085] An image denoising unit, used to perform conditional denoising on the noisy image according to a user prompt, and also to perform unconditional denoising on the noisy image;

[0086] A noise comparison unit is used to compare the predicted noise with the added noise and calculate the gradient of the noise tensor;

[0087] A parameter updating unit, used to update the 3D Gaussian model parameters through a back propagation algorithm according to the noise tensor gradient;

[0088] The similarity partial order loss constraint module includes:

[0089] A camera parameter sorting unit, used to sort the randomly generated camera parameters according to the ideal cosine similarity distribution law;

[0090] An unconditional denoising result acquisition unit, used for selecting the unconditional predicted noise in the image denoising unit according to the sorted camera parameters, and obtaining the unconditional denoising result according to the unconditional predicted noise;

[0091] The partial order loss calculation unit is used to calculate the cosine similarity between the unconditional denoising results. The partial order loss is calculated according to the ideal cosine similarity distribution law. This loss will be used together with the parameter update unit to update the 3D Gaussian model parameters during back propagation.

[0092] The present invention also provides an electronic device, comprising a processor and a storage medium;

[0093] The storage medium is used to store instructions;

[0094] The processor is configured to execute the steps of the method according to the instructions.

[0095] The present invention also provides a computer-readable storage medium on which a computer program is stored, and the program implements the steps of the method when executed by a processor.

[0096] The present invention significantly improves the 3D consistency of the generated target from two levels. On the one hand, in response to the conflict caused by view priors, a perspective decoupling method VDM is proposed. The perspective decoupling method VDM is a perspective feature decoupling method that effectively eliminates the perspective semantic ambiguity caused by typical perspective preferences, and enhances perspective control when generating 3D content, significantly enhancing the consistency of perspective semantics in the process from text to image and then to 3D; on the other hand, in response to the lack of explicit 3D consistency constraints, the present invention explores the distribution law of image similarity and proposes a similarity partial order loss PSL to enhance the perspective perception ability in the process of 2D to 3D promotion. It has been proved that the method of the present invention constructs a strong correlation of perspective semantics in data of different modalities, effectively alleviating the "multi-faceted" problem.

[0097] Beneficial effects: The present invention provides a zero-sample 3D generation method for enhancing perspective consistency. Aiming at the common multi-perspective consistency problem in traditional methods, namely the "multi-faceted" problem, a systematic optimization measure is proposed, which significantly improves the geometric and semantic consistency of the 3D model under different perspectives. By introducing the perspective decoupling method VDM and the similarity partial order loss PSL, the present invention not only enhances the perspective perception ability of the model, but also effectively solves the generation conflict and perspective ambiguity problems caused by perspective prior through fine-grained perspective control. In addition, the present invention improves the rendering efficiency and the expressiveness of the model by using 3D Gaussian splashing technology, allowing large-scale and dynamic scenes to be processed efficiently in real time, which is particularly important for real-time applications such as virtual reality and game design. The explicit 3D representation is combined with high-quality perspective consistency to ensure the coherence and authenticity when observed from different perspectives, greatly improving the user experience and the application value of the model. Through the comprehensive application of these technologies, the present invention significantly reduces the frequency of the "multi-faceted" problem and improves the generation speed and quality of 3D content. This method is not only applicable to a single scene, but also can adapt to changing environments and complex scene requirements, providing a wider range of practicality and flexibility for 3D content creation. Therefore, the present invention has broad application prospects in the fields of industrial design, virtual reality, game design, etc., and is expected to promote the development of 3D generation technology and bring innovation and value to related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 This is a training flow chart of a zero-sample 3D generation method for enhancing perspective consistency provided in the first embodiment of the present invention.

[0099] Figure 2 This is a model framework diagram of a zero-sample 3D generation method for enhancing perspective consistency provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0100] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0101] like Figure 1As shown, an embodiment of the present invention provides a zero-sample 3D generation method for enhancing perspective consistency. First, a training process is performed: at the beginning of training, the environment and 3D Gaussian model are initialized, and the 3D Gaussian model is used to represent 3D objects. This step is mainly to prepare the environment and model structure for subsequent training. During rendering, the objects in the 3D Gaussian model are projected onto the imaging plane of the camera and the projected two-dimensional Gaussian distribution is distributed to each tile. Then, the checkpoint is used to restore the state of the previous training. If a checkpoint file is provided, the program will load the previous model parameters and state to continue the previously interrupted training. Otherwise, the training will start from the beginning. Then, a pre-trained diffusion model (stable diffusion model SD) and text encoding are set to guide the training of the model. The text encoding is a vector representation extracted by using the pre-trained text encoder CLIP to select the subject word using the received user prompt word, and these codes will be used in the subsequent visual feature decoupling process. Next, the user prompt is input into the pre-trained three-dimensional diffusion model to generate a rough 3D point cloud. Then, check whether all specified training iterations have been completed. If the iteration round reaches the preset maximum value, the training will stop. Otherwise, enter the next round of iteration. Next, dynamically adjust the learning rate and gradually reduce the learning rate during the training process to stabilize the convergence. Then, configure the camera parameters and set parameters such as position, viewing angle and focal length. These parameters are used to select the current optimized view. Next, render the scene using the currently set camera parameters to generate a set of images, which will be used to train the 3D Gaussian model. Then, encode the prompt word text using the pre-trained text encoder CLIP, and decouple the viewing features through the prepared topic word encoding, eliminate the typical viewing angle preference and inject viewing angle control, and encode the rendered image through the pre-trained image encoder VAE. Next, input the text encoding and the rendered image encoding into the stable diffusion model SD for noise prediction, and weightedly aggregate the obtained unconditional noise prediction and the text-guided conditional noise prediction, and use the unconditional noise to calculate the similarity partial order loss PSL, prediction noise loss, scale loss and total editing difference loss. Next, update the 3D Gaussian model parameters through backpropagation, while recording processor events to monitor training time and resource consumption, and check whether the current iteration meets the conditions for saving the model (such as the specified saving interval). If so, save the state and parameters of the current model. Finally, the current training status, model parameters, and logs are saved to disk for subsequent training or evaluation. After all training steps are completed, the entire process ends.

[0102] like Figure 2 As shown, the network framework of this embodiment includes: first, obtaining the user input text prompt word, and selecting the theme word in the prompt word, for example, in the prompt word "a squirrel eating a hamburger", the theme word is "squirrel", and the theme word is directly input into the pre-trained CLIP encoder to obtain the encoding vector of the theme word without perspective description . Connect the subject word to the descriptive words of three different perspectives, that is, connect "front perspective", "side perspective" and "back perspective" to the end of the subject word respectively, to construct three phrases containing the subject word with perspective descriptions. For example, the subject word "squirrel" connected to the perspective description phrases becomes: "squirrel, front perspective", "squirrel, side perspective" and "squirrel, back perspective".

[0103] Input the three phrases containing the subject word with the perspective description into the CLIP encoder respectively and locate the position of the subject word code to obtain the subject word code containing the corresponding perspective description ,and The difference is that the subject word encoding containing the corresponding perspective description Contains additional contextual information, which provides a way to extract Content-independent view feature encoding , this process uses the method of extracting the vertical component and can be formulated as:

[0104]

[0105] in, , indicating the keyword code containing the corresponding perspective description Encoded vector describing the topic word without perspective The projection part on the image is obtained by subtracting this projection component to obtain the content-independent viewing feature encoding. The extracted pure perspective difference can be used to encode other prompt words, thereby realizing perspective control. The user prompt word is input into the pre-trained text into the three-dimensional diffusion model PointE to generate rough three-dimensional point cloud data, which contains the basic geometric information of the object surface. Anisotropic Gaussian volume is a three-dimensional Gaussian distribution that allows different variances in different directions, that is, the shape of the distribution can be stretched or scaled along all directions. This feature enables it to adapt to the local structure of point cloud data more flexibly. By fitting anisotropic Gaussian volume, a Gaussian volume matching the local distribution of the point cloud can be generated in each local area of ​​the point cloud, forming a continuous and smooth representation. This fitting process takes into account the density distribution and local geometric structure of the point cloud, thereby more accurately capturing the shape details of the object. Anisotropic Gaussian volume is used to capture the geometric features of the rough three-dimensional point cloud data generated by the three-dimensional diffusion model PointE, and these Gaussian volumes directly define the geometric structure of the rough three-dimensional point cloud data generated by the three-dimensional diffusion model PointE through information such as position, direction, and size, which is an explicit expression. This process fits a three-dimensional Gaussian model of explicit representation. The GPU-optimized tiled rasterization technology is a technology that efficiently renders 3D scenes into 2D images by dividing the image into small blocks and using the parallel processing capabilities of the GPU. The randomly generated camera parameters are used to render the 3D Gaussian model using the GPU-optimized tiled rasterization technology to obtain renderings from multiple perspectives. The user prompt word is input into the pre-trained text encoder CLIP to obtain the prompt word encoding vector, and the position of the topic word in the encoding vector is located according to the topic word.

[0106] The viewing angle control process can be adaptively implemented by the camera parameters of each optimization step. Specifically, when the azimuth angle in the camera parameters is located at ( )hour:

[0107]

[0108] in, The encoding vector representing the topic word without perspective description Content-independent frontal view feature encoding Projection in direction.

[0109] When the azimuth is at ( ) hour:

[0110]

[0111] Among them, the weight adjusted according to the azimuth angle in the camera parameters reflect The injected intensity is the closer the azimuth angle in the camera parameters is to 180° or -180°, that is, the closer the current viewing angle is to the front and back faces, the weight Through this encoding level of perspective control, the stable diffusion model SD can more accurately understand the perspective control specified by the rendering camera. , without being disturbed by the common viewing preference in the stable diffusion model SD, making the view prior part Viewing angle control with camera parameter control This step will get the new prompt word encoding after the perspective control. Next, the new prompt word encoding vector and the rendering Figure 1 Start by inputting the pre-trained stable diffusion model SD.

[0112] Furthermore, the cosine similarity relationship between images is explored based on the characteristics of unconditional noise. First, a random image is selected from all renderings as the reference image, and the cosine similarity between other renderings and the reference image is calculated. Considering the random fluctuations of the pitch angle and the field of view size during the optimization process and the flipping operation of the image, it is necessary to obtain more global features. The first layer vector in the pre-trained encoder ViT is an ideal choice, because the first layer vector in the encoder ViT not only retains the detailed information of each small block of the image, but also effectively integrates the overall layout and relationship of the entire image, so that the feature representation has a comprehensive global perspective. Through experiments, it is found that the cosine similarity distribution calculated from all renderings and the reference image is indeed related to the azimuth of the camera parameters. The cosine similarity score between the rendering and the reference image corresponding to the azimuth closer to the reference image is higher, and the score gradually decreases from the azimuth of the reference image to both sides. However, when the "multi-faceted" problem occurs, this cosine similarity distribution is disturbed, which provides a way to explicitly constrain the unconditional denoising results away from the "multi-faceted" problem. Specifically, in each optimization step, a random Viewing angle control , and control according to the viewing angle The azimuth angle in the above gives these viewing angle controls an order that obeys the above cosine similarity distribution law. A rectangular coordinate system is established, and a unit circle is constructed with the origin as the center. The angles corresponding to the points on the unit circle start from 0° in the negative direction of the vertical axis and gradually increase to 180° in the positive direction of the horizontal axis and decrease to -180° in the negative direction of the horizontal axis. Place them on the unit circle according to the azimuth angle, then randomly select one of the view controls as the reference camera, and construct a secant line parallel to the horizontal axis with this point as the starting point , calculate the distance secant line of the points corresponding to other cameras Distance ,according to Size pair Then, through these perspective control Get the rendered image , Render the image Potential representation of And rendered images The noisy potential representation of Potential representation of The corresponding unconditional noise prediction results after the stable diffusion model SD , The model estimates The noise component after the noise addition process. Then, Remove from the noised latent representation to get the unconditional denoising result And calculate the unconditional denoising result The partial order loss between . The loss function is as follows:

[0113]

[0114] in, Indicates the viewing angle control in each iteration The number of express The first index positions, Representative Rendered images Corresponding unconditional denoising results The first layer encoding result of ViT encoder is ,express and The cosine similarity between . The term will penalize the unconditional denoising results that do not satisfy the cosine similarity distribution, making them closer to the cosine similarity distribution without the "multi-faceted" problem. This partial order loss can repair multi-faceted objects in the early stage of iterative optimization, thereby effectively alleviating the "multi-faceted" problem. Finally, the model parameters of the 3D Gaussian model are updated through the back propagation algorithm to improve the quality of the 3D generated objects.

[0115] Another embodiment of the present invention provides a zero-sample 3D generation device for enhancing viewing angle consistency, comprising:

[0116] A text input module is used to obtain text prompt words input by the user;

[0117] Point cloud generation module, used to generate rough 3D point clouds based on text prompt words;

[0118] 3D fitting module, used to fit the rough point cloud using 3D Gaussian splashing technology to generate an explicit 3D Gaussian model;

[0119] A multi-view rendering module is used to render the 3D Gaussian model from multiple perspectives to obtain a multi-view rendering image;

[0120] View encoding control module, used to eliminate typical view preferences and inject view control to improve the view semantic consistency from text to 2D and then to 3D;

[0121] The text-generated image module is used to add noise to multi-view rendered images, predict noise based on user prompts and multi-view rendered images, and update 3D Gaussian model parameters based on the prediction results;

[0122] The similarity partial order loss constraint module is used to establish the perspective consistency between multi-view images and improve the perspective perception ability of the 3D model.

[0123] Progress tracking module, which monitors the training progress and records and displays various indicators during the training process;

[0124] The model checkpoint management module is responsible for saving and loading the model state to facilitate the continuity and recovery of model training;

[0125] The network interface module handles network communications, receives external commands and sends rendered images;

[0126] The video generation module generates a video file based on the rendered image.

[0127] The encoding control module comprises:

[0128] A view feature decoupling unit, used to extract view feature differences from text prompts;

[0129] A view control unit, which is used to eliminate typical view preferences and inject new view control by using the view feature difference after feature decoupling to enhance the view semantic clarity;

[0130] The text-to-image module comprises:

[0131] An image encoding unit, used for encoding multi-view images to obtain potential representations;

[0132] An image denoising unit, for denoising the latent representation of the multi-view image;

[0133] An image denoising unit, used to perform conditional denoising on the noisy image according to a user prompt, and also to perform unconditional denoising on the noisy image;

[0134] A noise comparison unit is used to compare the predicted noise with the added noise and calculate the gradient of the noise tensor;

[0135] A parameter updating unit, used to update the 3D Gaussian model parameters through a back propagation algorithm according to the noise tensor gradient;

[0136] The similarity partial order loss constraint module includes:

[0137] A camera parameter sorting unit, used to sort the randomly generated camera parameters according to the ideal cosine similarity distribution law;

[0138] An unconditional denoising result acquisition unit, used for selecting the unconditional predicted noise in the image denoising unit according to the sorted camera parameters, and obtaining the unconditional denoising result according to the unconditional predicted noise;

[0139] The partial order loss calculation unit is used to calculate the cosine similarity between the unconditional denoising results. The partial order loss is calculated according to the ideal cosine similarity distribution law. This loss will be used together with the parameter update unit to update the 3D Gaussian model parameters during back propagation.

[0140] Another embodiment of the present invention provides an electronic device, including a processor and a storage medium;

[0141] The processor is configured to operate according to the instructions to execute the steps of the method.

[0142] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when executed by a processor.

[0143] The method of the embodiment of the present invention focuses on solving the multi-view consistency problem that is prevalent in the prior art, namely the "multi-faceted" problem. This problem is often manifested as inconsistent images generated for the same 3D object at different viewpoints, which seriously affects the authenticity and visual effect of the generated results. In order to solve this problem, the embodiment of the present invention introduces 3D Gaussian splashing technology, perspective decoupling method VDM and similarity partial order loss PSL to form a systematic optimization process. First, a text prompt word describing the 3D object to be generated is input, and the model generates a rough 3D point cloud based on the prompt word. Although this initially generated point cloud captures the basic shape of the object, it is insufficient in details and consistency. In order to improve the accuracy and consistency of the generation, the present invention uses 3D Gaussian splashing technology to fit the rough point cloud and convert it into an explicit 3D representation composed of anisotropic Gaussian bodies. Next, a 5000-step iterative optimization strategy is adopted, and each iteration process includes the following key steps: first, a certain perspective of the current 3D model is selected, and the projection image under the perspective is rendered; then, the text prompt words, the rendered image and the camera parameters are input into the pre-trained diffusion model, and the explicit parameters of the 3D model are updated through the generated 2D image. Through multiple iterative optimizations, the consistency of the 3D model under different perspectives is continuously enhanced. In order to further alleviate the multi-faceted problem, the present invention introduces the perspective decoupling method VDM and the similarity partial order loss PSL. The perspective decoupling method VDM is a perspective feature decoupling method that effectively eliminates the perspective semantic ambiguity caused by typical perspective preference, enhances perspective control when generating 3D content, and significantly enhances the consistency of perspective semantics in the process from text to image to 3D. On the other hand, in view of the lack of explicit 3D consistency constraints, the present invention explores the distribution law of image similarity and proposes the similarity partial order loss PSL to enhance the perspective perception ability in the process of 2D to 3D promotion. The experimental results show that this method significantly reduces the frequency of the "multi-faceted" problem during the generation process, and the generated 3D model has higher authenticity and consistency under multiple perspectives. Compared with traditional methods, the present invention not only improves the generation speed and quality, but also shows significant advantages in solving the problem of multi-perspective consistency. It can better meet the needs of industrial design, virtual reality, game design and other fields for high-quality 3D content.

[0144] The present invention provides a zero-sample 3D generation method for enhancing perspective consistency. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A zero-sample 3D generation method with enhanced view consistency, It is characterized in that The following steps are involved: Step 1: mathematically analyze the multifaceted problems in the process of generating three-dimensional objects and propose optimization goals; Step 2, obtain the text prompt word input by the user and select the theme word in the prompt word, connect the theme word to three different perspective description words: connect the front perspective, side perspective and back perspective to the end of the theme word, construct three phrases containing the theme word with perspective description, input the three phrases containing the theme word into the pre-trained text encoder CLIP respectively, obtain three encoding vectors, extract the encoding parts corresponding to the theme word from the three encoding vectors respectively, and then use the method of extracting vertical components to extract the features of the front perspective, side perspective and back perspective of the theme word object from the three theme word encodings respectively; Step 3: Input the user prompt word into the pre-trained text into the 3D diffusion model PointE to generate rough 3D point cloud data. Meanwhile, the rough 3D point cloud is fitted using the 3D Gaussian splashing technique to generate an explicitly represented 3D Gaussian model. Then, the 3D Gaussian model is rendered using randomly generated camera parameters to obtain renderings from more than two perspectives. Step 4: Input the user prompt word into the pre-trained text encoder CLIP to obtain the prompt word encoding vector, and locate the position of the subject word in the encoding vector according to the subject word. According to the camera parameters randomly generated in step 3, the features of the front view, side view and back view extracted in step 2 are respectively injected into the subject word encoding part in the user prompt word encoding, and the prompt word encoding after view control corresponding to all camera parameters is obtained. The new prompt word encoding vector and the rendering images of more than two view angles obtained in step 3 are input into the pre-trained stable diffusion model SD. Step 5: The stable diffusion model SD generates a corresponding two-dimensional image based on the user prompt word and the rendered image, extracts the unconditional prediction noise generated during the generation process, and obtains the unconditional denoising result. It then explores the similarity distribution of the image and uses the similarity partial order loss to constrain the unconditional denoising result, thereby enhancing the perspective consistency of the three-dimensional Gaussian model.

2. The method according to claim 1, It is characterized in that In step 1, the mathematical level analysis includes: The denoising diffusion probability model DDPM is used to optimize the following simplified objectives: ， in, Represents the loss function of the denoising diffusion probability model DDPM; expected value Represents the original sample data ,noise and time step Calculate the expected value, where the original sample data As the input data of the denoising diffusion probability model DDPM, the noise represents random noise that follows a standard normal distribution with a mean of 0 and a variance of 1; the time step Represents the diffusion time step, which is used to control the degree of noise addition. The time step in the diffusion model ranges from 1 to gradually increase; Denotes the denoising diffusion probability model DDPM in the model parameters Next, according to the noise perturbation time step The sample after and time step Predicted noise value; added noise and model prediction noise The square of the two norm between Measuring the difference between predicted noise and actual added noise, during the inference process, the denoising diffusion probability model DDPM is Starting from the probability density Sampling the previous sample from the normal distribution of ; The following simplified objectives are optimized by the latent diffusion model LDM: ， in, is the loss function of the latent diffusion model LDM, and the expected value Represents the original sample data The potential encoding result of , User prompts , random noise that follows a standard normal distribution and time step Perform expectation calculation; conditional information encoding Indicates prompts to the user through the text encoder CLIP The result of encoding is The second norm of the difference between the added noise and the predicted noise; In the framework of the denoising diffusion probability model DDPM and the latent diffusion model LDM, the generation of the image is regarded as the final noise state To the noise-free initial clear state The reverse recovery process is expressed by the following formula to generate the final noise-free initial clear state The probability density of : ， in, express The maximum time step that can be taken, represents the probability distribution of the final state of the diffusion process, Indicates that given a user prompt and the current potential representation Under the condition, generate the potential representation of the previous step The conditional probability of is the differential element in the integral; Constructing 3D parameters The unnormalized probability density function of : ， in, Represents the set of all viewpoints; Redefine 3D parameters The density function of For a given series of optimization steps Viewing angle control , User prompts And the 3D Gaussian model projection under the corresponding perspective Under the condition, each iteration step The product of the conditional likelihoods is expressed as: ， in, express The maximum number of optimization steps that can be taken, Represents the iteration step In the three-dimensional parameters The 3D model state of the control; From the perspective of the stable diffusion model's understanding of user prompts, user prompts Divide into content sections and perspective priors , content part and perspective priors is orthogonal; three-dimensional parameters The density function of The new expression is: ， in Indicates that it is subject to , taking the logarithm of both sides of the equation gives: ， Next, using the chain rule, right Gradient It is expressed as: ， in, Indicates the status of the 3D model About 3D parameters The partial derivative of Represents the logarithmic probability density About 3D Model Status The partial derivative of right Further expansion using Bayes' theorem: ， Further broken down into: ， Among them, the viewing angle control and perspective priors Point-wise conditional mutual information , and is considered constant in the current optimization step and is therefore further simplified terms and expand using the definition of conditional probability: 。 3. The method according to claim 2, It is characterized in that Step 2 includes: Get the user input text prompt word, select the theme word in the prompt word, and directly input the theme word into the pre-trained CLIP encoder to obtain the encoding vector of the theme word without perspective description ; Connect the subject word to the three descriptive words of different perspectives, that is, connect the end of the subject word to the front perspective, the side perspective, and the back perspective, respectively, to construct three phrases containing the subject word with perspective descriptions; Input the three phrases containing the subject word with the perspective description into the CLIP encoder respectively and locate the position of the subject word code to obtain the subject word code containing the corresponding perspective description ,extract Content-independent view feature encoding , using the method of extracting the vertical component, the formula is: ， in, , Indicates the keyword code containing the corresponding perspective description Encoded vectors describing the topic words without perspective The projection part on the The prior perspective features are decoupled from the keyword encoding in the user prompt to eliminate the typical view preference, and the perspective control information is injected on this basis. For the back perspective, the back perspective feature is injected after decoupling the front perspective and side perspective features, which is formulated as: ， in, Indicates the need to eliminate the perspective of prior knowledge, including the front and side. The encoding vector representing the topic word without perspective description Content-independent view feature encoding Direction projection, Represents the content-independent back view feature encoding, is a scale factor.

4. The method according to claim 3, It is characterized in that In step 3, the rough three-dimensional point cloud data contains basic geometric information of the object surface.

5. The method according to claim 4, It is characterized in that Step 4 includes: Input the user prompt word into the pre-trained text encoder CLIP to obtain the prompt word encoding vector, and locate the position of the topic word in the encoding vector according to the topic word; Obtain the randomly generated camera parameters in step 3. The viewing angle control process can be adaptively implemented through the camera parameters of each optimization step, specifically including: when the azimuth angle in the camera parameters is located at ( )hour: ， in, The encoding vector representing the topic word without perspective description Content-independent frontal view feature encoding Directional projection; When the azimuth is at ( ) hour: ， Among them, the weight adjusted according to the azimuth angle in the camera parameters reflect The injected intensity is the closer the azimuth angle in the camera parameters is to 180° or -180°, that is, the closer the current viewing angle is to the front and back faces, the weight The bigger; Next, the new prompt word encoding vector and the perspective rendering image obtained in step 3 are input into the pre-trained stable diffusion model SD.

6. The method according to claim 5, It is characterized in that Step 5 includes: Uniform sampling is performed on the two groups of results with and without the multi-faceted problem. The sampling method is to sample uniformly spaced camera parameters from the upper hemisphere of a fixed radius. All camera parameters point to the center of the sphere at the same height, and render images from the 3D scene to explore the cosine similarity relationship between images and obtain the cosine similarity distribution law. Randomly select one of all the renderings as the reference image, and calculate the cosine similarity between other renderings and the reference image. In each round of optimization, randomly generate Viewing angle control , and control according to the viewing angle The azimuth in the image determines an order for the viewing angle control, and the order obeys the cosine similarity distribution law; a rectangular coordinate system is established, and a unit circle is constructed with the origin as the center. The angles corresponding to the points on the unit circle start from 0° in the negative direction of the vertical axis and gradually increase to 180° in the positive direction of the horizontal axis and decrease to -180° in the negative direction of the horizontal axis. Place them on the unit circle according to the azimuth angle, then randomly select one of the view controls as the reference camera, and construct a secant line parallel to the horizontal axis on the unit circle with the view control as the starting point , calculate the distance secant line of the points corresponding to other perspective controls Distance ,according to Size pair Sorting by perspective control; Control by viewing angle Get the rendered image , Render the image Potential representation of , and rendered images The noisy potential representation of Potential representation of The corresponding unconditional noise prediction results after the stable diffusion model SD , The stable diffusion model SD is estimated Noise component after noise addition processing; Will Remove from the noised latent representation to get the unconditional denoising result , and calculate the unconditional denoising result The partial order loss between them, the loss function is : ， in, Indicates the viewing angle control in each iteration The number of express The index position in the perspective control, Representative Rendered images Corresponding unconditional denoising results The first layer encoding result of ViT encoder is express and The cosine similarity between .

7. An electronic device, It is characterized in that including processor and storage medium; The storage medium is used to store instructions; The processor is used to execute the steps of the method according to any one of claims 1 to 6 according to the instructions.

8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images