Method and apparatus for generating three-dimensional content based on search enhancement and personalization generation

Abstract

The application discloses a three-dimensional content generation method and device based on retrieval enhancement and personalized generation. First, image data D1 is constructed based on text retrieval pictures; personalized images P0 are generated based on the image data D1 and input text, and the P0 are processed to obtain personalized images P1 with transparent backgrounds; three-dimensional models G1 are obtained by the P1 through a three-dimensional model generation method; images P2 that can be rendered under random visual angles through the P1, D1 and G1 are predicted through a personalized image generation model to obtain personalized conditional noise, and unconditional noise is predicted through a diffusion model; appearance score matching distillation loss is calculated through the personalized conditional noise and the unconditional noise to optimize the three-dimensional model; finally, the process is repeated to obtain a final three-dimensional model. The application realizes innovation from the aspects of controllability and generation results, and can realize three-dimensional model generation with higher quality and controllability.

Description

Technical Field

[0001] This invention relates to the field of 3D generation, and in particular to a method and apparatus for generating 3D content based on search enhancement and personalized generation. Background Technology

[0002] With the rapid development of technology, the application scenarios of 3D digital assets in modern society are becoming increasingly widespread, especially in fields such as the metaverse, extended reality, digital games, film production, and virtual social interaction, where 3D models are gradually becoming a core pillar. The concept of the metaverse outlines a future blueprint for immersive virtual experiences, and 3D models are the cornerstone of this vision. From the construction of virtual environments to user-customized digital avatars, every aspect relies on the support of high-quality 3D models.

[0003] However, the creation of high-quality 3D models is complex and time-consuming, requiring not only meticulous mesh design and texture rendering but also specialized skills and tools, which limits large-scale applications. To address this issue, several innovative 3D content generation methods have been proposed, effectively lowering the barrier to entry for 3D model creation and significantly improving generation efficiency, providing a novel solution to meet the growing demand for 3D content.

[0004] In recent years, the field of 3D content creation has experienced rapid development, drawing inspiration from recent breakthroughs in 2D content generation. Current research mainly focuses on two types of methods: native 3D generation methods and 3D generation methods based on 2D optimization. Native 3D generation methods directly encode input into a 3D representation by training an encoder using a 3D dataset. However, these methods rely on large-scale 3D datasets, the creation of which often depends on specialized skills, making high-quality 3D data scarcer than other visual media such as images or videos. The key breakthrough of 3D generation methods based on 2D priors lies in using the score function predicted by a pre-trained 2D diffusion model, such as Stable Diffusion, to approximate the 3D score function. This score is then used to optimize the 3D model, resulting in a superior 3D model.

[0005] 3D generation methods based on 2D priors still face significant limitations when handling specific proper nouns, such as "pipa" and "hulusi," where models often fail to generate images that semantically correspond to these specific items. This deficiency directly leads to poor generation results for these specific names. On the other hand, image-driven 3D generation techniques also have limitations, relying on pre-input 2D images for 3D reconstruction and lacking flexibility and personalization. These methods cannot effectively customize generation to meet specific user needs, thus limiting their potential in diverse application scenarios. These challenges indicate that current 3D generation techniques still need further improvement in scalability and applicability to better meet the needs of complex applications. This method achieves customized generation by retrieving images corresponding to the input text through retrieval augmentation using a large model, and simultaneously optimizes the 3D model using appearance score matching distillation loss to obtain a high-quality 3D model. Summary of the Invention

[0006] To address the aforementioned shortcomings, this invention provides a method and apparatus for generating 3D content based on retrieval enhancement and personalized generation. This method requires only one sentence of text description of a 3D model to generate a 3D model corresponding to the text description. This method innovatively proposes a new mode of personalized learning based on retrieval enhancement and iteration. At the same time, it integrates retrieval enhancement, personalized image generation, and a classifier-free guidance module to better preserve the details of the 3D model, improve the generation quality and diversity, and achieve higher quality 3D model generation.

[0007] To achieve the above objectives, the technical solution of the present invention is: a method for generating 3D content based on retrieval enhancement and personalized generation, the method comprising the following steps:

[0008] (1) Obtain the corresponding text description object image through the retrieval model, select several images according to the requirements, and construct the corresponding data D1;

[0009] (2) Using the data D1 constructed in step (1) and the text description input by the user, the personalized image P0 is generated by the full generalized group image reference based on the diffusion model of the multimodal large language model, and the background of the personalized image P0 is removed to obtain the personalized image P1 with a transparent background.

[0010] (3) Using the personalized image P1 obtained in step (2), a preliminary three-dimensional Gaussian model G1 is generated based on the structured 3D latent representation using the three-dimensional model generation method;

[0011] (4) Based on the randomly selected camera pose, render the corresponding image P2 and surface normal image SN1 using the three-dimensional Gaussian model G1 obtained in step (3). Use the image P2 and the personalized image P1 obtained in step (2) to invert and add noise through the denoising diffusion implicit model to obtain the latent representations respectively. Use the data D1 obtained in step (1), the two latent representations and the surface normal image SN1 as personalized conditions, and input them into the model generated by the personalized image in step (2) to predict the personalized condition noise.

[0012] (5) Based on the latent representation based on image P2 obtained in step (4) and the unconditional noise predicted by surface normal image SN1, the three-dimensional Gaussian model G1 is optimized by combining personalized conditional noise to obtain the three-dimensional Gaussian model G2.

[0013] (6) Iterate through steps (4) and (5) until the requirements are met or the set number of iterations is reached, and generate a clear three-dimensional Gaussian model result.

[0014] Further, step (1) includes: performing data processing, retrieving images corresponding to the object names in the input text, allowing the user to select one or more suitable images, and constructing data D1 based on the images selected by the user.

[0015] Furthermore, step (2) includes the following steps:

[0016] (2.1) Input the data D1 obtained in step (1) into the visual encoder and the large language model to obtain the image embedding EP1. Input the text into the text encoder to obtain the text embedding V1. Concatenate the image embedding EP1 and the text embedding V1 as the input of the U-Net network of the personalized image generation model.

[0017] (2.2) Based on the diffusion model, a noise addition and denoising process is performed to obtain the initial latent representation;

[0018] (2.3) Input the initial latent representation into the image decoder to obtain the personalized image P0;

[0019] (2.4) Separate the foreground and background of the personalized image P0 obtained in step (2.3), calculate the bounding box of the foreground area of ​​the image, determine its height and width, scale it to the specified size, and place it in the center of the square canvas to obtain the personalized graphic P1 with a transparent background.

[0020] Furthermore, step (3) includes the following steps:

[0021] (3.1) Encode the personalized image P1 obtained in step (2) into a feature vector using DINOv2;

[0022] (3.2) Input the feature vector obtained in step (3.1) into the conditional stream converter to predict the coarse latent representation, and input the coarse latent representation into the decoder to obtain the set of position indices of active voxels on the 3D mesh;

[0023] (3.3) Input the feature vector obtained in step (3.1) and the set of position indices obtained in step (3.2) into the sparse flow converter to obtain the local features corresponding to each voxel;

[0024] (3.4) Construct a structured latent representation based on the local features and position indexes corresponding to each voxel;

[0025] (3.5) Input the constructed structured latent representation obtained in step (3.4) into the trained three-dimensional Gaussian decoder to obtain the preliminary three-dimensional Gaussian model G1.

[0026] Furthermore, step (4) includes the following steps:

[0027] (4.1) Randomly generate a camera pose within a given range, and randomly set the background to black or white according to the set background probability value. Use the camera pose, background color and the three-dimensional model G1 obtained in step (3) as parameters for differentiable rasterization, and render the three-dimensional model G1 into image P2 and surface normal image SN1 according to the camera pose.

[0028] (4.2) Input the image P2 obtained in step (4.1) and the personalized image P1 obtained in step (2) into the VAE encoder of the Diffusion model to convert them into two latent representations. Then, predict the latent representation under the time step condition based on the noise attenuation coefficient through the DDIM inversion process.

[0029] (4.3) Extract the key and value features in the attention calculation of the model generated from the personalized image based on the latent representation obtained from the personalized image P1 in step (4.2), and concatenate them with the key and value of the latent representation obtained from the personalized image P1 to obtain the key and value of the denoising network.

[0030] (4.4) Input the surface normal image SN1 obtained in step (4.1) into ControlNet and the embedded text image after splicing in step (2.1), and the latent representation obtained based on image P2 in step (4.2), and input them into the denoising network obtained in step (4.3) to predict the personalized conditional noise based on the latent representation obtained based on image P2.

[0031] Furthermore, step (5) includes the following steps:

[0032] (5.1) Input the surface normal image SN1 obtained in step (4), the latent representation based on image P2, and the time step into the denoising network of the Diffusion model with ControlNet to obtain unconditional noise;

[0033] (5.2) Conditional guidance for noise prediction based on CFG technology, using the personalized conditional noise obtained in step (4) and unconditional noise Calculate the appearance score matching distillation ASMD loss;

[0034] (5.3) The Adam optimizer was used to optimize the position information, color, opacity, scale and rotation parameters of the three-dimensional Gaussian model, and the learning rate was adjusted to obtain the three-dimensional Gaussian model G2.

[0035] Furthermore, the noise prediction methods used in steps (4) and (5) include:

[0036] Conditional guidance for noise prediction using CFG technology involves calculating the residual between conditionally generated and unconditionally generated prediction results and using conditional guidance coefficients to amplify the influence of conditions during the generation process, thereby achieving the effect of conditional control enhancement. Steps (4) and (5) use CFG with combined textual and image conditions to improve the accuracy of noise prediction.

[0037] Secondly, the present invention also provides a three-dimensional content generation device based on search enhancement and personalized generation, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the three-dimensional content generation method based on search enhancement and personalized generation.

[0038] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for generating three-dimensional content based on retrieval enhancement and personalized generation.

[0039] Fourthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for generating three-dimensional content based on retrieval enhancement and personalized generation.

[0040] The beneficial effects of this invention are: it can generate a detailed 3D model that meets user needs with just a piece of text; this method innovatively proposes a new mode of 3D object generation based on retrieval enhancement and personalized learning. The model can retrieve the image corresponding to the text description based on the input text, and then generate an image that meets the frontal view of the user's description model through a personalized image generation method. This image can generate a preliminary 3D model through a 3D model generation method. This method greatly improves the relevance between the generated model and the text description, enabling the model to generate 3D models corresponding to any meaningful text description, unaffected by the insufficiency of semantic understanding in the diffusion model itself; this method innovatively proposes to use the appearance score matching distillation loss method for 3D model refinement. Personalized conditional noise and unconditional noise are predicted by the personalized image generation model and the diffusion model, respectively. These two noises are used to calculate the appearance score matching distillation loss to optimize the model, making the model more detailed and consistent across different perspectives; moreover, the 3D object generation based on retrieval enhancement and personalized learning of this invention can generate 3D models corresponding to almost all meaningful descriptions. Both 3D professionals and amateurs can easily obtain 3D models through this method, greatly facilitating real-world 3D production needs. Attached Figure Description

[0041] Figure 1 This is a structural diagram of the 3D content generation method based on search enhancement and personalized generation provided by the present invention.

[0042] Figure 2 These are the search results, personalized result images, and 3D generated result images provided by this invention.

[0043] Figure 3 This is a structural diagram of a three-dimensional content generation device based on search enhancement and personalized generation provided by the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described below with reference to the accompanying drawings.

[0045] For 3D generation tasks geared towards practical applications, this invention proposes a 3D content generation method based on retrieval enhancement and personalized generation. This method can generate a detailed 3D model that meets user needs from a single text. After retrieving the image corresponding to the input text using a large retrieval model, the user selects a desired image. The image is then processed and a personalized image is generated using a personalized image generation method. This image is then used to generate a 3D model. This method overcomes the limitations of text-driven 3D generation, which cannot generate entities corresponding to uncommon descriptions, and also overcomes the limitations of image-driven 3D generation, which cannot personalize the generated results. Considering that generating a 3D model using only a single image may result in inconsistencies in detail, an additional appearance score matching distillation loss method is used to optimize the 3D model. This method predicts personalized conditional noise and unconditional noise separately to calculate the appearance score matching distillation loss, further optimizing the 3D model and improving the consistency of the generated results.

[0046] like Figure 1 and Figure 2 As shown in the figure, the present invention provides a method for generating 3D content based on retrieval enhancement and personalized generation, the specific steps of which are as follows:

[0047] (1) For example, if a user wants to generate a 3D model of a "pipa", they can input the keyword "pipa" and retrieve images of pipa by searching large models. The user can then select six images that meet the requirements to construct the image data D1 of "pipa".

[0048] (1.1) Perform data processing. Use searchng to retrieve images with the corresponding object names from the input text. Allow the user to select one or more suitable images and save the selected images to the local machine by accessing the API.

[0049] (2) Using the “pipa” image data D1 constructed in step (1) and the text description input by the user such as “pipa, a traditional Chinese musical instrument”, a personalized “pipa” image P0 with the same text description as the “pipa” image data is generated by simply referencing the personalized image generation method (model of personalized image generation). The background of the personalized “pipa” image P0 is removed to obtain a personalized “pipa” image P1 with a transparent background.

[0050] (2.1) The data D1 obtained in step (1.1) is processed by a visual encoder and a large language model to obtain image embedding EP1. The input text is processed by a text encoder to obtain text embedding V1. The image embedding EP1 and the text embedding V1 are then concatenated. This is used as input to the U-Net network of the personalized image generation model;

[0051] (2.2) In the forward pass of the Diffusion model, the latent representation is... Add noise gradually Time step Usually ,exist The latent representation after adding noise is , such as from Step-by-step noise reduction The process of each step is as follows:

[0052]

[0053] Gaussian noise in each sampling step They are independent of each other, therefore from arrive The noise addition can be done in one step, as shown in the following formula:

[0054]

[0055] Among them, the above All of these are a series of hyperparameters that are defined in advance, and have , Commonly used parameter settings are It gradually increases from 0.0001 to 0.002.

[0056] During the denoising process, a denoising network is trained. Predict exist Step-by-step noise ,Will Subtract the predicted noise to obtain the predicted value. This process iterates until the latent representation is gradually restored. For the initial latent representation of the prediction The following is from Step to The noise reduction process in step 1:

[0057]

[0058] In order to improve the diversity of images generated by the Diffusion model, It's additional noise.

[0059] In the adaptive pose generation model, the process of the denoising network predicting the noise at the corresponding time step is optimized using mean squared error loss, as shown below:

[0060]

[0061] in, Indicates the first Pose coding after adding noise This indicates that the conditions will be entered. Model mapped to conditional vectors, Indicates noise. It is a noise prediction network used for prediction. arrive The noise.

[0062] During the sampling process, the input is Gaussian noise, and the model needs to progressively denoise the Gaussian noise at each time step. At that time, the model predicts the corresponding noise. ,Will Subtract the predicted noise to obtain the predicted value. This process iterates until the Gaussian noise is gradually restored to the initial latent representation of the prediction. ;

[0063] (2.3) Initial latent representation A personalized image P0 is obtained through an image decoder;

[0064] (2.4) Standardize the resolution of the acquired image to 256. 256. Using the u2net network, the opacities of the foreground and background of the image are set to 255 and 0 respectively. A grayscale mask is generated by comparing the image opacity with 0; the portion greater than 0 is the foreground, and the portion equal to 0 is the background, thus achieving foreground-background separation. The bounding box of the foreground region of the image is calculated, its height and width are determined, it is scaled proportionally to a specified size, and placed in the center of a square canvas to obtain a personalized graphic P1 with a transparent background.

[0065] (3) Using the personalized “pipa” image P1 with transparent background obtained in step (2), a preliminary “pipa” 3D Gaussian model G1 is generated by the structured 3D latent representation (trellis) for scalable and general 3D generation through the 3D model generation method.

[0066] (3.1) Encode the personalized image P1 obtained in step (2.4) into a feature vector using DINOv2. ;

[0067] (3.2) The feature vector obtained in step (3.1) A coarse latent representation is predicted using a conditional flow transformer. , will potential representation The decoder obtains the set of position indices of active voxels on the 3D mesh. ;

[0068] (3.3) The feature vector obtained in step (3.1) and the set of location indices obtained in step (3.2) The local features corresponding to each voxel are obtained through a sparse flow transformer. ;

[0069] (3.4) Constructing a structured latent representation As shown below:

[0070]

[0071] in, These are the local features corresponding to each voxel obtained in step (3.3). It is the set of location indices obtained in step (3.2);

[0072] (3.5) Construct a structured latent representation from the result of step (3.4). The initial three-dimensional Gaussian model G1 is obtained by decoding using a trained three-dimensional Gaussian decoder.

[0073] (4) Randomly generate a camera pose The three-dimensional model G1 of the "pipa" obtained in step (3) is used to render the "pipa" image P2 and the surface normal image SN1 of the "pipa" from this viewpoint. The "pipa" image P2 and the personalized "pipa" image P1 with transparent background obtained in step (2) are inverted and denoised using the Denoising Diffusion Implicit Models (DDIM) to obtain the latent representation of the rendered "pipa" image. And the potential representation of personalized "pipa" images The image data D1 of the "pipa" obtained in step (1) and the latent representation , The rendered surface normal image SN1 of the "pipa" shape is used as a personalization condition, and the personalization condition noise is predicted by a model generated from the personalized image with ControlNet. .

[0074] (4.1) At the vertical angle Horizontal angle Within a radius of 0, generate a camera pose using a random number generation function. Simultaneously generate a random number; if the random number is greater than 0.5, randomly set the background to black; otherwise, set it to white. Adjust the camera pose. The background color and the three-dimensional model G1 obtained in step (4) are used as parameters for differentiable rasterization. The three-dimensional model G1 is then arranged according to the camera pose. Render image P2 and surface normal image SN1;

[0075] (4.2) The rendered image P2 obtained in step (4.1) and the personalized image P1 obtained in step (2.4) are converted into latent representations by the VAE encoder of the Diffusion model. , The DDIM inversion process is used to predict the two latent representations at time steps. Latent representation under certain conditions , As shown below:

[0076]

[0077] in, Indicates the time step The latitude of the step The latent representation of the input, It is a noise prediction network used for prediction. arrive noise;

[0078] (4.3) The latent representation obtained in step (4.2) Key-value features are extracted from attention calculations using a personalized image generation model, as shown below:

[0079]

[0080] in, For potential representation The query. For potential representation key value For potential representation The value (value) is the K, V in the last five self-attention layers of the U-Net model for personalized image generation, compared with... The original attention calculation method is replaced by splicing, as shown below:

[0081]

[0082]

[0083]

[0084] Where Q is the query in the U-Net layer. V is the key of the U-Net layer, and V is the value of the U-Net layer. For potential representation key, For potential representation value, for splicing with K, for By concatenating V with the denoising network, we obtain the denoising network. ;

[0085] (4.4) The surface normal image SN1 obtained in step (4.1) is processed by ControlNet, and the embedding obtained in step (2.1) is used to embed the surface normal image SN1. The latent representation obtained in step (4.2) The denoising network obtained in step (4.3) As shown below:

[0086]

[0087] in, The latent representation of the input , Indicates the time step of denoising , Indicates text image embedding Predict exist Step-by-step personalized conditional noise .

[0088] The noise prediction method used in step (4) includes: conditional guidance for noise prediction using additional classifier-free guidance (CFG) technology. CFG calculates the residual between the conditionally generated and unconditionally generated prediction results and uses conditional guidance coefficients to amplify the influence of conditions in the generation process, thereby achieving the effect of conditional control enhancement. Step (4) uses an improved CFG with joint textual and image conditions to improve the accuracy of noise prediction. The specific formula is as follows:

[0089]

[0090] in, Representing text conditions The result of the prediction This indicates the result of prediction without textual guidance. This represents the textual conditional guidance coefficient. Representing text conditions With image conditions Jointly guided prediction results These are the image conditional guidance coefficients. During the sampling process, for each time step... The models predict noise without textual guidance. There are text conditions Guided noise and text conditions With image conditions Jointly guided noise Guided by pre-set text conditions and image conditional guidance coefficients Calculations are performed to obtain the final result at time step. Predictive noise.

[0091] (5) The latent representation of the rendered "pipa" image obtained in step (4) And the surface normal image SN1 of the "Pipa" predicts unconditional noise using a diffusion model with ControlNet. The personalized conditional noise predicted in step (4) and unconditional noise The Appearance Score Matching Distillation (ASMD) loss was calculated to optimize the coarse "Pipa" 3D Gaussian model G1 to obtain the refined "Pipa" 3D Gaussian model G2.

[0092] (5.1) Combine the surface normal image SN1 obtained in step (4.1) with the latent representation obtained in step (4.3). Time step Denoising network using a Diffusion model with ControlNet As shown below:

[0093]

[0094] in, The latent representation of the input , Indicates the time step of denoising Predict exist Step-by-step unconditional noise ;

[0095] (5.2) The personalized conditional noise obtained in step (4.2) and unconditional noise The ASMD loss is calculated as follows:

[0096]

[0097] in, The latent representation obtained in step (4.2) represents the latent representation. , This indicates that the embedding obtained in step (2.1) will be used. , Indicates noise. It is a noise prediction network for a Diffusion model with ControlNet. It is a noise prediction network for a personalized image generation model with ControlNet;

[0098] (5.3) Optimize using the Adam optimizer and adjust the learning rate of each Gaussian point parameter individually. The learning rate of the position information is dynamically adjusted. The initial value is set to 0.001 and the final value is set to 0.00002. The color learning rate is set to 0.01, the opacity learning rate is set to 0.05, the scale parameter learning rate is set to 0.005, and the rotation parameter learning rate is set to 0.005 to obtain the three-dimensional model G2.

[0099] The noise prediction method used in step (5) includes: conditional guidance for noise prediction using additional CFG technology. CFG calculates the residual between the conditionally generated and unconditionally generated prediction results and uses conditional guidance coefficients to amplify the influence of conditions in the generation process, thereby achieving the effect of conditional control enhancement. Step (5) uses an improved CFG with combined textual and image conditions to improve the accuracy of noise prediction. The specific formula is as follows:

[0100]

[0101] in, Representing text conditions The result of the prediction This indicates the result of prediction without textual guidance. This represents the textual conditional guidance coefficient. Representing text conditions With image conditions Jointly guided prediction results These are the image conditional guidance coefficients. During the sampling process, for each time step... The models predict noise without textual guidance. There are text conditions Guided noise and text conditions With image conditions Jointly guided noise Guided by pre-set text conditions and image conditional guidance coefficients Calculations are performed to obtain the final result at time step. Predictive noise.

[0102] (6) Iterate through steps (4) and (5) 50 times to generate a clear and excellent 3D model G3.

[0103] Corresponding to the aforementioned embodiment of a 3D content generation method based on retrieval enhancement and personalized generation, the present invention also provides an embodiment of a 3D content generation device based on retrieval enhancement and personalized generation.

[0104] See Figure 3 The present invention provides a three-dimensional content generation device based on search enhancement and personalized generation, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a three-dimensional content generation method based on search enhancement and personalized generation in the above embodiment.

[0105] The embodiment of the 3D content generation device based on retrieval enhancement and personalized generation provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a 3D content generation device based on search enhancement and personalized generation provided by the present invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0106] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0107] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0108] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a 3D content generation method based on retrieval enhancement and personalized generation as described in the above embodiments.

[0109] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0110] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for generating 3D content based on search enhancement and personalized generation.

[0111] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for generating three-dimensional content based on search enhancement and personalization generation, characterized by: The method comprises the following steps: (1) obtaining the corresponding text description object picture by searching the large model with the text, selecting several pictures according to the requirements, and constructing the corresponding data D1; (2) using the data D1 constructed in step (1) and the text description input by the user, based on the diffusion model of the multi-modal large language model, the personalized image P0 is generated, and the background of the personalized image P0 is removed to obtain the personalized image P1 with transparent background; (3) using the personalized image P1 obtained in step (2), a preliminary three-dimensional Gaussian model G1 is generated based on the structured 3D latent representation by the three-dimensional model generation method; (4) according to the randomly selected camera pose, the three-dimensional Gaussian model G1 obtained in step (3) is rendered into a corresponding image P2 and a surface normal image SN1, the image P2 and the personalized image P1 obtained in step (2) are respectively inverted and added noise by the denoising diffusion implicit model to obtain two latent representations, and the data D1 obtained in step (1), the two latent representations and the surface normal image SN1 are input as personalized condition noise into the model for predicting personalized condition noise in step (2); (5) based on the latent representation based on the image P2 and the surface normal image SN1 obtained in step (4), the unconditional noise is predicted, and the three-dimensional Gaussian model G1 is optimized to obtain the three-dimensional Gaussian model G2 in combination with the personalized condition noise; (6) iteratively process steps (4) and (5) until the requirements are met or the set number of iterations is reached, and a clear three-dimensional Gaussian model result is generated.

2. The method of claim 1, wherein the method further comprises: The step (1) comprises: performing data processing, searching for pictures corresponding to the object name in the input text according to the name of the object in the input text, allowing the user to select several pictures, and constructing the data D1 based on the pictures selected by the user.

3. The method of claim 1, wherein the method further comprises: The step (2) comprises the following steps: (2.1) inputting the data D1 obtained in step (1) into a visual encoder and a large language model to obtain an image embedding EP1, inputting the text into a text encoder to obtain a text embedding V1, splicing the image embedding EP1 and the text embedding V1, and inputting the same into a U-Net network of a personalized image generation model; (2.2) performing a noise adding process and a noise removing process based on the diffusion model to obtain an initial latent representation; (2.3) inputting the initial latent representation into an image decoder to obtain a personalized image P0; (2.4) separating the personalized image P0 obtained in step (2.3) into foreground and background, calculating the bounding box of the foreground area of the picture, determining the height and width thereof, scaling to a specified size, and placing the same in the center of a square canvas to obtain a personalized image P1 with transparent background.

4. The method of claim 1, wherein the method further comprises: The step (3) comprises the following steps: (3.1) encoding the personalized image P1 obtained in step (2) into a feature vector by DINOv2; (3.2) inputting the feature vector obtained in step (3.1) into a conditional flow converter to predict a rough latent representation, and inputting the rough latent representation into a decoder to obtain a position index set of active voxels on a 3D grid; (3.3) input the eigenvector obtained in step (3.1) and the position index set obtained in step (3.2) into a sparse flow converter to obtain a local feature corresponding to each voxel; (3.4) construct a structured latent representation based on the local feature corresponding to each voxel and the position index; (3.5) input the constructed structured latent representation obtained in step (3.4) into the trained three-dimensional Gaussian decoder to decode to obtain a preliminary three-dimensional Gaussian model G1.

5. The method of claim 1, wherein the method further comprises: The step (4) comprises the following steps: (4.1) randomly generate a camera pose in a given range, randomly set the background as black or white according to the set background probability value, take the camera pose, the background color and the three-dimensional model G1 obtained in step (3) as the differentiable rasterization parameters, and render the three-dimensional model G1 according to the camera pose to obtain an image P2 and a surface normal image SN1; (4.2) input the image P2 obtained in step (4.1) and the personalized image P1 obtained in step (2) into the VAE encoder of the Diffusion model to convert into two latent representations, and predict the latent representation under the condition of time step based on the noise attenuation coefficient through the DDIM inversion process; (4.3) input the latent representation obtained in step (4.2) based on the personalized image P1 into the model for extracting the key and value features in the attention calculation, splice the key and value of the latent representation obtained based on the personalized image P1 to obtain the key and value of the denoising network; (4.4) input the surface normal image SN1 obtained in step (4.1) into the ControlNet, the spliced text image embedding in step (2.1), the latent representation obtained based on the image P2 in step (4.2), and the denoising network obtained in step (4.3) to predict the personalized conditional noise of the latent representation obtained based on the image P2.

6. The method of claim 1, wherein: The step (5) comprises the following steps: (5.1) input the surface normal image SN1 obtained in step (4), the latent representation based on the image P2, and the time step into the denoising network of the Diffusion model with the ControlNet to obtain the unconditional noise; (5.2) perform conditional guidance for noise prediction based on the CFG technology, calculate the appearance score matching distillation ASMD loss of the personalized conditional noise and the unconditional noise obtained in step (4); (5.3) use the Adam optimizer to optimize and adjust the learning rate to optimize the position information, color, opacity, scale and rotation parameters of the three-dimensional Gaussian to obtain a three-dimensional Gaussian model G2.

7. The method of claim 5 or 6, wherein: The noise prediction method used in steps (4) and (5) comprises: The condition guiding of noise prediction using CFG technology, CFG calculates the residual of the conditional and non-conditional generated prediction results, and uses the conditional guiding coefficient to amplify the influence of the condition in the generation process, so as to achieve the effect of conditional control enhancement, steps (4) and (5) use the CFG of the joint text condition and image condition to improve the accuracy of the predicted noise.

8. A three-dimensional content generation apparatus based on search enhancement and individualization generation, comprising a memory and one or more processors, wherein the memory stores executable code, and the executable code comprises the following steps: a search enhancement and individualization generation module, configured to: receive a search query; and generate a search result based on the search query. The processor executes the executable code to implement the three-dimensional content generation method based on retrieval enhancement and personalized generation in any one of claims 1-7.

9. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the three-dimensional content generation method based on retrieval enhancement and personalized generation in any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the three-dimensional content generation method based on retrieval enhancement and personalized generation in any one of claims 1-7.

Images