Three-dimensional model generation method and device, electronic equipment and storage medium

Through multi-view image generation processing and noise prediction technology, the problems of low efficiency and poor quality of three-dimensional model generation in the prior art are solved, the generation speed and quality of the three-dimensional model are improved, and the matching ability with text prompts is enhanced.

CN118918257APending Publication Date: 2024-11-08TSINGHUA UNIVERSITY +1
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410979924.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, the efficiency and quality of generating three-dimensional models based on text descriptions are low and the quality of three-dimensional models cannot be effectively solved.

Method used

By obtaining text prompts and reference images, multi-view image generation process extracts the target hidden feature map, generates volume density field and view angle enhancement feature maps, fine-tune the pre-trained image denoising model based on these feature maps, generates an initialized three-dimensional model, and The model parameters are adjusted through noise prediction until the end condition is met, and the target three-dimensional model is obtained.

Benefits of technology

The convergence speed and quality of the three-dimensional model are improved, the matching between the target three-dimensional model and text prompts is enhanced, better collaboration is achieved, and the generation effect and stability are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918257A_ABST
    Figure CN118918257A_ABST
Patent Text Reader

Abstract

The invention relates to a three-dimensional model method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the multi-view-angle image generation processing based on a reference image corresponding to a text prompt and the guidance of the view angle information of a plurality of preset view angles, generating a volume density field and a view angle enhanced feature map of each preset view angle based on the target hidden feature map of each preset view angle, and generating a view angle image of each preset view angle based on each view angle enhanced feature map; finely tuning the pre-training image denoising model based on the text prompt and the view angle image of each preset view angle to obtain a target image denoising model; and adding noise to a rendered image of the initialized three-dimensional model generated based on the volume density field to obtain a noise-added rendered image, inputting the text prompt and the noise-added rendered image into the target image denoising model for noise prediction, and adjusting the initialized three-dimensional model based on a noise prediction result to obtain a target three-dimensional model. The convergence speed and quality of the three-dimensional model are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional model method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, "text-generated 3D" technology has emerged, which is a technology that generates a three-dimensional (3D) model that conforms to the description of the text from a given text. This technology greatly simplifies the 3D model creation process that traditionally requires professional knowledge and heavy labor, making batch generation of 3D content possible.

[0003] However, the method of generating a three-dimensional model based on text description in the related art not only has low efficiency in generating the three-dimensional model, but also has poor quality in the generated three-dimensional model. Summary of the invention

[0004] The present disclosure provides a 3D model generation method, device, electronic device and storage medium to at least solve the problem of low efficiency and poor quality of generating 3D models based on text in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating a three-dimensional model is provided, comprising:

[0006] Acquire a text prompt and a reference image corresponding to the text prompt;

[0007] Based on the guidance of the reference image and the perspective information of multiple preset perspectives, a multi-perspective image generation process is performed. During the multi-perspective image generation process, a target latent feature map of each preset perspective is extracted, a volume density field and a perspective enhancement feature map of each preset perspective are generated based on the target latent feature map of each preset perspective, and a perspective image of each preset perspective is generated based on the perspective enhancement feature map of each preset perspective;

[0008] Based on the text prompt and the perspective images of each preset perspective, fine-tuning the pre-trained image denoising model to obtain a target image denoising model;

[0009] generating an initialized three-dimensional model based on the volume density field;

[0010] Noise is added to the rendered image of the initialized three-dimensional model to obtain a noisy rendered image, the text prompt and the noisy rendered image are input into the target image denoising model for noise prediction, and model parameters of the initialized three-dimensional model are adjusted based on the result of the noise prediction until an end condition is met to obtain a target three-dimensional model.

[0011] In some exemplary embodiments, the generating of the volume density field and the viewing angle enhancement feature map of each preset viewing angle based on the target latent feature map of each preset viewing angle includes:

[0012] Using the target latent feature map of each preset perspective as a feature plane of a three-dimensional space to construct the three-dimensional space;

[0013] Projecting each spatial point in the three-dimensional space onto each of the target latent feature maps, respectively, and determining the three-dimensional feature of each of the spatial points based on the projection result corresponding to each of the spatial points on each of the target latent feature maps;

[0014] Performing perspective enhancement processing on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points;

[0015] Based on the multi-view three-dimensional features of each of the spatial points, neural radiation field prediction is performed to obtain volume density field and color feature information;

[0016] Volume rendering is performed at each of the preset viewing angles based on the volume density field and the color feature information to obtain a viewing angle enhancement feature map at each of the preset viewing angles.

[0017] In some exemplary embodiments, performing perspective enhancement processing on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points includes:

[0018] splicing preset marks in the three-dimensional features of each of the spatial points respectively to obtain the marked three-dimensional features of each of the spatial points;

[0019] The marked three-dimensional features of each of the spatial points are respectively subjected to self-attention processing to obtain a multi-view three-dimensional feature of each of the spatial points.

[0020] In some exemplary embodiments, the neural radiation field prediction is performed based on the multi-view three-dimensional features of each of the spatial points to obtain the volume density field and color feature information, including:

[0021] For each of the multi-view three-dimensional features of the spatial point, extract the features of the position corresponding to the preset mark from the multi-view three-dimensional features to obtain the multi-view global features of the spatial point; and use the multi-view three-dimensional features after extracting the multi-view global features as the view enhancement features of the spatial point;

[0022] Based on the multi-view global features of each of the spatial points, volume density prediction is performed to obtain the predicted volume density of each of the spatial points; the volume density field includes the predicted volume density of each of the spatial points;

[0023] The color feature information is obtained based on the viewing angle enhancement feature of each of the spatial points.

[0024] In some exemplary embodiments, determining the three-dimensional feature of each of the spatial points based on the projection result corresponding to each of the spatial points on each of the target latent feature maps includes:

[0025] For each of the spatial points, based on the projection results and linear interpolation corresponding to the spatial point on each of the target latent feature maps, a feature vector of each of the feature planes corresponding to the spatial point is obtained;

[0026] The feature vector of each feature plane corresponding to the spatial point is superimposed with the perspective embedding vector corresponding to the corresponding feature plane to obtain the three-dimensional feature of the spatial point; the perspective embedding vector corresponding to each feature plane is obtained based on the perspective information of the preset perspective corresponding to the feature plane.

[0027] In some exemplary embodiments, the step of inputting the text prompt and the noisy rendered image into the target image denoising model for noise prediction, and adjusting the model parameters of the initialized 3D model based on the result of the noise prediction until an end condition is satisfied, to obtain the target 3D model comprises:

[0028] Inputting the text prompt and the denoised rendered image into the target image denoising model, performing a first denoising process and a second denoising process on the denoised rendered image respectively, obtaining a first predicted noise in the first denoising process and a second predicted noise in the second denoising process; the first denoising process is performed under the guidance of the text prompt, and the second denoising process is performed without the guidance of the text prompt;

[0029] Determine a first loss term based on a difference between the first predicted noise and the second predicted noise; determine a second loss term based on the second predicted noise;

[0030] Based on the first loss term and the second loss term, the model parameters of the initialized three-dimensional model are adjusted, and iteration is performed based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0031] In some exemplary embodiments, adjusting the model parameters of the initialized three-dimensional model based on the first loss term and the second loss term, and iterating based on the adjusted model parameters until a preset end condition is satisfied to obtain the target three-dimensional model includes:

[0032] Determine a proportional hyperparameter based on a difference between the second loss term and the first loss term; the proportional hyperparameter is a positive number less than 1;

[0033] Linearly fuse the first loss term and the second loss term based on the proportional hyperparameter to obtain a fusion loss;

[0034] The model parameters of the initialized three-dimensional model are adjusted based on the fusion loss, and iteration is performed based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0035] In some exemplary embodiments, the generating of the viewing angle image corresponding to each preset viewing angle based on the viewing angle enhancement feature map of each preset viewing angle includes:

[0036] For each of the preset viewing angles, superimposing the target latent feature map of the preset viewing angle and the viewing angle enhancement feature map to obtain a feature map to be decoded of the preset viewing angle;

[0037] The feature images to be decoded of each of the preset viewing angles are decoded to obtain viewing angle images corresponding to each of the preset viewing angles.

[0038] According to a second aspect of an embodiment of the present disclosure, there is provided a three-dimensional model generating device, comprising:

[0039] A first acquisition unit is configured to acquire a text prompt and a reference image corresponding to the text prompt;

[0040] A multi-view image generation unit is configured to perform a multi-view image generation process based on the reference image and the view information of a plurality of preset view angles, wherein during the multi-view image generation process, a target latent feature map of each preset view angle is extracted, a volume density field and a view enhancement feature map of each preset view angle are generated based on the target latent feature map of each preset view angle, and a view image of each preset view angle is generated based on the view enhancement feature map of each preset view angle;

[0041] A pre-trained model fine-tuning unit is configured to perform model fine-tuning on the pre-trained image denoising model based on the text prompt and the perspective images of each preset perspective to obtain a target image denoising model;

[0042] A three-dimensional model initialization unit, configured to generate an initialized three-dimensional model based on the volume density field;

[0043] The three-dimensional model generation unit is configured to add noise to the rendered image of the initialized three-dimensional model to obtain a noisy rendered image, input the text prompt and the noisy rendered image into the target image denoising model for noise prediction, and adjust the model parameters of the initialized three-dimensional model based on the result of the noise prediction until an end condition is met to obtain a target three-dimensional model.

[0044] In some exemplary embodiments, the multi-view image generation unit includes:

[0045] A three-dimensional space construction unit is configured to construct the three-dimensional space by using the target latent feature map of each preset perspective as a feature plane of the three-dimensional space;

[0046] A spatial point representation unit is configured to project each spatial point in the three-dimensional space onto each target latent feature map, and determine the three-dimensional feature of each spatial point based on the projection result corresponding to each spatial point on each target latent feature map;

[0047] A spatial point representation enhancement unit is configured to perform a perspective enhancement process on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points;

[0048] A neural radiation field prediction unit is configured to perform neural radiation field prediction based on multi-view three-dimensional features of each of the spatial points to obtain volume density field and color feature information;

[0049] The volume rendering unit is configured to perform volume rendering at each of the preset viewing angles based on the volume density field and the color feature information to obtain a viewing angle enhancement feature map at each of the preset viewing angles.

[0050] In some exemplary embodiments, the spatial point representation enhancement unit includes:

[0051] A marking unit is configured to execute splicing of preset marks in the three-dimensional features of each of the spatial points to obtain a marked three-dimensional feature of each of the spatial points;

[0052] The attention processing unit is configured to perform self-attention processing on the marked three-dimensional features of each of the spatial points respectively to obtain the multi-view three-dimensional features of each of the spatial points.

[0053] In some exemplary embodiments, the neural radiation field prediction unit includes:

[0054] The multi-view global feature extraction unit is configured to perform multi-view three-dimensional features for each of the spatial points, extract features of the position corresponding to the preset mark from the multi-view three-dimensional features, and obtain multi-view global features of the spatial point; and use the multi-view three-dimensional features after extracting the multi-view global features as the view enhancement features of the spatial point;

[0055] A volume density prediction unit is configured to perform volume density prediction based on the multi-view global features of each of the spatial points to obtain a predicted volume density of each of the spatial points; the volume density field includes the predicted volume density of each of the spatial points;

[0056] The color feature information determining unit is configured to perform a viewing angle enhancement feature based on each of the spatial points to obtain the color feature information.

[0057] In some exemplary embodiments, the spatial point representation unit includes:

[0058] A projection interpolation unit is configured to perform, for each of the spatial points, a projection result corresponding to the spatial point on each of the target latent feature maps and a linear interpolation to obtain a feature vector corresponding to each of the feature planes of the spatial point;

[0059] The vector superposition unit is configured to superimpose the feature vector of each feature plane corresponding to the spatial point with the perspective embedding vector corresponding to the corresponding feature plane to obtain the three-dimensional feature of the spatial point; the perspective embedding vector corresponding to each feature plane is obtained based on the perspective information of the preset perspective corresponding to the feature plane.

[0060] In some exemplary embodiments, the three-dimensional model generating unit includes:

[0061] a noise prediction unit configured to input the text prompt and the denoised rendered image into the target image denoising model, perform a first denoising process and a second denoising process on the denoised rendered image respectively, and obtain a first predicted noise in the first denoising process and a second predicted noise in the second denoising process; the first denoising process is performed under the guidance of the text prompt, and the second denoising process is performed without the guidance of the text prompt;

[0062] a multi-loss term determination unit, configured to determine a first loss term based on a difference between the first predicted noise and the second predicted noise; and determine a second loss term based on the second predicted noise;

[0063] The three-dimensional model parameter adjustment unit is configured to adjust the model parameters of the initialized three-dimensional model based on the first loss term and the second loss term, and iterate based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0064] In some exemplary embodiments, the three-dimensional model parameter adjustment unit is specifically configured to perform: determining a proportional hyperparameter based on the difference between the second loss term and the first loss term; the proportional hyperparameter is a positive number less than 1; linearly fusing the first loss term and the second loss term based on the proportional hyperparameter to obtain a fusion loss; adjusting the model parameters of the initialized three-dimensional model based on the fusion loss, and iterating based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0065] In some exemplary embodiments, when the multi-view image generation unit generates the view images corresponding to each preset view based on the view enhancement feature map of each preset view, it is specifically used to: for each of the preset view, superimpose the target latent feature map of the preset view with the view enhancement feature map to obtain the feature map to be decoded of the preset view; and decode the feature map to be decoded of each of the preset view to obtain the view images corresponding to each of the preset view.

[0066] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0067] processor;

[0068] a memory for storing instructions executable by the processor;

[0069] The processor is configured to execute the instructions to implement the three-dimensional model generation method of the first aspect.

[0070] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the three-dimensional model generation method of the first aspect described above.

[0071] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which, when executed by a processor, implements the three-dimensional model generating method of the first aspect.

[0072] The disclosed embodiment obtains a text prompt and a reference image corresponding to the text prompt, and performs multi-view image generation processing based on the reference image and the view information of multiple preset view angles. In the multi-view image generation processing, a target latent feature map of each preset view angle is extracted, a volume density field and an enhanced feature map of each preset view angle are generated based on the target latent feature map of each preset view angle, and a view image corresponding to each preset view angle is generated based on the enhanced feature map of each preset view angle. Then, a pre-trained image denoising model is fine-tuned based on the text prompt and the view image of each preset view angle to obtain a target image denoising model, and an initialization 3D model is generated based on the generated volume density field, and noise is added to the rendered image of the initialization 3D model to obtain a noisy rendered image. The text prompt and the noisy rendered image are input into the aforementioned target image denoising model for noise prediction, and the model parameters of the initialization 3D model are adjusted based on the result of the noise prediction until the end condition is met to obtain the target 3D model, thereby unifying the generation direction of the initialization 3D model and the target image denoising model, achieving better collaboration, improving the convergence speed and quality of the 3D model, and achieving better matching between the target 3D model and the text prompt.

[0073] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0075] Figure 1 is a schematic diagram of an application environment of a three-dimensional model generation method according to an exemplary embodiment;

[0076] Figure 2 is a flow chart of a method for generating a three-dimensional model according to an exemplary embodiment;

[0077] Figure 3 is a flow chart of another three-dimensional model generation method according to an exemplary embodiment;

[0078] Figure 4 is a flowchart of another three-dimensional model generation method according to an exemplary embodiment;

[0079] Figure 5 This is a flow example of a method for generating a three-dimensional model according to an exemplary embodiment;

[0080] Figure 6 is an example diagram of performing multi-view image generation processing using a multi-view diffusion model with an integrated latent plane module according to an exemplary embodiment;

[0081] Figure 7 is a structural block diagram of a three-dimensional model generating device according to an exemplary embodiment;

[0082] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0083] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0085] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0086] See also Figure 1 , which is a schematic diagram of an application environment of a three-dimensional model generation method according to an exemplary embodiment, the application environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0087] The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 110 may be installed with client software such as an application (Application, referred to as App) that provides a three-dimensional content generation function. The application may be an application that specifically provides three-dimensional content generation, or may be other applications that have a three-dimensional content generation function, such as a live broadcast application, a short video application, etc. that have a three-dimensional content generation function. The application may be an independent application or a subroutine in the application. The user of the terminal 110 may log in to the application using pre-registered user information, which may include an account number and a password.

[0088] The server 120 may be a server that provides background services for the application in the terminal 110, and the background services may be services that generate corresponding three-dimensional content based on text. Specifically, when the terminal 110 needs to generate three-dimensional content, it may send the corresponding text to the server 120, and the server 120 performs three-dimensional model generation processing based on the text to generate a target three-dimensional model that conforms to the text description, i.e., three-dimensional content, and returns the target three-dimensional model to the terminal 110.

[0089] In the embodiment of the present application, the server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or an edge computing node, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0090] Figure 2 is a flow chart of a method for generating a three-dimensional model according to an exemplary embodiment. Figure 2 As shown, the three-dimensional model generating method of the embodiment of the present disclosure may include the following steps S201 to S209.

[0091] In step S201, a text prompt and a reference image corresponding to the text prompt are obtained.

[0092] The text prompt is used to describe the target three-dimensional model to be generated. The reference image corresponding to the text prompt is related to the description content of the text prompt. In a specific implementation, the reference image can be an image generated by inputting the text prompt into a text-generated graph model, wherein the text-generated graph model uses a deep learning algorithm to convert the text description into an image pixel value to achieve text-to-image conversion.

[0093] In step S203, a multi-view image generation process is performed based on the guidance of the reference image and the view information of multiple preset view angles. During the multi-view image generation process, a target latent feature map of each preset view angle is extracted, a volume density field and a view enhancement feature map of each preset view angle are generated based on the target latent feature map of each preset view angle, and a view image of each preset view angle is generated based on the view enhancement feature map of each preset view angle.

[0094] The perspective information of each preset perspective may include camera parameters of the virtual camera, such as the three-dimensional coordinates (x, y, z) and the observation angle of the location of the virtual camera. In a specific implementation, the multiple preset perspectives may be set in advance based on actual needs, for example, may include four different perspectives.

[0095] Specifically, the multi-view image generation process is performed based on the guidance of the reference image and the perspective information of multiple preset perspectives, and a noise image is randomly selected for each preset perspective, and then the noise image is denoised under the guidance of the reference image and the perspective information of the preset perspective. In a specific implementation, the multi-view image generation process can be implemented based on a generation model from text to multi-view images, and the generation model from text to multi-view images is hereinafter referred to as a multi-view diffusion model.

[0096] Among them, the target latent feature map corresponding to each preset perspective can select the feature map with the highest resolution and the smallest index in the denoising process, because it contains comprehensive geometric information, which is conducive to improving the accuracy of the predicted image. Exemplarily, the target latent feature map can be the last layer of encoded latent features.

[0097] In the disclosed embodiment, in order to improve the convergence speed and quality of the three-dimensional model, in the process of multi-view image generation and processing, the target latent feature map of each preset view is extracted, and the volume density field and the view enhancement feature map of each preset view are reconstructed based on the target latent feature map of each preset view, and then the view image of each preset view is generated based on the view enhancement feature map of each preset view. Compared with directly using the target latent feature map of each preset view to generate the view image of the corresponding preset view, the prediction accuracy of the view image of each preset view can be significantly improved.

[0098] Exemplarily, when generating viewing images corresponding to each preset viewing angle based on the viewing angle enhancement feature map of each preset viewing angle, for each preset viewing angle, the target latent feature map of the preset viewing angle and the viewing angle enhancement feature map can be superimposed to obtain the feature map to be decoded of the preset viewing angle, and then the feature map to be decoded of each preset viewing angle is decoded to obtain the viewing image corresponding to each preset viewing angle, so that the superposition of the target latent feature map and the viewing angle enhancement feature map of each preset viewing angle makes the feature map to be decoded of each preset viewing angle contain richer viewing angle information, thereby improving the accuracy of the viewing image of each preset viewing angle.

[0099] In some exemplary embodiments, Figure 3 As shown, the above step S203 may include, when generating the volume density field and the viewing angle enhancement feature map of each preset viewing angle based on the target latent feature map of each preset viewing angle:

[0100] S301, using the target latent feature map of each preset viewing angle as a feature plane of a three-dimensional space to construct the three-dimensional space.

[0101] S303, projecting each spatial point in the three-dimensional space onto each of the target latent feature maps respectively, and determining the three-dimensional features of each of the spatial points based on the projection results corresponding to each of the spatial points on each of the target latent feature maps.

[0102] In a specific implementation, the three-dimensional space is constructed by taking the target latent feature map of each preset perspective as the feature plane of the three-dimensional space, so that the representation of the spatial points can be achieved by using the three-plane "Tri-Plane" expression.

[0103] Exemplarily, based on the projection result corresponding to each of the spatial points on each target latent feature map, determining the three-dimensional feature of each of the spatial points may include:

[0104] For each of the spatial points, based on the projection results and linear interpolation corresponding to the spatial point on each of the target latent feature maps, a feature vector of each of the feature planes corresponding to the spatial point is obtained;

[0105] The feature vectors of the spatial points corresponding to the feature planes are spliced ​​with the perspective embedding vectors corresponding to the corresponding feature planes to obtain the three-dimensional features of the spatial points, wherein the perspective embedding vectors corresponding to the feature planes are obtained based on the perspective information of the preset perspectives corresponding to the feature planes.

[0106] Specifically, any spatial point in the three-dimensional space is projected onto each feature plane (i.e., the target latent feature map), and then trilinear interpolation is used to obtain the feature vector corresponding to each preset view. Thus, for N target latent feature maps of N preset viewpoints, any spatial point in the three-dimensional space can collect N feature vectors from the N preset viewpoints, and the N feature vectors are used to realize the preliminary representation of the spatial point. The preliminary representation of any spatial point i can be

[0107] In order to better represent the spatial point, the perspective information of the preset perspective corresponding to each feature plane (i.e., the target latent feature map) is further embedded (Embedding), so that the perspective embedding vector of each preset perspective can be obtained, including the camera position embedding vector and the camera parameter embedding vector. Then, the feature vector of the spatial point at each preset perspective is added to the perspective embedding vector of the preset perspective to achieve vector superposition, and the three-dimensional feature of the spatial point is obtained. The three-dimensional feature of any exemplary spatial point i can be

[0108] The above-mentioned implementation method achieves a more accurate representation of the spatial point by projecting any spatial point in the three-dimensional space onto each feature plane, and then using trilinear interpolation to obtain the feature vector corresponding to the spatial point at each preset perspective, and further superimposing the perspective embedding vector of the corresponding preset perspective on the feature vector of each preset perspective, which is beneficial to improve the accuracy of the subsequent volume density field and color feature information, thereby improving the accuracy of the initial three-dimensional model.

[0109] S305, performing perspective enhancement processing on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points.

[0110] Specifically, for the three-dimensional feature of each spatial point, self-attention processing can be performed on the three-dimensional feature to enhance the perspective information.

[0111] S307, predicting the neural radiation field based on the multi-view three-dimensional features of each of the spatial points to obtain volume density field and color feature information.

[0112] The volume density field includes the volume density of each spatial point, and the color feature information includes the color feature of each spatial point.

[0113] Specifically, the multi-view three-dimensional features of each spatial point can be respectively input into a multilayer perceptron (MLP) network for decoding processing, so as to output the volume density and color features of each spatial point.

[0114] In some exemplary embodiments, in order to obtain a more accurate three-dimensional model, the aforementioned step S305 may include: performing perspective enhancement processing on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points.

[0115] splicing preset marks in the three-dimensional features of each of the spatial points respectively to obtain the marked three-dimensional features of each of the spatial points;

[0116] The marked three-dimensional features of each of the spatial points are respectively subjected to self-attention processing to obtain a multi-view three-dimensional feature of each of the spatial points.

[0117] The preset mark can be set based on actual needs. For example, the preset mark can be [α]. The preset mark is spliced ​​into the three-dimensional features of each spatial point to obtain the marked three-dimensional features of each spatial point. For example, the marked three-dimensional features of any spatial point i can be

[0118] After self-attention processing is performed on the marked three-dimensional features of each spatial point, the features corresponding to the preset mark [α] position can fuse the information of each preset perspective. Subsequently, the volume density prediction is performed based on the features corresponding to the preset mark position, which can improve the prediction accuracy of the volume density corresponding to each spatial point.

[0119] Based on this, the aforementioned step S307 may include the following steps when performing neural radiation field prediction based on the multi-view three-dimensional features of each of the spatial points to obtain volume density field and color feature information:

[0120] For each of the multi-view three-dimensional features of the spatial point, extract the features of the position corresponding to the preset mark from the multi-view three-dimensional features to obtain the multi-view global features of the spatial point; and use the multi-view three-dimensional features after extracting the multi-view global features as the view enhancement features of the spatial point;

[0121] Based on the multi-view global features of each of the spatial points, volume density prediction is performed to obtain volume density information of each of the spatial points; the volume density field includes the volume density information of each of the spatial points;

[0122] The color feature information is obtained based on the viewing angle enhancement feature of each of the spatial points.

[0123] Specifically, the multi-view global features of each spatial point can be input into the MLP for volume density prediction to obtain the predicted volume density of each spatial point, thereby obtaining the volume density field. Since the perspective enhancement features of each spatial point have achieved perspective enhancement after the aforementioned self-attention processing, the above-mentioned perspective enhancement features of each spatial point can be directly used as its color features, thereby obtaining color feature information.

[0124] The above-mentioned implementation method obtains the multi-view global feature by extracting the features of the corresponding position of the preset mark from the multi-view three-dimensional features of each spatial point, and predicts the volume density of the spatial point based on the multi-view global feature. Since the multi-view global feature contains information of multiple preset viewpoints, the accuracy of the predicted volume density of each spatial point is improved, the accuracy of the volume density field is improved, and then the accuracy of the subsequent generation of the initial three-dimensional model based on the volume density field is improved.

[0125] S309, performing volume rendering at each of the preset viewing angles based on the volume density field and the color feature information, to obtain a viewing angle enhancement feature map at each of the preset viewing angles.

[0126] Specifically, the color features of each spatial point position on the light corresponding to each preset viewing angle can be weightedly summed to obtain pixel information of each pixel point on the viewing angle enhancement feature map of the corresponding preset viewing angle, thereby collecting the features enhanced at each spatial point.

[0127] The above-mentioned implementation method constructs the three-dimensional space by using the target latent feature map of each preset perspective as the feature plane of the three-dimensional space, and then uses the target latent feature map of each preset perspective to represent each spatial point to obtain the three-dimensional feature of each spatial point, and performs perspective enhancement processing on the three-dimensional feature of each spatial point to obtain the corresponding multi-perspective enhancement feature, reconstructs the volume density field based on the multi-perspective enhancement feature of each spatial point, and uses the reconstruction result to obtain the perspective enhancement feature map of each preset perspective through volume rendering, thereby improving the accuracy of multi-perspective prediction, which is beneficial to improving the accuracy of subsequent target three-dimensional models.

[0128] In step S205, based on the text prompt and the perspective images of each of the preset perspectives, the pre-trained image denoising model is fine-tuned to obtain a target image denoising model.

[0129] Among them, the pre-trained image denoising model can be a text-generated diffusion model that generates images from text, such as Stable Fusion (stable diffusion model), ControlNet (control network), etc. ControlNet is a neural network structure that controls the diffusion model by adding additional conditions.

[0130] In a specific implementation, random noise can be added to the perspective images of each preset perspective to obtain a noisy perspective image, and then each noisy perspective image is input into a pre-trained image denoising model, and a text prompt is used as a guiding condition to guide the pre-trained image denoising model to denoise each noisy perspective image, and the fine-tuned loss value is determined based on the difference between the predicted noise and the added noise of each noisy perspective image during the denoising process, and the model parameters of the pre-trained image denoising model are adjusted based on the fine-tuned loss value until the convergence condition is met to obtain the target image denoising model. Among them, the convergence condition can be that the fine-tuned loss value reaches a preset loss threshold, or the difference between the loss values ​​of two adjacent iterations reaches a preset difference threshold. In practical applications, the Stable Diffusion model can be fine-tuned using LoRA technology to obtain the target image denoising model.

[0131] By fine-tuning the pre-trained image denoising model based on text prompts and perspective images of various preset perspectives to obtain a target image denoising model, the generation target of the target image denoising model can match the text prompts and multiple preset perspectives, thereby achieving personalization of the target image denoising model.

[0132] In step S207, an initialized three-dimensional model is generated based on the volume density field.

[0133] Specifically, a first model representation of the target three-dimensional model to be generated can be generated based on the volume density field, and then the first model representation can be converted into a second model representation that can be adjusted. The second model representation that can be adjusted is referred to as an initialized three-dimensional model in the embodiment of the present disclosure. Exemplarily, the above-mentioned first model representation can be a three-dimensional mesh (Mesh), which can represent the rough geometric shape of the model and can be used to initialize the three-dimensional 3D Gaussian points on the surface of the object. The above-mentioned initialized three-dimensional model can be an initialized rough 3D Gaussian.

[0134] There is no limitation on the execution order of step S205 and step S207, and the two can also be performed in parallel and synchronously.

[0135] In step S209, noise is added to the rendered image of the initialized three-dimensional model to obtain a noisy rendered image, the text prompt and the noisy rendered image are input into the target image denoising model for noise prediction, and the model parameters of the initialized three-dimensional model are adjusted based on the result of the noise prediction until the end condition is met to obtain the target three-dimensional model.

[0136] Specifically, the initialized 3D model can be rendered at a random perspective to obtain a rendered two-dimensional image, which can be called a rendered image. Random noise is then added to the rendered image to obtain a noisy rendered image. The noisy rendered image and the text prompt are input into the target image denoising model obtained by the aforementioned fine-tuning for noise prediction. The initialized 3D model is then trained based on the result of the noise prediction, that is, the model parameters of the initialized 3D model are adjusted to obtain an adjusted 3D model. The adjusted 3D model is then rendered at a random perspective to obtain a rendered image. The rendered image is randomly noised to obtain a noisy rendered image. The noisy rendered image and the text prompt are input into the target image denoising model obtained by the aforementioned fine-tuning for noise prediction. The model parameters of the initialized 3D model are adjusted based on the result of the noise prediction to obtain an adjusted 3D model. This process is repeated until the adjustment is terminated when the end condition is met. The adjusted 3D model at the end of the adjustment is used as the target 3D model.

[0137] The adjustment end condition is associated with the optimization target. For example, the optimization target may be to minimize the difference between the predicted noise and the added noise. The Score Distillation Sampling loss may be used to provide a gradient to update the model parameters of the initialized three-dimensional model.

[0138] The technical solution of the disclosed embodiment unifies the generation direction of the initialization 3D model and the target image denoising model, achieves better collaboration, improves the convergence speed and quality of the 3D model, and better matches the target 3D model with the text prompt.

[0139] In some exemplary embodiments, in order to avoid the problem of lack of authenticity caused by oversaturation of the color of the target 3D model, Figure 4 As shown, the above step S209 may include:

[0140] In step S401, the text prompt and the denoised rendered image are input into the target image denoising model, and the denoised rendered image is subjected to a first denoising process and a second denoising process respectively to obtain a first predicted noise in the first denoising process and a second predicted noise in the second denoising process.

[0141] The first denoising process is performed under the guidance of the text prompt, and the second denoising process is performed without the guidance of the text prompt.

[0142] In step S403, a first loss term is determined based on a difference between the first predicted noise and the second predicted noise; and a second loss term is determined based on the second predicted noise.

[0143] In step S405, based on the first loss term and the second loss term, the model parameters of the initialized three-dimensional model are adjusted, and iteration is performed based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0144] Specifically, the first loss term δ CG It can be expressed as the following formula (1):

[0145]

[0146] The second loss term δ SG It can be expressed as the following formula (2):

[0147]

[0148] Among them, x t represents the input noisy image at time step t; y represents the text prompt; Indicates no text; ∈(x t ,t,y) represents the first prediction noise, represents the second prediction noise; σ t represents the noise added at time step t; p t represents the probability density at time step t.

[0149] Specifically, when adjusting the model parameters of the initialized three-dimensional model for iterative training based on the first loss term and the second loss term, the first loss term and the second loss term may be weightedly summed to obtain a total loss, and a gradient may be provided based on the total loss to update the model parameters of the initialized three-dimensional model. The preset end condition may be that the total loss reaches a preset loss threshold, or that the difference between the total losses of two adjacent iterations reaches a preset difference threshold.

[0150] In the above implementation, the first loss term is used to guide the optimized image to display rich details, but with artifacts and oversaturation; the second loss term is used to reduce the artifacts caused by the first loss term and guide the optimized image to develop toward the high-density area of ​​the 2D image distribution. By combining the first loss term and the second loss term to adjust the model parameters of the 3D model to obtain the target 3D model, it is helpful to achieve the target 3D model while maintaining a balanced saturation while displaying rich image details.

[0151] In some exemplary embodiments, considering that combining the first loss term and the second loss term only according to a fixed ratio may cause the final optimization direction to be negative in the optimization direction of one of the terms, thereby making the guiding optimization effect of the second loss term invalid, and ultimately leading to the appearance of oversaturation, reducing the quality stability of the target three-dimensional model. In order to avoid the above problem, improve the generation quality stability of the target three-dimensional model, and ensure that the generated target three-dimensional model maintains a balanced saturation while showing rich details, the above step S405 may include:

[0152] Determine a proportional hyperparameter based on a difference between the second loss term and the first loss term; the proportional hyperparameter is a positive number less than 1;

[0153] Linearly fuse the first loss term and the second loss term based on the proportional hyperparameter to obtain a fusion loss;

[0154] The model parameters of the initialized three-dimensional model are adjusted based on the fusion loss, and iterative training is performed based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0155] Specifically, the proportional hyperparameter α can be calculated by the following formula (3):

[0156]

[0157] Among them, λ is a preset coefficient, which takes a value between 0 and 1 and can be set based on actual experience; δ CG represents the first loss term, δ SG represents the second loss term.

[0158] Then, the fusion loss L obtained by linear fusion can be achieved by the following formula (4):

[0159] L=αλδ CG +(1-α)δ SG (4)

[0160] The fusion loss can be used to determine the gradient provided to the initial three-dimensional model, and then the model parameters of the three-dimensional model in training are adjusted based on the gradient to optimize the initial three-dimensional model to obtain the target three-dimensional model.

[0161] Specifically, the fusion loss can be used to adjust the model parameters of the initial 3D model based on the Multi-Gradient Descent Algorithm (MGDA) to obtain the target 3D model. In the specific implementation, the optimization of the initial 3D model can be regarded as a multi-objective optimization task, that is, the optimization target is δ SG and δ CG , then the multi-gradient descent algorithm of the following formula (5) is used to find the Pareto optimal point of the multi-objective optimization task, that is, no action can improve one goal without harming another goal. If the generation reaches the Pareto optimal point, it will stabilize at this optimal point, and the generated content will show rich details while maintaining a balanced saturation. In this case, the hyperparameters truly control the final inclination towards the two optimization directions, rather than trying to control the balance at each step:

[0162]

[0163] Among them, E represents the expected treatment, and the inverse of the formula is represents the noisy rendered image; θ represents the model parameters of the initial 3D model; w(t) represents the weight of the target image denoising model.

[0164] In the above implementation, the proportional hyperparameter is determined by the difference between the second loss term and the first loss term, and the two are linearly fused based on the proportional hyperparameter to obtain a fusion loss. Based on the fusion loss, the model parameters of the initialized three-dimensional model are adjusted to perform iterative training to obtain the target three-dimensional model, thereby improving the generation quality stability of the target three-dimensional model and ensuring that the generated target three-dimensional model maintains a balanced saturation while displaying rich details. At the same time, based on the fusion loss, the model parameters of the initial three-dimensional model are updated using a multi-gradient descent algorithm, which can greatly improve the generation effect and stability of the target three-dimensional model while maintaining the fastest generation speed, thereby avoiding the problem of color deviation.

[0165] In order to facilitate understanding of the technical solution of the embodiment of the present disclosure, Figure 5 and Figure 6The three-dimensional model generation method of the embodiment of the present disclosure is exemplified.

[0166] like Figure 5 and Figure 6 As shown, a latent plane module is integrated in the latent layer of the multi-view diffusion model, and the related steps of generating a volume density field based on the target latent feature map of each preset view and a view enhancement feature map of each preset view in the aforementioned step S203 of the embodiment of the present disclosure are performed through the latent plane module, so that the rapid reconstruction of the volume density field and the enhancement of the generated image are completed within 40 seconds.

[0167] Specifically, the text prompt "A ceramic teapot with floral patterns" generates a reference image through the Vincent graph model, and the reference image is input into the multi-view diffusion model for multi-view image generation processing after removing the background. In the multi-view image generation process, the target latent feature maps of multiple preset viewpoints are input into the latent plane module, and the volume density field is reconstructed by the latent plane module, and the view enhancement feature map of each preset viewpoint is obtained by volume rendering based on the reconstruction result. Subsequently, the volume density field is reconstructed into a rough mesh, which is used to initialize the 3D Gaussian points on the surface of the object to obtain a rough 3D Gaussian model (i.e., the initialized three-dimensional model). The view enhancement feature map output of each preset viewpoint is returned to the multi-view diffusion model, and a multi-view image is generated through the decoder function of the multi-view diffusion model. Then, the multi-view image is used to fine-tune the pre-trained Vincent graph diffusion model to obtain a personalized Vincent graph diffusion model (i.e., the target image denoising model). Afterwards, the rendered image obtained by rendering the rough 3D Gaussian model and the personalized Vincent graph diffusion model are used to train the initialized three-dimensional model using the aforementioned formula (5) of the embodiment of the present disclosure to obtain the final 3D Gaussian asset, i.e., the target three-dimensional model. The latent plane module is used to unify the generation directions of different pre-trained models, and better coordinate and cooperate with each other, thereby improving the generation speed and quality of the target three-dimensional model. At the same time, based on the above formula (5), the multi-objective optimization algorithm is integrated to greatly improve the generation effect and stability while maintaining the fastest speed.

[0168] Figure 7 is a structural block diagram of a three-dimensional model generation device according to an exemplary embodiment. Figure 7 , the three-dimensional model generating device 700 comprises:

[0169] A first acquisition unit 710 is configured to acquire a text prompt and a reference image corresponding to the text prompt;

[0170] The multi-view image generating unit 720 is configured to perform a multi-view image generating process guided by the reference image and the view information of the plurality of preset view angles, wherein during the multi-view image generating process, a target latent feature map of each preset view angle is extracted, a volume density field and a view enhancement feature map of each preset view angle are generated based on the target latent feature map of each preset view angle, and a view image of each preset view angle is generated based on the view enhancement feature map of each preset view angle;

[0171] The pre-trained model fine-tuning unit 730 is configured to perform model fine-tuning on the pre-trained image denoising model based on the text prompt and the perspective images of each preset perspective to obtain a target image denoising model;

[0172] A three-dimensional model initialization unit 740 is configured to generate an initialized three-dimensional model based on the volume density field;

[0173] The three-dimensional model generation unit 750 is configured to add noise to the rendered image of the initialized three-dimensional model to obtain a noisy rendered image, input the text prompt and the noisy rendered image into the target image denoising model for noise prediction, and adjust the model parameters of the initialized three-dimensional model based on the result of the noise prediction until the end condition is met to obtain the target three-dimensional model.

[0174] In some exemplary embodiments, the multi-view image generating unit 720 includes:

[0175] A three-dimensional space construction unit is configured to execute the construction of the three-dimensional space by using the target latent feature map of each preset perspective as a feature plane of the three-dimensional space;

[0176] The spatial point representation unit is configured to project each spatial point in the three-dimensional space onto each target latent feature map, and determine the three-dimensional feature of each spatial point based on the projection result corresponding to each spatial point on each target latent feature map;

[0177] A spatial point representation enhancement unit is configured to perform a perspective enhancement process on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points;

[0178] A neural radiation field prediction unit is configured to perform neural radiation field prediction based on multi-view three-dimensional features of each spatial point to obtain volume density field and color feature information;

[0179] The volume rendering unit is configured to perform volume rendering at each preset viewing angle based on the volume density field and the color feature information, and obtain a viewing angle enhancement feature map at each preset viewing angle.

[0180] In some exemplary embodiments, the spatial point representation enhancement unit includes:

[0181] A marking unit is configured to execute splicing of preset marks in the three-dimensional features of each of the space points to obtain a marked three-dimensional feature of each of the space points;

[0182] The attention processing unit is configured to perform self-attention processing on the marked three-dimensional features of each of the spatial points to obtain the multi-view three-dimensional features of each of the spatial points.

[0183] In some exemplary embodiments, the neural radiation field prediction unit includes:

[0184] The multi-view global feature extraction unit is configured to perform multi-view 3D features for each of the spatial points, extract features of the position corresponding to the preset mark from the multi-view 3D features, and obtain multi-view global features of the spatial point; and use the multi-view 3D features after extracting the multi-view global features as the view enhancement features of the spatial point;

[0185] A volume density prediction unit is configured to perform volume density prediction based on the multi-view global features of each of the spatial points to obtain a predicted volume density of each of the spatial points; the volume density field includes the predicted volume density of each of the spatial points;

[0186] The color feature information determining unit is configured to perform a viewing angle enhancement feature based on each of the spatial points to obtain the color feature information.

[0187] In some exemplary embodiments, the spatial point representation unit includes:

[0188] The projection interpolation unit is configured to perform, for each of the spatial points, a projection result corresponding to the spatial point on each of the target latent feature maps and a linear interpolation to obtain a feature vector corresponding to each of the feature planes of the spatial point;

[0189] The vector superposition unit is configured to superimpose the feature vector of each feature plane corresponding to the spatial point with the perspective embedding vector corresponding to the corresponding feature plane to obtain the three-dimensional feature of the spatial point; the perspective embedding vector corresponding to each feature plane is obtained based on the perspective information of the preset perspective corresponding to the feature plane.

[0190] In some exemplary embodiments, the three-dimensional model generating unit 750 includes:

[0191] The noise prediction unit is configured to input the text prompt and the noisy rendered image into the target image denoising model, perform a first denoising process and a second denoising process on the noisy rendered image respectively, and obtain a first predicted noise in the first denoising process and a second predicted noise in the second denoising process; the first denoising process is performed under the guidance of the text prompt, and the second denoising process is performed without the guidance of the text prompt;

[0192] A multi-loss term determination unit is configured to determine a first loss term based on a difference between the first predicted noise and the second predicted noise; and determine a second loss term based on the second predicted noise;

[0193] The three-dimensional model parameter adjustment unit is configured to adjust the model parameters of the initialized three-dimensional model based on the first loss term and the second loss term, and iterate based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0194] In some exemplary embodiments, the three-dimensional model parameter adjustment unit is specifically configured to perform: determining a proportional hyperparameter based on the difference between the second loss term and the first loss term; the proportional hyperparameter is a positive number less than 1; linearly fusing the first loss term and the second loss term based on the proportional hyperparameter to obtain a fusion loss; adjusting the model parameters of the initialized three-dimensional model based on the fusion loss, and iterating based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

[0195] In some exemplary embodiments, when the multi-view image generation unit generates the view images corresponding to each preset view based on the view enhancement feature map of each preset view, it is specifically used to: for each of the preset view, superimpose the target latent feature map of the preset view with the view enhancement feature map to obtain the feature map to be decoded of the preset view; and decode the feature map to be decoded of each of the preset view to obtain the view images corresponding to each of the preset view.

[0196] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, the three-dimensional model generation method provided in any of the above embodiments is implemented.

[0198] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 8is a block diagram of an electronic device for executing a three-dimensional model generation method according to an exemplary embodiment. Figure 8 As shown, the server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 810 (the processor 810 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 830 for storing data, and one or more storage media 820 (such as one or more mass storage devices) for storing application programs 823 or data 822. Among them, the memory 830 and the storage medium 820 can be short-term storage or permanent storage. The program stored in the storage medium 820 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 810 can be configured to communicate with the storage medium 820 and execute a series of instruction operations in the storage medium 820 on the server 800. The server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input and output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0199] The input / output interface 840 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 800. In one example, the input / output interface 840 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 840 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0200] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only and does not limit the structure of the above electronic device. Figure 8 More or fewer components as shown, or with Figure 8 Different configurations are shown.

[0201] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 830 including instructions, and the above instructions can be executed by the processor 810 of the device 800 to complete any one of the three-dimensional content generation methods in the embodiments of the present disclosure. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0202] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the three-dimensional model generation method provided in any of the above embodiments is implemented.

[0203] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0204] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional model generation method, characterized in that: include: Acquire a text prompt and a reference image corresponding to the text prompt; Based on the guidance of the reference image and the perspective information of multiple preset perspectives, a multi-perspective image generation process is performed. During the multi-perspective image generation process, a target latent feature map of each preset perspective is extracted, a volume density field and a perspective enhancement feature map of each preset perspective are generated based on the target latent feature map of each preset perspective, and a perspective image of each preset perspective is generated based on the perspective enhancement feature map of each preset perspective; Based on the text prompt and the perspective images of each preset perspective, fine-tuning the pre-trained image denoising model to obtain a target image denoising model; generating an initialized three-dimensional model based on the volume density field; Noise is added to the rendered image of the initialized three-dimensional model to obtain a noisy rendered image, the text prompt and the noisy rendered image are input into the target image denoising model for noise prediction, and model parameters of the initialized three-dimensional model are adjusted based on the result of the noise prediction until an end condition is met to obtain a target three-dimensional model.

2. The three-dimensional model generation method according to claim 1, characterized in that: The generating of the volume density field and the viewing angle enhancement feature map of each preset viewing angle based on the target latent feature map of each preset viewing angle comprises: Using the target latent feature map of each preset perspective as a feature plane of a three-dimensional space to construct the three-dimensional space; Projecting each spatial point in the three-dimensional space onto each of the target latent feature maps, respectively, and determining the three-dimensional feature of each of the spatial points based on the projection result corresponding to each of the spatial points on each of the target latent feature maps; Performing perspective enhancement processing on the three-dimensional feature of each of the spatial points to obtain a multi-perspective three-dimensional feature of each of the spatial points; Based on the multi-view three-dimensional features of each of the spatial points, neural radiation field prediction is performed to obtain volume density field and color feature information; Volume rendering is performed at each of the preset viewing angles based on the volume density field and the color feature information to obtain a viewing angle enhancement feature map at each of the preset viewing angles.

3. The three-dimensional model generation method according to claim 2, characterized in that: The performing perspective enhancement processing on the three-dimensional feature of each of the spatial points to obtain the multi-perspective three-dimensional feature of each of the spatial points comprises: splicing preset marks in the three-dimensional features of each of the spatial points respectively to obtain the marked three-dimensional features of each of the spatial points; The marked three-dimensional features of each of the spatial points are respectively subjected to self-attention processing to obtain a multi-view three-dimensional feature of each of the spatial points.

4. The three-dimensional model generation method according to claim 3, characterized in that: The neural radiation field prediction is performed based on the multi-view three-dimensional features of each of the spatial points to obtain the volume density field and color feature information, including: For each of the multi-view three-dimensional features of the spatial point, extract the features of the position corresponding to the preset mark from the multi-view three-dimensional features to obtain the multi-view global features of the spatial point; and use the multi-view three-dimensional features after extracting the multi-view global features as the view enhancement features of the spatial point; Based on the multi-view global features of each of the spatial points, volume density prediction is performed to obtain the predicted volume density of each of the spatial points; the volume density field includes the predicted volume density of each of the spatial points; The color feature information is obtained based on the viewing angle enhancement feature of each of the spatial points.

5. The three-dimensional model generation method according to any one of claims 2 to 4, characterized in that: The determining of the three-dimensional feature of each of the spatial points based on the projection result corresponding to each of the spatial points on each of the target latent feature maps comprises: For each of the spatial points, based on the projection results and linear interpolation corresponding to the spatial point on each of the target latent feature maps, a feature vector of each of the feature planes corresponding to the spatial point is obtained; The feature vector of each feature plane corresponding to the spatial point is superimposed with the perspective embedding vector corresponding to the corresponding feature plane to obtain the three-dimensional feature of the spatial point; the perspective embedding vector corresponding to each feature plane is obtained based on the perspective information of the preset perspective corresponding to the feature plane.

6. The three-dimensional model generation method according to claim 1, characterized in that: The step of inputting the text prompt and the noisy rendered image into the target image denoising model for noise prediction, adjusting the model parameters of the initialized 3D model based on the result of the noise prediction until an end condition is met, and obtaining the target 3D model comprises: Inputting the text prompt and the denoised rendered image into the target image denoising model, performing a first denoising process and a second denoising process on the denoised rendered image respectively, obtaining a first predicted noise in the first denoising process and a second predicted noise in the second denoising process; the first denoising process is performed under the guidance of the text prompt, and the second denoising process is performed without the guidance of the text prompt; Determine a first loss term based on a difference between the first predicted noise and the second predicted noise; determine a second loss term based on the second predicted noise; Based on the first loss term and the second loss term, the model parameters of the initialized three-dimensional model are adjusted, and iteration is performed based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

7. The three-dimensional model generation method according to claim 6, characterized in that: The step of adjusting the model parameters of the initialized three-dimensional model based on the first loss term and the second loss term, and iterating based on the adjusted model parameters until a preset end condition is satisfied to obtain a target three-dimensional model includes: Determine a proportional hyperparameter based on a difference between the second loss term and the first loss term; the proportional hyperparameter is a positive number less than 1; Linearly fuse the first loss term and the second loss term based on the proportional hyperparameter to obtain a fusion loss; The model parameters of the initialized three-dimensional model are adjusted based on the fusion loss, and iteration is performed based on the adjusted model parameters until a preset end condition is met to obtain a target three-dimensional model.

8. The three-dimensional model generation method according to claim 1, characterized in that: The generating of the viewing angle images corresponding to the preset viewing angles based on the viewing angle enhancement feature maps of the preset viewing angles comprises: For each of the preset viewing angles, superimposing the target latent feature map of the preset viewing angle and the viewing angle enhancement feature map to obtain a feature map to be decoded of the preset viewing angle; The feature images to be decoded of each of the preset viewing angles are decoded to obtain viewing angle images corresponding to each of the preset viewing angles.

9. A three-dimensional model generating device, characterized in that: include: A first acquisition unit is configured to acquire a text prompt and a reference image corresponding to the text prompt; A multi-view image generation unit is configured to perform a multi-view image generation process based on the reference image and the view information of a plurality of preset view angles, wherein during the multi-view image generation process, a target latent feature map of each preset view angle is extracted, a volume density field and a view enhancement feature map of each preset view angle are generated based on the target latent feature map of each preset view angle, and a view image of each preset view angle is generated based on the view enhancement feature map of each preset view angle; A pre-trained model fine-tuning unit is configured to perform model fine-tuning on the pre-trained image denoising model based on the text prompt and the perspective images of each preset perspective to obtain a target image denoising model; A three-dimensional model initialization unit, configured to generate an initialized three-dimensional model based on the volume density field; The three-dimensional model generation unit is configured to add noise to the rendered image of the initialized three-dimensional model to obtain a noisy rendered image, input the text prompt and the noisy rendered image into the target image denoising model for noise prediction, and adjust the model parameters of the initialized three-dimensional model based on the result of the noise prediction until an end condition is met to obtain a target three-dimensional model.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional model generation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the three-dimensional model generating method according to any one of claims 1 to 8.

Citation Information

Cited By

  • Training method of image correction model, image correction method, device and equipment

    CN120047343A

  • Three-dimensional model generation method and device, computer equipment and storage medium

    CN120747356A

  • Three-dimensional content generation method and device based on retrieval enhancement and personalized generation

    CN121392164A

  • Method and apparatus for generating three-dimensional content based on search enhancement and personalization generation

    CN121392164B

  • Model training method, effect picture generation method, device and equipment

    CN122223199A