Three-dimensional object generation method, apparatus, device, storage medium

Abstract

The application provides a three-dimensional object generation method, device, equipment and storage medium, the method comprises the steps of: obtaining a natural language prompt text; generating an initial model through a neural radiance field; determining a first loss function based on the natural language prompt text and the perspective image of the initial model; optimizing the neural radiance field according to the first loss function to obtain a final neural radiance field; and generating a three-dimensional object through the final neural radiance field. The method provided by the application optimizes the neural radiance field based on the natural language prompt text and the perspective image of the initial model, so that the optimization of the neural radiance field is no longer only based on the text description, but the neural radiance field is optimized through the natural language prompt text and the perspective image of the initial model, and the generation quality of the three-dimensional model is improved through the perspective image of the initial model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a three-dimensional object generation method, device, equipment and storage medium. BACKGROUND

[0002] Text-to-3D technology can automatically generate a three-dimensional object corresponding to a given natural language description. One method of text-to-3D is implemented based on a two-dimensional pre-trained text-image matching model. This method uses a pre-trained text-image matching model (such as a CLIP (Contrastive Language-Image Pre-training) model) to directly optimize a three-dimensional object represented by a neural radiance field (NeRF). During optimization, a two-dimensional image is rendered using a differentiable rendering method, and the rendered image is constrained to match the input text by the pre-trained text-image matching model, thereby optimizing the neural radiance field in the direction of the text description, and ultimately generating a three-dimensional object that conforms to the text description.

[0003] This method may cause distortion of the three-dimensional model, unrealistic texture, or inconsistency across views during the process of optimizing the neural radiance field in the direction of the text description due to the complexity of the text description. SUMMARY

[0004] To solve one of the above technical defects, the present application provides a three-dimensional object generation method, device, equipment and storage medium.

[0005] In a first aspect, the present application provides a three-dimensional object generation method, comprising:

[0006] obtaining a natural language prompt text;

[0007] generating an initial model through a neural radiance field;

[0008] determining a first loss function based on the natural language prompt text and a view image of the initial model;

[0009] optimizing the neural radiance field according to the first loss function to obtain a final neural radiance field;

[0010] generating a three-dimensional object through the final neural radiance field.

[0011] Optionally, the view image includes a front view image, a left side view image, a back view image and a right side view image.

[0012] Before determining the first loss function based on the natural language prompt text and the view image of the initial model, the method further comprises:

[0013] Determine the second image corresponding to the natural language prompt text through the text-to-image model, and take the second image as the front view image;

[0014] Generate the left side view image, the back view image and the right side view image corresponding to the second image;

[0015] Determine the first loss function based on the natural language prompt text and the view image of the initial model, including:

[0016] Determine the first image corresponding to the initial model;

[0017] Determine the view angle of the first image;

[0018] Select the processing view image from the view image based on the view angle of the first image;

[0019] Determine the first loss function based on the first image and the processing view image.

[0020] Optionally, determining the first loss function based on the first image and the processing view image includes:

[0021] Determine the image difference based on the image encoding of the first image and the processing view image, wherein the image encoding of the processing view image is consistent with the encoding dimension of the natural language prompt text;

[0022] Determine the first loss function according to the expectation of the derivative of the image difference.

[0023] Optionally, the neural radiance field is optimized by using a second loss function;

[0024] The second loss function is the sum of the first loss function, a third loss function and a fourth loss function;

[0025] The third loss function is determined according to the natural language prompt text;

[0026] The fourth loss function is determined according to the processing view image.

[0027] Optionally, the third loss function is an expected value of a mapping function;

[0028] The mapping function is used to convert a reward model scoring function into a loss function;

[0029] The reward model scoring function is used to score the relationship between the current model generated by the neural radiance field and the natural language prompt text.

[0030] Optionally, the fourth loss function is determined according to the normalized image features of the model generated by the current neural radiance field and the normalized image features of the processing view image.

[0031] Optionally, the initial model is generated through the neural radiance field, including:

[0032] determining a light ray going in any view direction from the current camera optical center position, the light ray taking time as an independent variable;

[0033] generating an initial model by determining color values of each pixel point through the neural radiance field according to the light ray.

[0034] In a second aspect, the present application provides a three-dimensional object generation device, the device comprising:

[0035] an acquisition module configured to acquire a natural language prompt text;

[0036] a first generation module configured to generate an initial model through a neural radiance field;

[0037] a first determination module configured to determine a first loss function based on the natural language prompt text acquired by the acquisition module and a view image of the initial model generated by the first generation module;

[0038] a second determination module configured to optimize the neural radiance field according to the first loss function determined by the first determination module to obtain a final neural radiance field;

[0039] a second generation module configured to generate a three-dimensional object through the final neural radiance field determined by the second determination module.

[0040] In a third aspect, the present application provides an electronic device, comprising:

[0041] a memory;

[0042] a processor; and

[0043] a computer program;

[0044] wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of the first aspect.

[0045] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method of the first aspect.

[0046] This application provides a method, apparatus, device, and storage medium for generating three-dimensional objects. The method includes: acquiring natural language prompt text; generating an initial model using a neural radiation field; determining a first loss function based on the natural language prompt text and a viewpoint image of the initial model; optimizing the neural radiation field according to the first loss function to obtain a final neural radiation field; and generating a three-dimensional object using the final neural radiation field. The method provided in this application optimizes the neural radiation field based on the natural language prompt text and the viewpoint image of the initial model, so that the optimization of the neural radiation field is no longer solely based on text description, but rather optimizes the neural radiation field through the natural language prompt text and the viewpoint image of the initial model, thereby improving the quality of the generated three-dimensional model. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 A flowchart illustrating a method for generating a three-dimensional object provided in an embodiment of this application;

[0049] Figure 2 A schematic diagram illustrating the implementation architecture of a three-dimensional object generation method provided in this application embodiment;

[0050] Figure 3 This is a schematic diagram of the structure of a three-dimensional object generation device provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of another three-dimensional object generation device provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0054] In the process of realizing this application, the inventors discovered that one method of text-generated 3D is based on a two-dimensional pre-trained text-image matching model. In the process of optimizing the neural radiation field in the direction of text description, this method may cause problems such as distortion of the 3D model, unrealistic texture, or inconsistency across viewpoints due to the complexity of the text description.

[0055] To solve the above problems, the embodiment of the present application provides a three-dimensional object generation method, device, equipment and storage medium. The method comprises the following steps: obtaining a natural language prompt text; generating an initial model through a neural radiation field; determining a first loss function based on the natural language prompt text and a perspective image of the initial model; optimizing the neural radiation field according to the first loss function to obtain a final neural radiation field; and generating a three-dimensional object through the final neural radiation field. The method provided by the present application optimizes the neural radiation field based on the natural language prompt text and the perspective image of the initial model, so that the optimization of the neural radiation field is no longer only based on the text description, but the neural radiation field is optimized through the natural language prompt text and the perspective image of the initial model, and the generation quality of the three-dimensional model is improved through the perspective image of the initial model.

[0056] Referring to Figure 1 , the embodiment provides a three-dimensional object generation method, and the implementation process is as follows:

[0057] 101. Obtain a natural language prompt text.

[0058] The natural language prompt text is a description of a three-dimensional object and can be provided by a user.

[0059] 102. Generate an initial model through a neural radiation field.

[0060] This step can first determine a light ray starting from the current camera optical center position to any perspective direction, and then determine the color value of each pixel point according to the light ray to determine the initial model of the generated three-dimensional object.

[0061] The light ray takes time as the independent variable.

[0062] In a specific implementation, the initial model of this step is realized based on NeRF (Neural Radiance Field), that is, NeRF is used to implicitly represent a three-dimensional object.

[0063] NeRF can be formally defined as a mapping function F θ : (p, d)→(c, σ), where p is any point of a three-dimensional model, d is any perspective, c is the color value of any point, and σ is the radiation field intensity of a three-dimensional object under the perspective d. NeRF takes each 3D point p and perspective d as input, and predicts the radiation field intensity σ and RGB color value c of the 3D point under this perspective through a neural network F θ Based on the radiation field, the initial model can be synthesized through volume rendering.

[0064] Therefore, when determining the light ray starting from the current camera optical center position to any view direction, if the current camera optical center position is o, a light ray r(k) = o + kd can be obtained according to the optical center o and the view direction d, where r is the light ray, and k is the time.

[0065] When determining the color value of each pixel point according to the light ray, and determining the initial model of the three-dimensional object, the color value of the pixel point rendered by the light ray can be determined according to the volume rendering principle, which is emitted by the optical center o along the view direction d. Further based on the color value The two-dimensional image under any view angle can be rendered from the neural radiance field by volume rendering, and finally the initial model of the three-dimensional object is generated. Subsequently, the parameters of the neural radiance field are optimized by step 104 based on the first loss function obtained in step 103, and the final neural radiance field is obtained. Further, in step 105, the three-dimensional object is generated by the final neural radiance field.

[0066] Wherein, k n is the starting point of the light ray, k f is the end point of the light ray, and W(k) is the cumulative transmittance of the light ray.

[0067]

[0068] 103, determine a first loss function based on the natural language prompt text and the view angle image of the initial model.

[0069] Wherein, the view angle image includes: front view angle image, left side view angle image, back view angle image, right side view angle image.

[0070] The determination process of the view angle image is: 1) determining the second image corresponding to the natural language prompt text by the text-to-image model, and taking the second image as the front view angle image. 2) generating the left side view angle image, the back view angle image and the right side view angle image corresponding to the second image.

[0071] Wherein, the text-to-image model can be pre-trained, for example, a training text-to-image model (Stable Diffusion) is trained, which generates the corresponding image according to the input natural language prompt text.

[0072] For example, for the generated visual prompt image, a pre-trained view conversion model Zero-1-to-3 is used to generate images of other views corresponding thereto. Through the text-to-image model, a second image corresponding to the natural language prompt text is determined, and the second image is forced to be a front view image by adding the prefix "the front view of" in the natural language prompt text, and then the second image (i.e. the visual prompt image under the front view) is sent to the visual conversion model Zero-1-to-3 to generate images of its left side, back and right side views, thereby obtaining view images (a set of view-related visual prompt images) including front view images, left side view images, back view images and right side view images.

[0073] The implementation process of step 103 can be implemented based on the distillation sampling technology, and the implementation process of step 103 is as follows:

[0074] 103-1, determining a first image corresponding to the initial model.

[0075] After obtaining the initial model in step 102, an image under any view is rendered from the NeRF through volume rendering. In order to distinguish from other images, the image according to the initial model is recorded as the first image here. For convenience of description, the first image is simply denoted as.

[0076] 103-2, determining the view of the first image.

[0077] 103-3, selecting a processing view image from the view images based on the view of the first image.

[0078] Since the view images include front view images, left side view images, back view images and right side view images, the same view image as the first image will be selected as the processing view image in step 103-3.

[0079] For example, if the view of the first image is the front view (if the view of the first image falls within the range of the front view, the view of the first image is considered to be the front view), then the processing view image is the front view image. If the view of the first image is the left side view (if the view of the first image falls within the range of the left side view, the view of the first image is considered to be the left side view), then the processing view image is the left side view image.

[0080] 103-4, determining a first loss function based on the first image and the processing view image.

[0081] Specifically, the implementation process of step 103-4 is as follows:

[0082] 103-4-1, determining an image difference based on the image encoding of the first image and the processing view image.

[0083] wherein the image encoding of the processed view image is consistent with the natural language prompt text encoding dimension.

[0084] In execution, step 103-4-1 can be implemented by the following process:

[0085] 1. Determine the third image ∈ φ (x k ; k, y, v') according to the image encoding of the processed view image.

[0086] In executing this step, the processed view image v selected in step 103-3 can be obtained first, then converted into an image encoding v' consistent with the text encoding dimension through the image encoder of the text-image matching model (such as the CLIP (Contrastive Language-Image Pre-training) model), and finally the image is obtained through the related parameters φ of the text-to-image model. In order to distinguish from other images, the image obtained through the related parameters φ is denoted as the third image. For convenience of description, the third image is denoted as ∈ φ (x k ; k, y, v').

[0087] wherein k is time, x k is the independent variable, and x k may be an image obtained by rendering through the differentiable volume renderer g θ (π). θ is the parameter of NeRF. That is, the differentiable volume renderer g θ (π) performs random initialization on the initial function constructed based on NeRF with the parameter θ, performs rendering operation on NeRF, and renders the light ray under the camera view angle π pixel by pixel.

[0088] φ is the parameter of the text-to-image model, which is the text-to-image model used when generating the second image. At the same time, when training the text-to-image model, the weight w(k) is also obtained, which takes time as the independent variable.

[0089] y is the natural language prompt text, and v' is the image encoding of the processed view image.

[0090] 2. Determine the image difference ∈ φ (x k ; k, y, v') between the third image ∈ φ (x k ; k, y, v') and the first image ∈ according to the third image ∈ φ (x k ; k, y, v').

[0091] 103-4-2, determine the first loss function according to the expectation of the derivative of the image difference.

[0092] In step 103-4-1, the image difference (i.e., the difference between the third image and the first image) is calculated, and the initial loss function is determined according to the expectation of the derivative of the image difference.

[0093] wherein the image difference is ∈ φ (x k ; k, y, v') - ∈, and the weight w(k) is obtained when training the text-to-image model (Stable Diffusion), so that the first loss function L φ (x k ; k, y, I s ) can be obtained based on the derivative of ∈ VP-SDS , i.e.

[0094] wherein E k，∈ [] is the expectation function of k and ∈.

[0095] Since the third image is obtained from the second image, the first loss function actually provides the optimization direction of the neural radiance field (i.e., the gradient direction of the optimization of the neural radiance field is ) by comparing the differences between the first image and the second image with the same viewing angle. The first image is obtained according to the initial model, and the second image is obtained according to the natural language prompt text, so that the optimization direction provided by the first loss function can ensure that the image rendered by the optimized neural radiance field is consistent with the content of the input natural language prompt text, so that the three-dimensional object represented by the neural radiance field conforms to the content described in the input natural language prompt text.

[0096] In addition, when calculating the first loss function, the difference between the third image and the first image is considered, and the third image is based on the natural language prompt text y and the image encoding v' of the processed viewing angle image v, and then the processed viewing angle image is introduced as an additional condition into the first loss function, so that in the process of optimizing the neural radiance field by the first loss function, the image rendered by the guided neural radiance field is also aligned with the processed viewing angle image (as a visual prompt image), further narrowing the high probability density area of the rendered view of the neural radiance field in the text-to-image model, and achieving the optimization of the neural radiance field guided by the processed viewing angle image as a visual prompt, so that a three-dimensional object that is closely aligned with the natural language prompt text y and the processed viewing angle image v (i.e., the visual prompt) at the same time can be finally obtained.

[0097] In addition, the processing view image v has the same view angle as the first image, so that the processing view image v has a view angle attribute, and the view angle attribute is also transmitted to the neural radiance field as a prompt attribute through the prompting effect of the processing view image v on the neural radiance field, so as to ensure that the generated three-dimensional object is consistent in geometry across the view angle, and the problem of "multi-face (or multi-view geometry inconsistency)" in the prior art (i.e., the three-dimensional object sees different and unmatched images at different angles) is avoided.

[0098] 104, optimizing the neural radiance field according to the first loss function to obtain a final neural radiance field.

[0099] The loss function considered when optimizing the neural radiance field is a second loss function, which is obtained based on the first loss function.

[0100] That is, the second loss function = the first loss function + the third loss function + the fourth loss function.

[0101] The second loss function is denoted as L total , and the first loss function is L vp-SDS obtained in step 103.

[0102] The third loss function is determined according to the natural language prompt text, and the third loss function is denoted as L hf-reward . The third loss function is the expected value of the mapping function. The mapping function is used to convert the reward model score function into a loss function. The reward model score function is used to score the relationship between the current neural radiance field generated model and the natural language prompt text. For example, it can be obtained based on human feedback reward, such as using an image reward model r to score the generated image, and then the reward model can be used to optimize the neural radiance field in the direction of high score, thereby improving the performance of the text-to-image model of the neural radiance field. Specifically, L hf-reward = E z [M(r(z, y))]; where z is the current optimized neural radiance field, y is the natural language prompt text, r() is the reward model score function, M is the mapping function that converts r() into a loss function, and E z [] is the expected function of z.

[0103] The fourth loss function is determined according to the processing view image, and the fourth loss function is denoted as L vc-reward . The fourth loss function is determined according to the standardized image features of the current neural radiance field generated model and the standardized image features of the processing view image. For example, the standardized image features F dino (x) and F dinoL vc-reward Consistent semantic-level representations between different viewpoints can be captured. Specifically, L vc-reward- is represented by minimizing the difference between the estimated rendered image and the normalized image features of the visual cues: L vc-reward =||F dino (x)-F dino (v)| 2

[0104] where F dino (z) is the normalized image features of the image rendered by the current optimized neural radiance field, and F dino (v) is the normalized image features of the processed viewpoint image.

[0105] In optimizing the neural radiance field, L total = L vp-sDs + L hf-reward + L vc-reward is achieved.

[0106] 105, a three-dimensional object is generated by the final neural radiance field.

[0107] After optimizing the neural radiance field through the above steps, the final neural radiance field is obtained, and at this time a three-dimensional object can be generated by the final neural radiance field.

[0108] The three-dimensional object generation method provided by the embodiment is a visual cue guided text-to-three-dimensional method, which uses additional visual cues to strengthen the loss function in three-dimensional object generation, effectively alleviating the problem of poor generation results of the three-dimensional object generation method when facing complex natural language prompt text. In addition, the loss function of the three-dimensional object generation method provided by the embodiment introduces a viewpoint by processing a viewpoint image, avoiding the occurrence of "multiple faces", and in addition to considering the difference between the natural language prompt text and the processed viewpoint image, the loss function also considers the difference between the reward model score, the model generated by the neural radiance field and the processed viewpoint image, which can further improve the three-dimensional object generation quality of the three-dimensional object generation method.

[0109] In specific implementation, the three-dimensional object generation method provided by the embodiment can be implemented based on the architecture shown in Figure 2 , which mainly includes a neural radiance field, a viewpoint-related visual cue guided score distillation sampling module, and a reward feedback learning module. The viewpoint-related visual cue guided score distillation sampling module is used to execute steps 101 to 103 based on the neural radiance field, and the reward feedback learning module is used to optimize the neural radiance field in step 104. The final neural radiance field is used for three-dimensional object generation, so that the generated three-dimensional model conforms to the attributes and characteristics described in the natural language prompt text.

[0110] The embodiment provides a three-dimensional object generation method, acquires natural language prompt text; generates an initial model through a neural radiance field; determines a first loss function based on the natural language prompt text and perspective images of the initial model; optimizes the neural radiance field according to the first loss function to obtain a final neural radiance field; and generates a three-dimensional object through the final neural radiance field. The method provided in the embodiment optimizes the neural radiance field based on the natural language prompt text and the perspective images of the initial model, so that the optimization of the neural radiance field is no longer only based on the text description, but the neural radiance field is optimized through the natural language prompt text and the perspective images of the initial model, and the generation quality of the three-dimensional model is improved through the perspective images of the initial model.

[0111] Based on the same inventive concept of the three-dimensional object generation method, the embodiment provides a three-dimensional object generation device, referring to Figure 3 The device comprises:

[0112] The acquisition module 301 is configured to acquire natural language prompt text.

[0113] The first generation module 302 is configured to generate an initial model through a neural radiance field.

[0114] The first determination module 303 is configured to determine a first loss function based on the natural language prompt text acquired by the acquisition module 301 and perspective images of the initial model generated by the first generation module 302.

[0115] The second determination module 304 is configured to optimize the neural radiance field according to the first loss function determined by the first determination module 303 to obtain a final neural radiance field.

[0116] The second generation module 305 is configured to generate a three-dimensional object through the final neural radiance field determined by the second determination module 304.

[0117] The perspective images comprise a front perspective image, a left side perspective image, a back perspective image and a right side perspective image.

[0118] Referring to Figure 4 The device further comprises a third generation module 306 configured to determine a second image corresponding to the natural language prompt text acquired by the acquisition module 301 through a text-to-image model, and take the second image as a front perspective image. The third generation module 306 is further configured to generate a left side perspective image, a back perspective image and a right side perspective image corresponding to the second image.

[0119] The first determination module 303 is configured to determine a first image corresponding to the initial model generated by the first generation module 302, determine a view angle of the first image, select a processing view angle image from the view angle images generated by the third generation module 306 based on the view angle of the first image, and determine the first loss function based on the first image and the processing view angle image.

[0120] The first determination module 303 is configured to determine an image difference based on image encoding of the first image and the processing view angle image, wherein the image encoding of the processing view angle image is consistent with the natural language prompt text encoding dimension, and determine the first loss function according to an expectation of a derivative of the image difference.

[0121] The second loss function is used to optimize the neural radiance field.

[0122] The second loss function is a sum of the first loss function, the third loss function, and the fourth loss function.

[0123] The third loss function is determined according to the natural language prompt text.

[0124] The fourth loss function is determined according to the processing view angle image.

[0125] The third loss function is an expectation value of a mapping function.

[0126] The mapping function is used to convert a reward model score function into a loss function.

[0127] The reward model score function is used to score a relationship between a model generated by the current neural radiance field and the natural language prompt text.

[0128] The fourth loss function is determined according to a standardized image feature of the model generated by the current neural radiance field and a standardized image feature of the processing view angle image.

[0129] The first generation module 302 is configured to determine a light ray starting from a current camera optical center position in any view angle direction, the light ray taking time as an independent variable, determine a color value of each pixel point through the neural radiance field based on the light ray, and generate the initial model.

[0130] The device provided in the embodiment is based on the natural language prompt text and the view angle image of the initial model, and optimizes the neural radiance field, so that the optimization of the neural radiance field is no longer only based on the text description, but the neural radiance field is optimized through the natural language prompt text and the view angle image of the initial model, and the generation quality of the three-dimensional model is improved through the view angle image of the initial model.

[0131] Based on the same inventive concept of the three-dimensional object generation method, the embodiment provides an electronic device, which is like Figure 5As shown, it comprises a memory 501, a processor 502, and a computer program.

[0132] The computer program is stored in the memory 501 and is configured to be executed by the processor 502 to implement the three-dimensional object generation method described above.

[0133] Specifically,

[0134] Obtain the natural language prompt text.

[0135] Generate an initial model through a neural radiance field.

[0136] Determine a first loss function based on the natural language prompt text and the perspective images of the initial model.

[0137] Optimize the neural radiance field according to the first loss function to obtain a final neural radiance field.

[0138] Generate a three-dimensional object through the final neural radiance field.

[0139] Optionally, the perspective images include a front perspective image, a left side perspective image, a back perspective image, and a right side perspective image.

[0140] Before determining the first loss function based on the natural language prompt text and the perspective images of the initial model, the method further comprises:

[0141] Determine a second image corresponding to the natural language prompt text through a text-to-image model, and take the second image as the front perspective image.

[0142] Generate a left side perspective image, a back perspective image, and a right side perspective image corresponding to the second image.

[0143] Determine the first loss function based on the natural language prompt text and the perspective images of the initial model, comprising:

[0144] Determine a first image corresponding to the initial model.

[0145] Determine the perspective of the first image.

[0146] Select a processing perspective image from the perspective images based on the perspective of the first image.

[0147] Determine the first loss function based on the first image and the processing perspective image.

[0148] Optionally, determining the first loss function based on the first image and the processing perspective image comprises:

[0149] Determine an image difference based on the image encoding of the first image and the processing perspective image, wherein the image encoding of the processing perspective image is consistent with the dimension of the natural language prompt text encoding.

[0150] The first loss function is determined according to the derivative of the image difference.

[0151] Optionally, the neural radiance field is optimized by using the second loss function.

[0152] The second loss function is a sum of the first loss function, the third loss function and the fourth loss function.

[0153] The third loss function is determined according to the natural language prompt text.

[0154] The fourth loss function is determined according to the processed view image.

[0155] Optionally, the third loss function is an expected value of a mapping function.

[0156] The mapping function is used to convert a reward model score function into a loss function.

[0157] The reward model score function is used to score the relationship between the current model generated by the neural radiance field and the natural language prompt text.

[0158] Optionally, the fourth loss function is determined according to the normalized image features of the current model generated by the neural radiance field and the normalized image features of the processed view image.

[0159] Optionally, the initial model is generated by the neural radiance field, including:

[0160] A light ray is determined from the current camera optical center position to any view direction, and the light ray takes time as the independent variable.

[0161] According to the light ray, the color value of each pixel point is determined by the neural radiance field to generate the initial model.

[0162] The electronic device provided in the embodiment is used to execute the computer program by the processor to optimize the neural radiance field based on the natural language prompt text and the view image of the initial model, so that the optimization of the neural radiance field is no longer only based on the text description, but the neural radiance field is optimized by the natural language prompt text and the view image of the initial model, and the generation quality of the three-dimensional model is improved by the view image of the initial model.

[0163] Based on the same inventive concept of the three-dimensional object generation method, the embodiment provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium. The computer program is executed by the processor to implement the three-dimensional object generation method.

[0164] Specifically,

[0165] The natural language prompt text is obtained.

[0166] The initial model is generated by the neural radiance field.

[0167] determine a first loss function based on the natural language prompt text and the perspective images of the initial model.

[0168] optimize the neural radiance field according to the first loss function to obtain a final neural radiance field.

[0169] generate the three-dimensional object through the final neural radiance field.

[0170] Optionally, the perspective images include: a front perspective image, a left side perspective image, a back perspective image, and a right side perspective image.

[0171] Before determining the first loss function based on the natural language prompt text and the perspective images of the initial model, the method further includes:

[0172] determine a second image corresponding to the natural language prompt text through the text-to-image model, and take the second image as the front perspective image.

[0173] generate a left side perspective image, a back perspective image, and a right side perspective image corresponding to the second image.

[0174] Determining the first loss function based on the natural language prompt text and the perspective images of the initial model includes:

[0175] determine a first image corresponding to the initial model.

[0176] determine a perspective of the first image.

[0177] select a processing perspective image from the perspective images based on the perspective of the first image.

[0178] determine the first loss function based on the first image and the processing perspective image.

[0179] Optionally, determining the first loss function based on the first image and the processing perspective image includes:

[0180] determine an image difference based on image encoding of the first image and the processing perspective image, wherein the image encoding of the processing perspective image is consistent with a dimension of encoding of the natural language prompt text.

[0181] determine the first loss function according to an expectation of a derivative of the image difference.

[0182] Optionally, the neural radiance field is optimized by using a second loss function.

[0183] wherein the second loss function is a sum of the first loss function, a third loss function, and a fourth loss function.

[0184] The third loss function is determined according to the natural language prompt text.

[0185] The fourth loss function is determined according to the processed view image.

[0186] Optionally, the third loss function is an expected value of the mapping function.

[0187] The mapping function is used to convert the reward model score function into the loss function.

[0188] The reward model score function is used to score the relationship between the current neural radiance field generated model and the natural language prompt text.

[0189] Optionally, the fourth loss function is determined according to the normalized image features of the current neural radiance field generated model and the normalized image features of the processed view image.

[0190] Optionally, the initial model is generated by the neural radiance field, including:

[0191] The light ray is determined from the current camera optical center position to any view direction, and the light ray takes time as the independent variable.

[0192] According to the light ray, the color value of each pixel point is determined by the neural radiance field to generate the initial model.

[0193] The computer readable storage medium provided by the embodiment, the computer program on the computer readable storage medium is executed by the processor to optimize the neural radiance field based on the natural language prompt text and the view image of the initial model, so that the optimization of the neural radiance field is no longer only based on the text description, but the neural radiance field is optimized by the natural language prompt text and the view image of the initial model, and the generation quality of the three-dimensional model is improved by the view image of the initial model.

[0194] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming language Java and interpreted scripting language JavaScript.

[0195] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0196] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0197] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0198] In addition, the terms "first", "second", etc., are used herein only to describe different instances, and do not imply or suggest relative importance or a number of indicated technical features.

[0199] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to embrace all such variations and modifications as fall within the scope of the application.

[0200] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of three-dimensional object generation, characterized by, The method comprises: acquiring natural language prompt text; generating an initial model through a neural radiance field; determining a first loss function based on the natural language prompt text and perspective images of the initial model; optimizing the neural radiance field according to the first loss function to obtain a final neural radiance field; generating a three-dimensional object through the final neural radiance field; the perspective images comprise front perspective images, left side perspective images, back perspective images and right side perspective images; before the determining of the first loss function based on the natural language prompt text and the perspective images of the initial model, the method further comprises: determining a second image corresponding to the natural language prompt text through a text-to-image model, and taking the second image as a front perspective image; generating left side perspective images, back perspective images and right side perspective images corresponding to the second image; the determining of the first loss function based on the natural language prompt text and the perspective images of the initial model comprises: determining a first image corresponding to the initial model; determining a perspective of the first image; selecting a processing perspective image from the perspective images based on the perspective of the first image; determining a first loss function based on the first image and the processing perspective image.

2. The method of claim 1, wherein, the determining of the first loss function based on the first image and the processing perspective image comprises: determining an image difference based on image encodings of the first image and the processing perspective image, wherein the image encoding of the processing perspective image is consistent with a natural language prompt text encoding dimension; determining the first loss function according to an expectation of a derivative of the image difference.

3. The method of claim 1, wherein, optimizing the neural radiance field by using a second loss function; wherein the second loss function is a sum of the first loss function, a third loss function and a fourth loss function; the third loss function is determined according to the natural language prompt text; the fourth loss function is determined according to the processing perspective image.

4. The method of claim 3, wherein, the third loss function is an expectation value of a mapping function; wherein the mapping function is used to convert a reward model score function into a loss function; the reward model score function is used to score a relationship between a model generated by a current neural radiance field and the natural language prompt text.

5. The method of claim 3, wherein, the fourth loss function is determined according to a standardized image feature of a model generated by the current neural radiance field and a standardized image feature of the processing perspective image.

6. The method of claim 1, wherein, the generating of the initial model through the neural radiance field comprises: determining a light ray starting from a current camera optical center position to any perspective direction, the light ray taking time as an independent variable; determining color values of each pixel point through the neural radiance field according to the light ray to generate the initial model.

7. A three-dimensional object generation apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire natural language prompt text; a first generation module configured to generate an initial model through a neural radiance field; a first determination module configured to determine a first loss function based on natural language prompt text acquired by the acquisition module and perspective images of the initial model generated by the first generation module; a second determination module configured to optimize the neural radiance field according to the first loss function determined by the first determination module to obtain a final neural radiance field; A second generation module is configured to generate a three-dimensional object based on the final neural radiance field determined by the second determination module. The perspective images include: a front perspective image, a left side perspective image, a back perspective image, and a right side perspective image. The device further includes a third generation module configured to determine a second image corresponding to the natural language prompt text obtained by the acquisition module based on a text-to-image model, and determine the front perspective image, the left side perspective image, the back perspective image, and the right side perspective image corresponding to the second image. The first determination module is configured to determine a first image corresponding to the initial model generated by the first generation module, determine a perspective of the first image, select a processing perspective image from the perspective images generated by the third generation module based on the perspective of the first image, and determine a first loss function based on the first image and the processing perspective image.

8. An electronic device, comprising: comprise: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method of any one of claims 1-6.

Images