Image generation method, apparatus, device, and storage medium

Abstract

The application provides an image generation method, device, equipment and storage medium. The method comprises the following steps: obtaining a text prompt; decomposing the text prompt into a foreground prompt and a background prompt; generating a dynamic four-dimensional foreground guided by the foreground prompt; generating a static three-dimensional background guided by the background prompt based on the dynamic four-dimensional foreground; and rendering a video along a given camera track based on the dynamic four-dimensional foreground and the static three-dimensional background. The method provided by the application generates a dynamic four-dimensional foreground and a static three-dimensional background based on a text prompt, and then renders a video based on the dynamic four-dimensional foreground and the static three-dimensional background, so that the video is more consistent with the physical laws in the real world, and precise camera motion control can be achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method, apparatus, device, and storage medium. Background Technology

[0002] Natural language video generation is a fundamental topic in computer vision and multimedia. Given a sentence of natural language description as input, this technique aims to automatically generate a video corresponding to the input text.

[0003] Existing text-to-video methods typically model the video as a set of time-varying two-dimensional images (i.e., video frames), and then learn the probability distribution transformation from text to video from a large amount of text-video data pairs.

[0004] In existing text-to-video technology, the video is simply modeled as a two-dimensional image that changes over time, and then the distribution transformation between text and video is learned through supervised learning. This results in problems such as object distortion, motion distortion, and inconsistencies between different viewpoints. Summary of the Invention

[0005] To address one of the aforementioned technical deficiencies, this application provides an image generation method, apparatus, device, and storage medium.

[0006] A first aspect of this application provides an image generation method, the method comprising:

[0007] Get text hints;

[0008] Break down text prompts into foreground and background prompts;

[0009] Generate dynamic four-dimensional foreground using foreground cues;

[0010] Based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated.

[0011] Render video along a given camera trajectory based on a dynamic 4D foreground and a static 3D background.

[0012] Optionally, guided by foreground cues, a dynamic four-dimensional foreground is generated, including:

[0013] Foreground hints To guide this process, a dynamic four-dimensional foreground model was obtained using Neural Radiation Field (NeRF). and coherent multi-view images ,in, For any point in the dynamic four-dimensional foreground model, For any perspective, For time variables, The mapping relationship between color and radiation field intensity is as follows: , For any point, For dynamic four-dimensional foreground models in view The intensity of the radiation field under;

[0014] based on right Rendering is performed, and in time steps. Add Gaussian noise To obtain the corresponding image ;

[0015] predict The noise is calculated, and optimization is performed based on the predicted noise. The loss function used in the optimization is a weighted sum of the loss function for optimizing the 3D structure, the loss function for optimizing the visual appearance, and the loss function for optimizing the motion features.

[0016] Based on the optimized Generate a dynamic four-dimensional foreground.

[0017] Optionally, optimize the loss function of the three-dimensional structure. ;

[0018] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For a multi-view diffusion model, To pass predict noise, For the parameters of the multi-view diffusion model, These are the camera parameters corresponding to each view.

[0019] Optionally, the loss function for optimizing the visual appearance is: ;

[0020] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise, For the parameters of the text-to-image diffusion model, These are the camera parameters corresponding to each view.

[0021] Optionally, by time step Add Gaussian noise Simultaneously, the video sequence of that step size is rendered using NeRF. ,in, This represents the total number of video frames in the video sequence.

[0022] The loss function for optimizing motion features is: ;

[0023] in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, exist Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

[0024] Optionally, based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated, including:

[0025] Obtain the rendered image and opacity map of the dynamic four-dimensional foreground;

[0026] Using background cues as guidance, the background area of ​​the rendered image is identified through an opacity map;

[0027] By repairing the model, the background area is repaired;

[0028] A static 3D background is generated based on the repaired background area;

[0029] The loss function used in the process of generating the static 3D background is: ;

[0030] in, The loss function used in the process of generating a static 3D background. For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For conditional diffusion models, To pass predict noise, For the parameters of the conditional diffusion model, For the background area to be repaired, and For camera orientation.

[0031] Optionally, the camera trajectory is determined by the elevation angle. Azimuth and distance In time composition.

[0032] A second aspect of this application provides an image generation method apparatus, the apparatus comprising:

[0033] The acquisition module is used to acquire text prompts;

[0034] The decomposition module is used to decompose the text prompts obtained by the acquisition module into foreground prompts and background prompts;

[0035] The foreground generation module is used to generate a dynamic four-dimensional foreground guided by the foreground cues obtained from the decomposition module.

[0036] The background generation module is used to generate a static 3D background based on the dynamic 4D foreground generated by the foreground generation module and guided by the background cues obtained from the decomposition module.

[0037] The rendering module is used to render video along a given camera trajectory, based on the dynamic four-dimensional foreground generated by the foreground generation module and the static three-dimensional background generated by the background generation module.

[0038] A third aspect of this application provides an electronic device, comprising:

[0039] Memory;

[0040] Processor; and

[0041] Computer programs;

[0042] The computer program is stored in the memory and configured to be executed by the processor to implement the method described in the first aspect above.

[0043] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described in the first aspect above.

[0044] This application provides an image generation method, apparatus, device, and storage medium. The method includes: acquiring text prompts; decomposing the text prompts into foreground prompts and background prompts; generating a dynamic four-dimensional foreground guided by the foreground prompts; generating a static three-dimensional background based on the dynamic four-dimensional foreground and guided by the background prompts; and rendering a video along a given camera trajectory based on the dynamic four-dimensional foreground and the static three-dimensional background. The method provided by this application generates a dynamic four-dimensional foreground and a static three-dimensional background based on text prompts, and then renders a video based on the dynamic four-dimensional foreground and the static three-dimensional background. This makes the video more consistent with the physical laws of the real world and also enables precise camera motion control. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 A flowchart illustrating an image generation method provided in an embodiment of this application;

[0047] Figure 2 A schematic diagram illustrating the implementation architecture of an image generation method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0051] In the process of developing this application, the inventors discovered that in existing text-based video technology, the video is simply modeled as a two-dimensional image that changes over time, and then the distribution transformation between text and video is learned through supervised learning. This results in problems such as object distortion, motion distortion, and inconsistencies between different perspectives.

[0052] To address the aforementioned problems, this application provides an image generation method, apparatus, device, and storage medium. The method includes: acquiring text prompts; decomposing the text prompts into foreground prompts and background prompts; generating a dynamic four-dimensional foreground guided by the foreground prompts; generating a static three-dimensional background based on the dynamic four-dimensional foreground and guided by the background prompts; and rendering a video along a given camera trajectory based on the dynamic four-dimensional foreground and the static three-dimensional background. The method provided by this application generates a dynamic four-dimensional foreground and a static three-dimensional background based on text prompts, and then renders a video based on the dynamic four-dimensional foreground and the static three-dimensional background. This makes the video more consistent with the physical laws of the real world and also enables precise camera motion control.

[0053] See Figure 1 This embodiment provides an image generation method, which includes:

[0054] 101, Get text hints.

[0055] The text prompt obtained in step 101 is the prompt text for generating the image. The method provided in this embodiment will generate the corresponding image based on the text prompt.

[0056] 102. Decompose the text prompt into foreground prompts and background prompts.

[0057] Step 102 can be implemented based on an existing Large Language Model (LLM). That is, the text prompts obtained in step 101 are processed using a Large Language Model (LLM). Decompose into foreground text prompts and background text prompts .

[0058] For example, if the text prompt obtained in step 101 is "Given you a description of video clip, you need to describe the foreground subject and the background scene in detail", then in step 102, the text prompt is input into the Large Language Model (LLM) so that it can effectively process the input prompt. Divided into foreground text prompts and background text prompts .

[0059] 103. Using foreground cues as guidance, generate dynamic four-dimensional foreground.

[0060] The implementation process of step 103 is as follows:

[0061] 103-1, with foreground hints To guide this process, a dynamic four-dimensional foreground model was obtained using NeRF (Neural Radiance Field). and coherent multi-view images .

[0062] in, For any point in the dynamic four-dimensional foreground model, For any perspective, For time variables, The mapping relationship between color and radiation field intensity is as follows: , For any point, For dynamic four-dimensional foreground models in view The intensity of the radiation field under [the specified conditions].

[0063] It is a 4D dynamic model that describes the visual appearance, density, and motion of foreground objects; therefore, The volume density and color were predicted conditioned on the time exponent.

[0064] Step 103-1 can be implemented using an SDS with a 3D-aware text-to-image diffusion model. This model can detect foreground cues... Generate coherent multi-view images .

[0065] 103-2, based on right Rendering is performed, and in time steps. Add Gaussian noise To obtain the corresponding image .

[0066] This step still uses SDS with a 3D perceptual text-to-image diffusion model, which can be achieved by combining multi-view images. from Rendering to differentiable volumetric rendering, and at time step Add Gaussian noise To obtain its noise counterpart .

[0067] In addition, with time step Add Gaussian noise Simultaneously, the video sequence of that step size will be rendered using NeRF. ,in, This represents the total number of video frames in the video sequence.

[0068] 103-3, Prediction The noise is calculated, and optimization is performed based on the predicted noise. .

[0069] The loss function used in the optimization is a weighted sum of the loss function for optimizing the 3D structure, the loss function for optimizing the visual appearance, and the loss function for optimizing the motion features.

[0070] Right now .

[0071] For example, , and .

[0072] The loss function used during optimization. For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. To optimize the weights of the loss function for the 3D structure, To optimize the loss function of the three-dimensional structure, To optimize the weights of the loss function for visual appearance, To optimize the loss function for visual appearance, To optimize the weights of the loss function for motion features, The loss function is used to optimize motion features.

[0073] 1. Optimize the loss function of the 3D structure

[0074]

[0075] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For a multi-view diffusion model, To pass predict noise, For the parameters of the multi-view diffusion model, These are the camera parameters corresponding to each view.

[0076] pass Optimization of 3D structures can be achieved. The optimization process for 3D structures can still be implemented using SDS (Score Distillation Sampling), which employs a 3D-aware text-to-image diffusion model. Noise can be predicted and through Update 4D animation .

[0077] SDS loss encourages the generation of multi-view images before the 3D perceptual diffusion model. Consistency and alignment facilitate multi-view Figure 1 This improved consistency and the overall quality of the generated images.

[0078] 2. Loss function for optimizing visual appearance

[0079]

[0080] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise (which can be obtained by using LoRA on a rendering image of NeRF in progress). For the parameters of the text-to-image diffusion model, These are the camera parameters corresponding to each view.

[0081] pass It can achieve the optimization of three-dimensional structures. Noise can be predicted and through Update 4D animation .

[0082] Among them, the text-to-image diffusion model It can enhance the visual appearance of foreground objects; in practical implementation, the text-to-image diffusion model... This can be a stable diffusion model.

[0083] In addition, using VSD (Variational Score Distillation) to extract two-dimensional appearance knowledge from the diffusion model can generate results with higher fidelity, making it a more effective option.

[0084] 3. Optimize the loss function for motion features

[0085]

[0086] in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, exist Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

[0087] In step 103-3, optimization of the three-dimensional structure, visual appearance, and motion characteristics was performed.

[0088] 103-4, based on the optimized Generate a dynamic four-dimensional foreground.

[0089] In step 103, a dynamic four-dimensional foreground is generated, guided by foreground cues. First, foreground cues are used... To guide the generation of a dynamic four-dimensional foreground model using NeRF (Neural Radiance Field). and coherent multi-view images .in, It describes the visual appearance, density, and motion of the foreground object.

[0090] In order to It can generate time-varying density and color, reshaping them by incorporating an additional input time variable 𝑡. Furthermore, it optimizes the three fundamental components of foreground objects in a four-dimensional world: 3D structure, visual appearance, and motion characteristics.

[0091] Thus, the optimized Videos can be rendered from any predefined viewpoint, thereby generating dynamic four-dimensional foregrounds.

[0092] 104. Based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated.

[0093] Although optimized Videos can be rendered from any predefined viewpoint, but rendered videos often lack a meaningful background. To address this, step 104 generates a static 3D background based on a dynamic 4D foreground and guided by background cues, thus creating a background that complements the background cues. Matching static 3D background.

[0094] The implementation process of step 104 is as follows:

[0095] 104-1, Obtain the rendered image and opacity map of the dynamic four-dimensional foreground.

[0096] In step 104-1, an image is rendered in the dynamic four-dimensional foreground (i.e., the rendered image), and the optimized image is obtained. The opacity map (i.e., the opacity map).

[0097] Opacity maps can be used as foreground masks to identify foreground areas.

[0098] 104-2, guided by background cues, identifies the background area of ​​the rendered image through an opacity map.

[0099] 104-3, Repair the background area by repairing the model.

[0100] For example, a pre-trained stable diffusion model can be used to repair complex background regions.

[0101] 104-4, Generate a static 3D background based on the repaired background area.

[0102] When generating a static 3D background, it can be based on a static 3D background model conditioned on a reference background image. When an input image (such as a repaired background region) is provided, this static 3D background model can generate a new perspective image of the same object under any specified camera transformation. Alternatively, a conditional diffusion model can also be utilized. The knowledge from different viewpoints is extracted into a static 3D background through the SDS process.

[0103] In practical implementation, this can be achieved by training a conditional diffusion model. This enhances its ability to generate background landscapes.

[0104] The loss function used in the process of generating the static 3D background is: .

[0105] in, The loss function used in the process of generating a static 3D background. For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For conditional diffusion models, To pass predict noise, For the parameters of the conditional diffusion model, For the background area to be repaired, and For camera orientation.

[0106] The static 3D background generated in step 104 can be harmoniously merged with the dynamic 4D foreground. In practice, after obtaining the dynamic 4D foreground and the static 3D background, before executing step 105, the generated 4D foreground model and the static 3D background model are combined into a unified 4D world. In this way, in step 105, the video can be rendered based on the 4D world along the given camera trajectory.

[0107] 105. Render video along a given camera trajectory based on a dynamic 4D foreground and a static 3D background.

[0108] Among them, the camera trajectory is determined by the elevation angle. Azimuth and distance In time Composition. The trajectory can be indicated by a human user or a predefined camera movement pattern.

[0109] The generated 4D foreground model and static 3D background model are combined into a unified 4D world, and a rough video is rendered along a given camera trajectory. Finally, the rough video is further refined using a video diffusion model to achieve higher visual fidelity and consistency.

[0110] In generating the dynamic four-dimensional foreground and static three-dimensional background, step 105 will render a scene based on camera trajectory. The video used as a condition is the video corresponding to the text prompt obtained in step 101, which is a 4D world view. To address the issue of high computational cost and low resolution in NeRF-rendered videos, step 105 employs a coarse-to-fine rendering process.

[0111] First, render a low-resolution video from a 4D world, then upscale it to a higher resolution.

[0112] In low-resolution video rendering, a series of foreground images, background images, and opacity maps are rendered by volumetric rendering of a dynamic 4D foreground and a static 3D background. The opacity map is used to blend the foreground and background images to achieve coarse video frames at low resolution (e.g., 256×256).

[0113] In high-resolution video rendering, to generate high-fidelity rendered videos, this embodiment refactors the widely used SVD (Stable-Video-Diffusion) model. By using coarse video as input, it generates consecutive frames with enhanced resolution (e.g., 1024×1024). The coarse video encoded by VAE (Variational Auto-Encoder) can be concatenated with a noisy latent image, and SVD can be trained to denoise the latent image while preserving the semantic content of the coarse video and adding rich visual details.

[0114] In existing text-to-video technologies, algorithms simply model the video as a two-dimensional image that changes over time, and then learn the distribution transformation between text and video through supervised learning, ignoring the inherent four-dimensional (space-depth-time) priors of the physical world behind the video. This leads to problems such as object distortion, motion distortion, and inconsistencies between different viewpoints in the generated videos.

[0115] The image generation method provided in this embodiment is a text-based video technology based on constructing a four-dimensional world. First, a four-dimensional world containing a dynamic four-dimensional foreground and a static three-dimensional background is constructed based on the input text prompts. Then, a video is rendered from this four-dimensional world. This approach not only makes the generated video more consistent with the physical laws of the real world but also enables precise camera motion control.

[0116] The image generation method provided in this embodiment can be implemented in a specific way by... Figure 2 The architecture shown consists of two stages: (a) four-dimensional world generation and (b) four-dimensional world rendering.

[0117] In the first stage, the input text prompts are first decomposed into foreground prompts and background prompts using a Large Language Model (LLM) (i.e., steps 101 and 102). These two text prompts guide the generation processes of the dynamic four-dimensional foreground and the static three-dimensional background, respectively (i.e., steps 103 and 104).

[0118] In the second stage, the generated four-dimensional foreground model and static three-dimensional background model are combined into a unified four-dimensional world, and a rough video is rendered along a given camera trajectory. Finally, the rough video is further refined using a video diffusion model to achieve higher visual fidelity and consistency (i.e., step 105).

[0119] The image generation method provided in this embodiment can realize text-based videos of the four-dimensional world. By constructing a four-dimensional world model that conforms to the principles of real physics, high-fidelity modeling of the video generation process is achieved. Compared with traditional space-time diffusion models, the image generation method provided in this embodiment can better handle complex spatiotemporal dynamics and camera perspective changes, reducing distortion of objects and motion. In addition, the image generation method provided in this embodiment not only performs well in terms of geometric consistency, but also can generate multi-lens videos with multi-view consistency, and supports video generation with different camera movements and backgrounds.

[0120] This embodiment provides an image generation method that obtains text prompts; decomposes the text prompts into foreground and background prompts; generates a dynamic four-dimensional foreground guided by the foreground prompts; generates a static three-dimensional background based on the dynamic four-dimensional foreground and guided by the background prompts; and renders a video along a given camera trajectory based on the dynamic four-dimensional foreground and the static three-dimensional background. The method provided in this embodiment generates a dynamic four-dimensional foreground and a static three-dimensional background based on the text prompts, and then renders a video based on these, making the video more consistent with the physical laws of the real world and enabling precise camera motion control.

[0121] Based on the same inventive concept of image generation methods, this embodiment provides an image generation method apparatus, see [link to relevant documentation]. Figure 3 The device includes:

[0122] Module 301 is used to obtain text prompts.

[0123] The decomposition module 302 is used to decompose the text prompts acquired by the acquisition module 301 into foreground prompts and background prompts.

[0124] The foreground generation module 303 is used to generate a dynamic four-dimensional foreground guided by the foreground cues obtained from the decomposition module 302.

[0125] Background generation module 304 is used to generate a static three-dimensional background based on the dynamic four-dimensional foreground generated by foreground generation module 303 and guided by the background cues obtained by decomposition module 302.

[0126] The rendering module 305 is used to render the video along a given camera trajectory based on the dynamic four-dimensional foreground generated by the foreground generation module 303 and the static three-dimensional background generated by the background generation module 304.

[0127] Among them, the foreground generation module 303 is used to provide foreground prompts. To guide this process, a dynamic four-dimensional foreground model was obtained using Neural Radiation Field (NeRF). and coherent multi-view images ,in, For any point in the dynamic four-dimensional foreground model, For any perspective, For time variables, The mapping relationship between color and radiation field intensity is as follows: , For any point, For dynamic four-dimensional foreground models in view The intensity of the radiation field under [the following]. Based on [the following]: right Rendering is performed, and in time steps. Add Gaussian noise To obtain the corresponding image .predict The noise is calculated, and optimization is performed based on the predicted noise. The optimization loss function is a weighted sum of the loss functions for optimizing 3D structure, visual appearance, and motion features. Based on the optimized... Generate a dynamic four-dimensional foreground.

[0128] Among them, the loss function for optimizing the three-dimensional structure .

[0129] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For a multi-view diffusion model, To pass predict noise, For the parameters of the multi-view diffusion model, These are the camera parameters corresponding to each view.

[0130] The loss function for optimizing visual appearance is: .

[0131] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise, For the parameters of the text-to-image diffusion model, These are the camera parameters corresponding to each view.

[0132] Among them, time step Add Gaussian noise Simultaneously, the video sequence of that step size is rendered using NeRF. ,in, This represents the total number of video frames in the video sequence.

[0133] The loss function for optimizing motion features is: .

[0134] in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, exist Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

[0135] Background generation module 304 is used to acquire the rendered image and opacity map of the dynamic four-dimensional foreground. Guided by background cues, it identifies the background area of ​​the rendered image through the opacity map. It then repairs the background area using a repair model. Based on the repaired background area, it generates a static three-dimensional background.

[0136] The loss function used in the process of generating the static 3D background is: .

[0137] in, The loss function used in the process of generating a static 3D background. For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For conditional diffusion models, To pass predict noise, For the parameters of the conditional diffusion model, For the background area to be repaired, and For camera orientation.

[0138] Among them, the camera trajectory is determined by the elevation angle. Azimuth and distance In time composition.

[0139] The device provided in this embodiment generates a dynamic four-dimensional foreground and a static three-dimensional background based on text prompts, and then renders a video based on the dynamic four-dimensional foreground and the static three-dimensional background. This can make the video more in line with the physical laws of the real world and can also achieve precise camera motion control.

[0140] Based on the same inventive concept of image generation methods, this embodiment provides an electronic device, which is as follows: Figure 4 As shown, it includes: a memory 401, a processor 402, and a computer program.

[0141] The computer program is stored in memory 401 and configured to be executed by processor 402 to implement the above-described image generation method.

[0142] Specifically,

[0143] Get the text hint.

[0144] Break down text prompts into foreground prompts and background prompts.

[0145] Using foreground cues as guidance, generate dynamic four-dimensional foreground.

[0146] Based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated.

[0147] Render video along a given camera trajectory based on a dynamic 4D foreground and a static 3D background.

[0148] Optionally, guided by foreground cues, a dynamic four-dimensional foreground is generated, including:

[0149] Foreground hints To guide this process, a dynamic four-dimensional foreground model was obtained using Neural Radiation Field (NeRF). and coherent multi-view images ,in, For any point in the dynamic four-dimensional foreground model, For any perspective, For time variables, The mapping relationship between color and radiation field intensity is as follows: , For any point, For dynamic four-dimensional foreground models in view The intensity of the radiation field under [the specified conditions].

[0150] based on right Rendering is performed, and in time steps. Add Gaussian noise To obtain the corresponding image .

[0151] predict The noise is calculated, and optimization is performed based on the predicted noise. The loss function used in the optimization is a weighted sum of the loss function for optimizing the 3D structure, the loss function for optimizing the visual appearance, and the loss function for optimizing the motion features.

[0152] Based on the optimized Generate a dynamic four-dimensional foreground.

[0153] Optionally, optimize the loss function of the three-dimensional structure. .

[0154] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For a multi-view diffusion model, To pass predict noise, For the parameters of the multi-view diffusion model, These are the camera parameters corresponding to each view.

[0155] Optionally, the loss function for optimizing the visual appearance is: .

[0156] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise, For the parameters of the text-to-image diffusion model, These are the camera parameters corresponding to each view.

[0157] Optionally, by time step Add Gaussian noise Simultaneously, the video sequence of that step size is rendered using NeRF. ,in, This represents the total number of video frames in the video sequence.

[0158] The loss function for optimizing motion features is: .

[0159] in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, exist Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

[0160] Optionally, based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated, including:

[0161] Obtain the rendered image and opacity map of the dynamic four-dimensional foreground.

[0162] Guided by background cues, the background area of ​​the rendered image is identified through an opacity graph.

[0163] By repairing the model, the background area is repaired.

[0164] A static 3D background is generated based on the repaired background area.

[0165] The loss function used in the process of generating the static 3D background is: .

[0166] in, The loss function used in the process of generating a static 3D background. For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For conditional diffusion models, To pass predict noise, For the parameters of the conditional diffusion model, For the background area to be repaired, and For camera orientation.

[0167] Optionally, the camera trajectory is determined by the elevation angle. Azimuth and distance In time composition.

[0168] The electronic device provided in this embodiment has a computer program executed by a processor to generate a dynamic four-dimensional foreground and a static three-dimensional background based on text prompts, and then render a video based on the dynamic four-dimensional foreground and the static three-dimensional background. This can make the video more in line with the physical laws of the real world and can also achieve precise camera motion control.

[0169] Based on the same inventive concept as the image generation method, this embodiment provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the above-described image generation method.

[0170] Specifically,

[0171] Get the text hint.

[0172] Break down text prompts into foreground prompts and background prompts.

[0173] Using foreground cues as guidance, generate dynamic four-dimensional foreground.

[0174] Based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated.

[0175] Render video along a given camera trajectory based on a dynamic 4D foreground and a static 3D background.

[0176] Optionally, guided by foreground cues, a dynamic four-dimensional foreground is generated, including:

[0177] Foreground hints To guide this process, a dynamic four-dimensional foreground model was obtained using Neural Radiation Field (NeRF). and coherent multi-view images ,in, For any point in the dynamic four-dimensional foreground model, For any perspective, For time variables, The mapping relationship between color and radiation field intensity is as follows: , For any point, For dynamic four-dimensional foreground models in view The intensity of the radiation field under [the specified conditions].

[0178] based on right Rendering is performed, and in time steps. Add Gaussian noise To obtain the corresponding image .

[0179] predict The noise is calculated, and optimization is performed based on the predicted noise. The loss function used in the optimization is a weighted sum of the loss function for optimizing the 3D structure, the loss function for optimizing the visual appearance, and the loss function for optimizing the motion features.

[0180] Based on the optimized Generate a dynamic four-dimensional foreground.

[0181] Optionally, optimize the loss function of the three-dimensional structure. .

[0182] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For a multi-view diffusion model, To pass predict noise, For the parameters of the multi-view diffusion model, These are the camera parameters corresponding to each view.

[0183] Optionally, the loss function for optimizing the visual appearance is: .

[0184] in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise, For the parameters of the text-to-image diffusion model, These are the camera parameters corresponding to each view.

[0185] Optionally, by time step Add Gaussian noise Simultaneously, the video sequence of that step size is rendered using NeRF. ,in, This represents the total number of video frames in the video sequence.

[0186] The loss function for optimizing motion features is: .

[0187] in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, exist Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

[0188] Optionally, based on a dynamic four-dimensional foreground and guided by background cues, a static three-dimensional background is generated, including:

[0189] Obtain the rendered image and opacity map of the dynamic four-dimensional foreground.

[0190] Guided by background cues, the background area of ​​the rendered image is identified through an opacity graph.

[0191] By repairing the model, the background area is repaired.

[0192] A static 3D background is generated based on the repaired background area.

[0193] The loss function used in the process of generating the static 3D background is: .

[0194] in, The loss function used in the process of generating a static 3D background. For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For conditional diffusion models, To pass predict noise, For the parameters of the conditional diffusion model, For the background area to be repaired, and For camera orientation.

[0195] Optionally, the camera trajectory is determined by the elevation angle. Azimuth and distance In time composition.

[0196] The computer-readable storage medium provided in this embodiment has a computer program thereon that is executed by a processor to generate a dynamic four-dimensional foreground and a static three-dimensional background based on text prompts, and then render a video based on the dynamic four-dimensional foreground and the static three-dimensional background. This can make the video more consistent with the physical laws of the real world and can also achieve precise camera motion control.

[0197] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0198] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0202] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image generation method characterized by, The method comprises: obtaining a text prompt; decomposing the text prompt into a foreground prompt and a background prompt; generating a dynamic four-dimensional foreground guided by the foreground prompt; generating a static three-dimensional background guided by the background prompt based on the dynamic four-dimensional foreground; rendering a video along a given camera trajectory based on the dynamic four-dimensional foreground and the static three-dimensional background; wherein the generating the dynamic four-dimensional foreground guided by the foreground prompt comprises: A dynamic four-dimensional foreground model is obtained by a neural radiance field (NeRF) for guidance and coherent multi-view images wherein, for any point in the dynamic four-dimensional foreground model, for any view angle, is a time variable, and the mapping relationship between color and radiance field intensity is: , is the color value of any point, is the radiance field intensity of the dynamic four-dimensional foreground model at the view angle . based on performing rendering on the rendering, and taking time steps adding Gaussian noise , obtaining the corresponding image ; predicting the noise of the and optimizing the ; wherein the loss function used in the optimization is a weighted sum of a loss function for optimizing the three-dimensional structure, a loss function for optimizing the visual appearance, and a loss function for optimizing the motion characteristics; based on the optimized , generating a dynamic four-dimensional foreground; In the loss function for optimizing the three-dimensional structure ; wherein, is a loss function for optimizing the three-dimensional structure, is a gradient operator for optimization, is a parameter of a multi-layer perceptron representing implicitly a dynamic four-dimensional foreground model, is an expectation function, is a weighting function, is a multi-view diffusion model, is a noise by predicting is a noise by is a parameter of a multi-view diffusion model, is a camera parameter corresponding to each view; wherein the loss function to optimize visual appearance is ; in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise, For the parameters of the text-to-image diffusion model, For the camera parameters corresponding to each view; wherein the time step adding Gaussian noise at the same time, rendering a video sequence for this step by NeRF wherein is the total number of video frames in the video sequence; The loss function of the optimization motion feature is ; in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, In order to be in Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

2. The method of claim 1, wherein, the generating the static three-dimensional background guided by the background prompt based on the dynamic four-dimensional foreground comprises: obtaining a rendered image and an opacity map of the dynamic four-dimensional foreground; identifying a background region of the rendered image through the opacity map guided by the background prompt; repairing the background region through a repair model; generating a static three-dimensional background based on the repaired background region; The loss function used in the generation of the static three-dimensional background is: ; wherein, a loss function employed for generating the static three-dimensional background process, a gradient operator for optimization, parameters of a multi-layer perceptron representing implicitly the dynamic four-dimensional foreground model, an expectation function, a weighting function, a conditional diffusion model, a prediction a noise, of the prediction, parameters of the conditional diffusion model, a repaired background region, and a camera pose.

3. The method of claim 1, wherein, The camera trajectory consists of an elevation angle , an azimuth angle , and a distance at a time .

4. An image generation method apparatus characterized by comprising: The device comprises: an obtaining module configured to obtain a text prompt; a decomposing module configured to decompose the text prompt obtained by the obtaining module into a foreground prompt and a background prompt; a foreground generating module configured to generate a dynamic four-dimensional foreground guided by the foreground prompt obtained by the decomposing module; a background generating module configured to generate a static three-dimensional background guided by the background prompt obtained by the decomposing module based on the dynamic four-dimensional foreground generated by the foreground generating module; a rendering module configured to render a video along a given camera trajectory based on the dynamic four-dimensional foreground generated by the foreground generating module and the static three-dimensional background generated by the background generating module; The foreground generation module 303 is configured to generate a foreground prompt A dynamic four-dimensional foreground model is obtained by a neural radiance field (NeRF) for guidance and coherent multi-view images , wherein For any point in the dynamic four-dimensional foreground model, For any view angle, is a time variable, and the The mapping relationship between color and radiance field intensity is: , is the color value of any point, is the radiance field intensity of the dynamic four-dimensional foreground model at the view angle ; based on rendering the , and adding Gaussian noise at a time step , to obtain the corresponding image ; predicting the noise of the , and optimizing the based on the predicted noise; wherein the loss function used in optimization is the weighted sum of the loss function for optimizing the three-dimensional structure, the loss function for optimizing the visual appearance, and the loss function for optimizing the motion characteristics; based on the optimized , a dynamic four-dimensional foreground is generated; In the loss function for optimizing the three-dimensional structure ; wherein, is a loss function for optimizing the three-dimensional structure, is a gradient operator for optimization, is a parameter of a multi-layer perceptron representing implicitly a dynamic four-dimensional foreground model, is an expectation function, is a weighting function, is a multi-view diffusion model, is a function of is a prediction of a noise, is a parameter of a multi-view diffusion model, is a camera parameter corresponding to each view; wherein the loss function to optimize visual appearance is ; in, To optimize the loss function of the three-dimensional structure, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-image diffusion models, To pass predict Predictive noise, To pass predict Fine-tuning noise, For the parameters of the text-to-image diffusion model, For the camera parameters corresponding to each view; wherein the time step adding Gaussian noise at the same time, rendering a video sequence for this step by NeRF wherein is the total number of video frames in the video sequence; The loss function of the optimization motion feature is ; in, To optimize the loss function for motion features, For the optimized gradient operator, The parameters of the multilayer perceptron are implicitly represented for the dynamic four-dimensional foreground model. Let be the expected function. For weighted functions, For text-to-video diffusion models, To pass predict noise, In order to be in Sampling noise at the location, These are the parameters for the text-to-video diffusion model.

5. An electronic device, comprising: comprise: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-3. having a computer program stored thereon; the computer program being executed by a processor to implement the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, ​

Images