Method and device for generating video content, equipment and storage medium

Abstract

The embodiment of the invention relates to a method and device for generating video content, equipment and a storage medium. The method provided by the invention comprises the following steps: constructing three-dimensional point cloud data based on a target image and depth information associated with the target image; generating a first group of images corresponding to a plurality of states of the virtual camera based on the three-dimensional point cloud data according to the motion trail of the virtual camera; and based on the first group of images and the prompt item, utilizing the target model to generate video content. In this way, according to the embodiment of the invention, the video content matched with the preset operation mirror can be generated, so that the quality of the generated video content is improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to a method, apparatus, device, and computer-readable storage medium for generating video content. Background Art

[0002] In recent years, video generation technology has made significant progress, especially video synthesis technology based on text prompts or image inputs. These technologies can generate videos with rich dynamic content through machine learning models (e.g., diffusion models). However, traditional video generation schemes lack effective control over camera movement when generating videos, which limits the expressiveness and diversity of video content. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for generating video content is provided. The method includes: constructing three-dimensional point cloud data based on a target image and depth information associated with the target image; generating a first group of images corresponding to multiple states of the virtual camera based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera; and generating video content using a target model based on the first group of images and prompt items.

[0004] In a second aspect of the present disclosure, a device for generating video content is provided. The device includes: a point cloud construction module configured to construct three-dimensional point cloud data based on a target image and depth information associated with the target image; an image generation module configured to generate a first group of images corresponding to multiple states of a virtual camera based on the three-dimensional point cloud data according to a motion trajectory of the virtual camera; and a video generation module configured to generate video content using a target model based on the first group of images and prompt items.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which some embodiments of the present disclosure can be implemented;

[0010] Figure 2 A flowchart showing a process of generating video content according to some embodiments of the present disclosure;

[0011] Figure 3 An example architecture of a video generation system according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic structural block diagram showing an example apparatus for generating video content according to some embodiments of the present disclosure; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0017] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0018] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0019] As briefly mentioned above, some video generation techniques can use machine learning models (e.g., diffusion models) to generate videos with rich dynamic content. However, traditional video generation schemes lack effective control over camera movements when generating videos, which limits the expressiveness and diversity of video content.

[0020] The embodiments of the present disclosure propose a scheme for generating video content. According to the scheme, three-dimensional point cloud data can be constructed based on a target image and depth information associated with the target image. Further, a first group of images corresponding to multiple states of a virtual camera can be generated based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera. Accordingly, video content can be generated using a target model based on the first group of images and prompt items.

[0021] In this way, the embodiments of the present disclosure can generate video content that matches the preset camera movement, thereby improving the quality of the generated video content.

[0022] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0023] Example Environment

[0024] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0025] In some embodiments, the electronic device 110 may obtain an input target image 120 and a prompt item 130. Further, the electronic device 110 may utilize a video generation system 140 to generate video content 150 based on the target image 120 and the prompt item 130. Such a prompt item 130 may include, for example, text content for describing the video content 150 to be generated, also referred to as a text prompt item or prompt word.

[0026] In some embodiments, such a video generation system 140 may be deployed locally on the electronic device 110 , or may be deployed at a suitable remote device, for example.

[0027] In some examples, the electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as "wearable" circuits, etc.).

[0028] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0029] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0030] Example Process

[0031] Figure 2 FIG. 2 is a flowchart showing an example process 200 for generating video content according to some embodiments of the present disclosure. The process 200 may be implemented, for example, in Figure 1 The electronic device 110 shown in FIG. Figure 1 The process 200 is described below.

[0032] like Figure 2 As shown, in box 210, the electronic device 110 constructs three-dimensional point cloud data based on the target image and depth information associated with the target image.

[0033] Figure 3A block diagram of an example video generation system 300 according to some embodiments of the present disclosure is shown. Such a video generation system 300 may correspond to Figure 1 The video generation system 140 shown in FIG.

[0034] Specifically, Figure 3 As shown, the electronic device 110 may acquire the target image 120, and may further determine depth information 305 associated with the target image 120. Such depth information 305 may, for example, include a depth map corresponding to the target image 120 to indicate the depth value of each pixel.

[0035] In some embodiments, the target image 120 may include a depth image, and its depth information may be directly determined based on the depth image. In some embodiments, the target image 120 may include a two-dimensional image, and the electronic device 110 may use a depth estimation module to determine the depth information 305. For example, the depth estimation module may use any appropriate depth estimation model to determine the depth information 305 corresponding to the target image 120.

[0036] Furthermore, if Figure 3 As shown, the electronic device 110 can convert multiple pixel points in the target image 120 into a three-dimensional space based on the target image 120 and the depth information 305 to generate three-dimensional point cloud data 315 .

[0037] Specifically, the construction process of the three-dimensional point cloud data 315 can be expressed as:

[0038]

[0039] in, represents the three-dimensional point cloud data 315, φ represents the mapping function from the two-dimensional image space to the three-dimensional space, represents the target image 120 , D0 represents the depth information 305 , K represents the internal parameters of the virtual camera corresponding to the target image 120 , and P0 represents the external parameters of the virtual camera.

[0040] Continue to refer Figure 2 In box 220, the electronic device 110 generates a first set of images corresponding to multiple states of the virtual camera based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera.

[0041] like Figure 3 As shown, the electronic device 110 can determine the motion trajectory 320 of the virtual camera, which can be expressed as [K, P i ], where K is the internal parameter of the virtual camera, P iRepresents the external parameters of the virtual camera in multiple states. Such external parameters may include, for example, the position, rotation, translation, and posture of the virtual camera in three-dimensional space.

[0042] Furthermore, the electronic device 110 may project the three-dimensional point cloud data onto a corresponding two-dimensional plane based on the external parameters of the virtual camera in multiple states to generate a corresponding first set of images. This process may be expressed as:

[0043]

[0044] Among them, I i represents the image corresponding to the i-th state of the virtual camera, and ψ represents the projection function.

[0045] In this way, the electronic device 110 can obtain a set of images corresponding to different states of the camera, and the dynamic switching between such images can correspond to the camera's mirror movement effects, such as zooming, translation, rotation, etc.

[0046] Continue to refer Figure 2 In block 230 , the electronic device 110 generates video content using a target model based on the first set of images and prompt items.

[0047] In some embodiments, Figure 3 As shown, the electronic device 110 may, for example, directly use the projected first set of images as input to the video generation model 335 to generate a corresponding second set of images as a plurality of video frames of the video content 150 .

[0048] Considering that the three-dimensional point cloud data 315 has holes at certain locations, the first set of images generated by the projection of the three-dimensional point cloud data 315 may have empty pixels. In some scenarios, in order to improve the quality of video content generation, the electronic device 110 may also use an image repair model to process one or more images in the first set of images to generate a repaired set of images (also called a third set of images). For example, the image repair model may use an appropriate image repair technology (in-painting) to fill in the empty pixels in the image.

[0049] In some embodiments, the electronic device 110 may use such a third set of images as input to the video generation model 335 to generate a second set of images.

[0050] In addition, since the filled empty pixels may be misaligned in the three-dimensional point cloud space, the electronic device 110 may further project the repaired third group of images into the three-dimensional space corresponding to the three-dimensional point cloud data 315 .

[0051] Furthermore, the electronic device 110 may update the three-dimensional point cloud data 315 by aligning multiple positions corresponding to the third set of images in the three-dimensional space. The alignment process may be expressed as:

[0052]

[0053] in, and Respectively represent the modified third set of images and the corresponding depth information, d i Indicates the depth parameter to be optimized, M indicates and The overlapping area of ​​​​||·|| indicates that the L1 loss is calculated

[0054] Based on this approach, the embodiments of the present disclosure can find the best depth coefficient so that the point cloud representations of the front and back images in the overlapping area are as consistent as possible. Thus, the embodiments of the present disclosure can correct the relative depth error caused by monocular depth estimation, thereby maintaining the coherence and consistency of objects and scenes in the generated video.

[0055] Additionally, the electronic device 110 may generate a fourth set of images corresponding to multiple states of the virtual camera based on the updated three-dimensional point cloud data. The process may be expressed as:

[0056]

[0057] by Figure 3 As an example, the fourth group of images generated by the electronic device 110 may be as follows: Figure 3 A set of images 325 is shown. Accordingly, the electronic device 110 may process the set of images 325 and the prompt item 130 using the video generation model 335 to generate a second set of images as a plurality of video frames.

[0058] In some embodiments, the video generation model 335 may include, for example, a diffusion model. Specifically, in the noise addition stage, the diffusion model may generate a set of hidden features corresponding to the fourth set of images by adding noise that satisfies a preset distribution. Further, in the denoising stage, the diffusion model may generate a corresponding second set of images based on the set of hidden features. The process may be expressed as:

[0059]

[0060] Specifically, formula (5) is used to obtain the noise potential representation from the rendered image sequence V0 325 through the forward diffusion process. 330. is the variance used in the DDIM scheduler (Denoising Diffusion Implicit Models); ∈ is the random noise sampled from the standard normal distribution, which is used to perturb the latent representation and increase the diversity of the generated images; t0 is a time step in the diffusion process, which determines the strength of the noise.

[0061] Formula (6) describes the use of the noisy latent representation Steps to generate a video via the back-diffusion process. is the denoised latent representation at time step t-1. t is a scheduling parameter that controls the denoising process. ∈ is the random noise sampled at each step. is the noise prediction part of the diffusion model. t Determines whether the denoising process is deterministic or probabilistic, and is usually set to 1 to encourage diversity in the generated results. t is the number of time steps in the diffusion process.

[0062] In some embodiments, the electronic device 110 can also balance the authenticity and diversity of the video content by controlling the time step t0, for example. In the generation process summary, the time step t0 used to generate the noise potential representation is a key factor affecting this trade-off. A larger t0 value will make the generated video closer to the original guided camera motion, but may sacrifice the dynamics and rationality of the video content. On the contrary, a smaller t0 value can produce a more reasonable video, but may not fully conform to the desired camera motion.

[0063] In this way, embodiments of the present disclosure are able to control camera motion during video generation by manipulating noise in the latent space, thereby achieving video generation in which camera motion can be controlled without additional training.

[0064] Furthermore, the electronic device 110 may generate the video content 150 by sequentially combining the second group of images so that the multiple video frames of the video content 150 can correspond to the multiple states [K, P i ]. It should be understood that such correspondence is intended to indicate that the video frames have similar camera control, but is not intended to indicate that the video frames completely correspond to the external parameters of the virtual camera in the corresponding state.

[0065] In some embodiments, the electronic device 110 may also generate the final video content 150 by adding other appropriate elements such as audio content, subtitles, etc.

[0066] Therefore, the embodiments of the present disclosure can utilize the explicit rearrangement of image layouts in 3D point cloud space and the layout priors of noise latent representations to generate videos with rich dynamic content and high realism, while maintaining the efficiency of the processing and the robustness of the model.

[0067] In addition, the embodiments of the present disclosure can support complex hybrid camera motions and can be widely used in fields such as 3D video generation and virtual reality content creation, thereby improving the flexibility and innovation of video content creation.

[0068] Example devices and equipment

[0069] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example apparatus 400 for generating video content according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in an electronic device. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0070] like Figure 4 As shown, the device 400 includes a point cloud construction module 410, which is configured to construct three-dimensional point cloud data based on a target image and depth information associated with the target image; an image generation module 420, which is configured to generate a first group of images corresponding to multiple states of the virtual camera based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera; and a video generation module 430, which is configured to generate video content using a target model based on the first group of images and prompt items.

[0071] In some embodiments, the image generation module 420 is further configured to: determine multiple sets of external parameters of the virtual camera in multiple states based on the motion trajectory of the virtual camera; and generate a first set of images corresponding to the multiple sets of external parameters by projecting three-dimensional point cloud data.

[0072] In some embodiments, the video generation module 430 is further configured to: process at least one image in the first group of images using an image inpainting model to generate a third group of images; and generate a second group of images using a target model based on the third group of images and prompt items.

[0073] In some embodiments, the video generation module 430 is also configured to: project a third set of images into a three-dimensional space corresponding to the three-dimensional point cloud data; update the three-dimensional point cloud data by aligning multiple positions corresponding to the third set of images in the three-dimensional space; generate a fourth set of images corresponding to multiple states of the virtual camera based on the updated three-dimensional point cloud data; and provide the fourth set of images and prompt items to the target model to generate a second set of images.

[0074] In some embodiments, the target model is a diffusion model, and the diffusion model is configured to: generate a set of hidden features corresponding to the fourth group of images by adding noise that satisfies a preset distribution; and generate the corresponding second group of images based on the set of hidden features.

[0075] In some embodiments, the apparatus 400 further includes a depth estimation module configured to: process the target image using a depth estimation model to determine depth information.

[0076] In some embodiments, the video content includes a plurality of video frames corresponding to a plurality of states of the virtual camera.

[0077] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used for Figure 1 An electronic device 110 is shown.

[0078] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0079] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 500.

[0080] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0081] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0082] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0083] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0084] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0086] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0087] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0088] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating video content, comprising: constructing three-dimensional point cloud data based on the target image and depth information associated with the target image; generating a first group of images corresponding to a plurality of states of the virtual camera based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera; as well as Based on the first set of images and prompt items, video content is generated using a target model.

2. The method according to claim 1, wherein generating a first set of images corresponding to a plurality of states of the virtual camera based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera comprises: Based on the motion trajectory of the virtual camera, determining multiple groups of external parameters of the virtual camera in the multiple states; as well as The first set of images corresponding to the multiple sets of external parameters are generated by projecting the three-dimensional point cloud data.

3. The method of claim 1, wherein generating video content using a target model based on the first set of images and prompt items comprises: processing at least one image in the first set of images using an image inpainting model to generate a third set of images; as well as Based on the third set of images and the prompt item, a second set of images in the video content is generated using the target model.

4. The method of claim 3, wherein generating a second set of images in the video content using the target model based on the third set of images and the prompt item comprises: Projecting the third set of images into a three-dimensional space corresponding to the three-dimensional point cloud data; updating the three-dimensional point cloud data by aligning a plurality of positions corresponding to the third set of images in the three-dimensional space; generating a fourth set of images corresponding to the multiple states of the virtual camera based on the updated three-dimensional point cloud data; as well as The fourth set of images and the prompt items are provided to the target model to generate the second set of images.

5. The method according to claim 4, wherein the target model is a diffusion model, and the diffusion model is configured as follows: Generating a set of hidden features corresponding to the fourth set of images by adding noise satisfying a preset distribution; and Based on the set of hidden features, a corresponding second set of images is generated.

6. The method according to claim 1, further comprising: The target image is processed using a depth estimation model to determine the depth information. 7 . The method of claim 1 , wherein the video content comprises a plurality of video frames corresponding to the plurality of states of the virtual camera.

8. An apparatus for generating video content, comprising: a point cloud construction module configured to construct three-dimensional point cloud data based on a target image and depth information associated with the target image; An image generation module is configured to generate a first set of images corresponding to a plurality of states of the virtual camera based on the three-dimensional point cloud data according to the motion trajectory of the virtual camera; as well as The video generation module is configured to generate video content using a target model based on the first group of images and prompt items.

9. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 7 when executed by the at least one processing unit.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images