Layout-controllable video generation method and device, equipment, medium and product

Abstract

Embodiments of the present disclosure disclose a layout controllable video generation method, device, equipment, medium and product, the method comprising: inputting sparse layout guide conditions, text information features and preset noise hidden space features into a pre-trained video generation model for preset time step denoising processing, in each time step denoising processing: an interpolation module in the video generation model processes the sparse layout guide conditions to obtain first layout features, an adapter module in the video generation model performs guide regulation processing on the first layout features to output second layout features, a video generation module in the video generation model obtains intermediate features based on the text information features, the second layout features and the preset noise hidden space features, and determines a target video based on the denoising processing of each time step and the intermediate features of each time step. Thus, the consistency and coherence of the generated target video are improved, thereby improving the quality of the target video.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a video generation technique, in particular to a layout-controllable video generation method, device, equipment, medium and product. BACKGROUND

[0002] Video generation refers to automatically or semi-automatically creating dynamic image content with certain visual coherence and semantic consistency by using artificial intelligence and deep learning technology. Video generation technology is widely used in animation production, video editing, virtual reality (VR) environment generation, film post-production, and game development. In the related art, video generation usually involves automatically generating a continuous frame sequence that meets specific requirements by using model algorithms based on given input conditions (such as text descriptions, etc.), thereby forming a complete video segment. In recent years, layout-controllable video generation technology has emerged, which allows users to explicitly specify the spatial position, layout relationship and dynamic changes of each element in the picture during the video generation process. However, the existing layout-controllable video generation method still has obvious deficiencies, for example, the granularity of layout constraints is relatively coarse, and it is difficult to accurately specify the detailed position and size ratio of each element in the video picture; in addition, the control of the spatial layout during the video generation process often cannot maintain coherence and stability, resulting in deviation of the generated video content from the user-specified spatial layout constraints, thereby reducing the visual effect and actual usability of the generated video. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present disclosure provide a layout-controllable video generation method, device, equipment, medium and product.

[0004] In an aspect of the embodiments of the present disclosure, a layout controllable video generation method is provided, including: obtaining sparse layout guide conditions corresponding to a target video, text information features, and hidden space features of preset noise, wherein the sparse layout guide conditions include hidden space features of a sparse layout image sequence corresponding to the target video, and the sparse layout image sequence includes layout information of the target video; inputting the sparse layout guide conditions, the text information features, and the hidden space features of the preset noise into a video generation model for preset time step denoising processing to obtain the target video; the video generation model includes an interpolation module, an adapter module, and a video generation module; in each time step of denoising processing: the interpolation module performs interpolation processing on the sparse layout guide conditions to obtain dense layout guide conditions, and performs fusion processing on the dense layout guide conditions to obtain first layout features, wherein the dense layout guide conditions include hidden space features of a dense layout image sequence, and the dense layout image sequence includes the sparse layout image sequence and an interpolation image obtained through interpolation processing; the adapter module performs guide regulation processing on the first layout features to output second layout features; the video generation module obtains intermediate features based on the text information features, the second layout features, and the hidden space features of the preset noise; and the target video is determined based on the denoising processing of each time step and the intermediate features of each time step.

[0005] In another aspect of the embodiments of the present disclosure, a layout controllable video generation device is provided, including: a first data acquisition module configured to obtain sparse layout guide conditions corresponding to a target video, text information features, and hidden space features of preset noise, wherein the sparse layout guide conditions include hidden space features of a sparse layout image sequence corresponding to the target video, and the sparse layout image sequence includes layout information of the target video;

[0006] The video generation module is configured to input the sparse layout guide condition, the text information feature, and the hidden space feature of the preset noise into a video generation model for denoising processing at a preset time step to obtain the target video. The video generation model comprises an interpolation module, an adapter module, and a video generation module. In the denoising processing at each time step: the interpolation module performs interpolation processing on the sparse layout guide condition to obtain a dense layout guide condition, and performs fusion processing on the dense layout guide condition to obtain a first layout feature, wherein the dense layout guide condition comprises a hidden space feature of a dense layout image sequence, and the dense layout image sequence comprises the sparse layout image sequence and an interpolation image obtained through interpolation processing; the adapter module performs guide regulation processing on the first layout feature to output a second layout feature; and the video generation module obtains an intermediate feature based on the text information feature, the second layout feature, and the hidden space feature of the preset noise. The target video is determined based on the denoising processing at each time step and the intermediate feature at each time step.

[0007] In yet another aspect of the embodiments of the present disclosure, an electronic device is provided, which includes a memory configured to store a computer program, and a processor configured to execute the computer program stored in the memory, and the computer program, when executed, implements the method described above.

[0008] In yet another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the method described above.

[0009] In yet another aspect of the embodiments of the present disclosure, a computer program product is provided, which includes computer program instructions, and the computer program instructions, when executed by a processor, implement the method described above.

[0010] In the embodiments of the present disclosure, the interpolation module increases the hidden space feature of the interpolation image by performing interpolation processing on the hidden space feature of each layout image in the sparse layout image sequence, so that the video generation module can better fuse the first layout feature, the text information feature, and the hidden space feature of the preset noise, and the consistency and continuity of the generated target video can be improved. Meanwhile, the adapter module maps lines with different degrees of abstraction in the first layout feature to different fusion weights, so that the video generation module can more flexibly follow the control of the layout information, thereby further improving the quality of the target video.

[0011] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0013] The present disclosure can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the following drawings of which:

[0014] Figure 1 is a flowchart of a layout-controllable video generation method provided by an exemplary embodiment of the present disclosure;

[0015] Figure 2 is a flowchart of step S100 provided by an exemplary embodiment of the present disclosure;

[0016] Figure 3 is an application diagram of a layout-controllable video generation method provided by an exemplary embodiment of the present disclosure;

[0017] Figure 4 is a flowchart of a layout-controllable video generation method provided by another exemplary embodiment of the present disclosure;

[0018] Figure 5 is a structural diagram of an embodiment of a layout-controllable video generation device based on the present disclosure;

[0019] Figure 6 is a structural diagram of an application embodiment of an electronic device of the present disclosure. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present disclosure will now be described in detail below with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments are not intended to limit the scope of the present disclosure unless otherwise specifically stated.

[0021] Those skilled in the art can understand that the terms "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they represent a necessary logical sequence between them.

[0022] It should also be understood that in the embodiments of the present disclosure, "multiple" can mean two or more, and "at least one" can mean one, two, or more.

[0023] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present disclosure, unless specifically limited or given a contrary implication by the context or prior art, it can generally be understood as one or more.

[0024] In addition, the term "and / or" in the present disclosure is merely a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0025] It should also be understood that the description of the various embodiments of the present disclosure focuses on the differences between the various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be described one by one.

[0026] At the same time, it should be understood that, for the convenience of description, the size of each part shown in the drawings is not drawn according to the actual proportional relationship.

[0027] The following description of at least one example embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0028] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification where appropriate.

[0029] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0030] Embodiments of the present disclosure can be applied to terminal devices, computer systems, servers, and other electronic devices, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems.

[0031] Electronic devices such as terminal devices, computer systems, servers, and the like can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer systems / server can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local and / or remote computer system storage media including memory storage devices.

[0032] In practical applications, during the video generation process, due to the coarse granularity of the layout constraint and the inability to accurately specify the detailed position and size ratio of each element in the video picture, the generated video content deviates from the spatial layout constraint specified by the user, reducing the visual effect and actual usability of the generated video.

[0033] Figure 1 is a flowchart of a layout-controllable video generation method provided by an example embodiment of the present disclosure. The embodiment can be applied on an electronic device, such as a terminal device, a server, a computer system, etc. Figure 1 As shown in the figure, the method comprises the following steps:

[0034] In step S100, the sparse layout guide condition corresponding to the target video, the text information feature, and the hidden space feature of the preset noise are obtained.

[0035] The sparse layout guide condition comprises the hidden space feature of the sparse layout image sequence corresponding to the target video, and the sparse layout image sequence comprises the layout information of the target video.

[0036] The sparse layout image sequence comprises a plurality of layout images with a time sequence relationship. Each layout image can comprise layout information of the target video, which can include, for example, the position, size, and composition of at least one target element in the target video. The target element can be, for example, a person, an animal, furniture, a road sign, etc. The hidden space feature of the layout image can be obtained by encoding the image feature of the layout image. The hidden space feature of each layout image can include the time sequence and spatial semantic representation of the layout information. The text information comprises the video content information of the target video, which can be obtained by encoding the text information using a text encoder.

[0037] The preset noise can include Gaussian noise. Exemplarily, a Gaussian noise matrix can be randomly generated in advance as the preset noise. The preset noise can be subjected to latent space mapping processing to obtain a latent space feature of the preset noise. For example, the preset noise can be input into an encoder of a 3D variational autoencoder (3D VAE), and the latent space feature of the preset noise is output by the 3D VAE.

[0038] In step S110, the sparse layout guidance condition, the text information feature, and the latent space feature of the preset noise are input into the video generation model for denoising processing at a preset time step to obtain the target video.

[0039] The video generation model includes an interpolation module, an adapter module, and a video generation module. The adapter module is used to guide and regulate the layout information to enhance the consistency and controllability of generating the first layout feature. The adapter module can use a ControlNet adapter, for example. The ControlNet adapter is an efficient adaptation framework derived from the ControlNet technology, which can control the content and style of generated images through image hints (matte), and can solve complex problems such as multi-condition control, cross-model compatibility, and video generation in diffusion models. The video generation module is used to generate a video, and the video generation module can use a DiT model (Diffusion Transformer), for example. The DiT model is a diffusion model combined with a Transformer architecture, mainly used for image and video generation tasks. The core of the DiT model is to integrate the Transformer architecture into the diffusion model to replace the traditional convolutional network (such as the U-Net network) to process the latent representation of high-dimensional data. The DiT model can capture global dependencies using the self-attention mechanism of the Transformer, which can significantly improve the quality of generated images and videos. The interpolation module can use a deep learning model, which is used to perform interpolation processing on the layout image. The interpolation module can use a Unet network, a MoviiGen1.1 model, or a Wan2.1 model, for example.

[0040] In the denoising processing at each time step: the interpolation module performs interpolation processing on the sparse layout guidance condition to obtain a dense layout guidance condition, and performs fusion processing on the dense layout guidance condition to obtain the first layout feature; the adapter module performs guidance and regulation processing on the first layout feature to output a second layout feature; and the video generation module obtains an intermediate feature based on the text information feature, the second layout feature, and the latent space feature of the preset noise, and determines the target video based on the denoising processing at each time step and the intermediate feature at each time step.

[0041] The dense layout guiding condition comprises the latent space feature of the sparse layout image sequence and the latent space feature of the interpolated image obtained through the interpolation processing.

[0042] In an embodiment, the interpolation module inserts the latent space feature of the interpolated image into the latent space features of the plurality of layout images in the sparse layout image sequence based on the frame rate of the target video, so that the frame rate of the dense layout image sequence after interpolation is the same as the frame rate of the target video. The first layout feature can comprise layout information of the dense layout image sequence. The dense layout image sequence can be obtained by feature fusion of the latent space features of the dense layout image sequence.

[0043] In an embodiment, the intermediate feature of each time step can perform the above operation as the latent space feature of the preset noise of the next time step, and the target video is determined based on the intermediate feature of the last time step. For example, the intermediate feature of the last time step can be input into the decoder, and the intermediate feature can be decoded by the decoder to obtain the target video.

[0044] In the embodiments of the present disclosure, the interpolation module increases the latent space feature of the interpolated image by performing interpolation processing on the latent space features of the layout images in the sparse layout image sequence, so that the video generation module can better fuse the latent space features of the first layout feature, the text information feature and the preset noise, and can improve the consistency and continuity of the generated target video. At the same time, the adapter module maps the lines with different degrees of abstraction in the first layout feature to different fusion weights, so that the video generation module can more flexibly follow the control of the layout information, thereby further improving the quality of the target video.

[0045] In some optional embodiments, in the embodiments of the present application, the interpolation processing of the sparse layout guiding condition by the interpolation module can comprise: the interpolation module performs interpolation processing on the sparse layout guiding condition based on the frame rate of the target video to obtain the dense layout guiding condition. The frame rate of the dense layout image sequence is the same as the frame rate of the target video.

[0046] In the embodiments of the present application, the frame rate of the target video is used as the frame rate of the dense layout image sequence, so that a video with a corresponding frame rate can be generated according to requirements.

[0047] Figure 2 is a flowchart of step S100 provided by an exemplary embodiment of the present disclosure. In some optional embodiments, as shown in Figure 2 step S100 can comprise the following steps:

[0048] Step S101, image feature extraction is performed on the sparse layout image sequence to obtain image features corresponding to the sparse layout image sequence.

[0049] The image features corresponding to the sparse layout image sequence include image features of each layout image in the sparse layout image sequence. The image features can include structured visual features of target elements in the target video, i.e., can include layout reference lines of different levels of abstraction of the target elements. The structured visual features refer to organizing and representing the visual information of the target elements in the layout image according to certain structures and levels. Exemplarily, the structured visual features can include, but are not limited to, line drawings, doodles, stick figures, etc.

[0050] In an embodiment, for each frame of the layout image in the sparse layout image sequence, a plurality of structured visual features of the layout image can be extracted by using an image processing algorithm or a layout analysis model of a lightweight neural network to obtain the image features of the layout image. The layout analysis model can be, for example, MobileNet or GhostNet, etc.

[0051] Step S102, hidden space mapping processing is performed on the image features corresponding to the sparse layout image sequence to obtain sparse layout guide conditions.

[0052] In an embodiment, for the image features corresponding to the sparse layout image sequence, the image features can be encoded to perform hidden space mapping processing on the image features to obtain hidden space features corresponding to the image features. Exemplarily, the image features can be input into an encoder of a 3D VAE to extract the temporal and spatial semantic representation of each image feature in the hidden space to obtain the hidden space features (tensor) of the image features. The hidden space features of the image constitute the sparse layout guide conditions.

[0053] In the embodiments of the present disclosure, the image features of each layout image in the sparse layout image sequence are obtained, and the image features are mapped to the hidden space to obtain the hidden space features corresponding to the sparse layout image sequence, thereby efficiently obtaining the hidden space features of the layout images.

[0054] In some optional embodiments, the text information features can be obtained in the embodiments of the present disclosure by: obtaining text information corresponding to the target video, and then using a text encoder to extract features of the text information to obtain the text information features.

[0055] The text encoder can employ a deep learning model. For example, the text encoder can employ a Contrastive Language-Image Pre-training (CLIP) model. The CLIP model is a deep learning model that combines image and text information. The CLIP model is trained by simultaneously considering images and text descriptions related to the images. The CLIP model can represent text and images in a unified embedding space. Specifically, text information can be input into the CLIP model, and the CLIP model outputs a text vector (text information feature). The text vector will be used together with the latent space feature of the preset noise and the layout image in the process of generating the target video.

[0056] In the embodiments of the present disclosure, the text information is processed by the text encoder to obtain a text information feature. The text information feature can be combined with the latent space feature of the layout image, so that the video generation module can consider the layout image and the text information at the same time, and the text information and the layout image can be compared and analyzed in the same space, which facilitates feature association and learning.

[0057] In some optional embodiments, the first layout feature can be obtained by performing feature fusion processing on the latent space features of the dense layout guide conditions to obtain a layout fusion feature, and then performing feature alignment processing on the layout fusion feature and the current feature in the video generation module based on the noise information of the current time step in the video generation model to obtain the first layout feature.

[0058] The latent space features in the dense image sequence can be fused according to a weight fusion manner to obtain the layout fusion feature. A time step represents a time interval. The video generation module can define that a preset time step is spent to add noise to an image to obtain pure noise, or to remove noise from the image to obtain an image. In each time step, the video generation module processes data, usually by adding or removing noise from the data. The noise information of the current time step in the video generation module can include noise added or removed from the data in the video generation module. The current feature in the video generation module represents a feature corresponding to the latent space feature of the preset noise after denoising at the current time step. The first layout feature can be added with noise based on the noise in the noise information of the current time step, so that the first layout feature is aligned with the current feature in the video generation module.

[0059] In the embodiments of the present disclosure, the latent space features of the dense layout guide conditions are fused and aligned with the current feature in the video generation module, thereby ensuring that the layout information is continuous and reasonable in each time segment in the target video.

[0060] In some optional embodiments, in the embodiments of the present disclosure, the adapter module comprises n first network layers, the video generation module comprises n second network layers and n third network layers, and the n first network layers correspond to the n second network layers one by one.

[0061] The adapter module and the video generation module have the same structure. The number of network layers in the adapter module is half of the number of network layers in the video generation module. In this embodiment, the network layers in the adapter module are referred to as first network layers. In the video generation module, the first network layer to the n-th network layer are referred to as second network layers, and the n+1-th network layer to the 2n-th network layer are referred to as third network layers. Exemplarily, the first network layers, the second network layers and the third network layers can adopt a Transformer model.

[0062] Correspondingly, the adapter module and the video generation module in the embodiments of the present disclosure can process the first layout feature, the text information feature and the hidden space feature of the preset noise in the following manner: each first network layer sequentially performs feature extraction processing on the first layout feature to obtain a second layout feature, for each second network layer, the first vector output by the first network layer corresponding to the second network layer is input into the second network layer, the second network layer performs feature fusion based on the input first vector, the hidden space feature of the preset noise and the text information feature to obtain a second vector, and the second vector output by the n-th second network layer is used to determine the intermediate fusion feature.

[0063] The second layout feature comprises the first vectors output by the first network layers respectively. In one embodiment, in the denoising processing at each time step, the first layout feature can be input into the adapter module, each first network layer in the adapter module sequentially performs guided adjustment processing on the first layout feature, each first network layer outputs a second vector, and each second vector is cached.

[0064] In one embodiment, in the denoising processing at the m-th (m is a time step in the preset time steps) time step, the first first network layer to the n-th first network layer correspond to the first second network layer to the n-th second network layer one by one.

[0065] For the i-th (i < n) second network layer, the first vector output by the i-th first network layer is input into the i-th second network layer, the i-th second network layer fuses the input first vector (the first vector output by the i-th first network layer), the preset noise hidden space feature and the text information feature, outputs a second vector, and takes the second vector output by the i-th second network layer as the preset noise hidden space feature of the (i+1)-th second network layer. The above operation is repeated until the n-th second network layer, and the second vector output by the n-th second network layer is determined as the intermediate fusion feature. The intermediate fusion feature is determined as the input information input into the 1st third network layer, and the input information is denoised based on the text information feature by the 1st third network layer to obtain an intermediate result. The intermediate result is taken as the input information of the 2nd third network layer, and the operation in the 1st third network layer is repeated until the n-th third network layer. The intermediate result output by the n-th third network layer is determined as the intermediate feature of the m-th time step.

[0066] The intermediate feature of the m-th time step is determined as the preset noise hidden space feature of the (m+1)-th time step, and then the denoising processing of the (m+1)-th time step is performed until the denoising processing of the last time step. The intermediate feature of the last time step is input into the decoder of the 3D VAE, and the intermediate feature is decoded by the decoder of the 3D VAE to obtain the target video.

[0067] Correspondingly, in the embodiment of the present disclosure, the video generation model further comprises an initialization module. The initialization module is configured to adjust the data distribution. The initialization module can adopt a linear neural network. For example, the initialization module can adopt a zero-initialized linear network. The zero-initialized linear network refers to initializing all weight parameters (weight matrix and bias vector) of the linear neural network (without a nonlinear activation function, only containing a linear transformation) to zero. The initialization module can also adopt a convolutional neural network and the like.

[0068] In the embodiment of the present disclosure, before the first vectors are input into the video generation module, the first vectors are further input into an initialization module, and the data distribution of each first vector is adjusted by the initialization module to align the data distribution of each first vector with the data distribution in the video generation module.

[0069] The data distribution is used to describe the dispersion of data in different value intervals. Each first vector can be input into the initialization module, and the data distribution of each first vector is adjusted by the initialization module to make the data distribution of each first vector the same as the data distribution in the video generation module.

[0070] In the embodiments of the present disclosure, the first layout features are processed by each first network layer in the adapter module, and each first network layer outputs a corresponding first vector. Each first vector is processed by each second network layer in the video generation module, so that the denoising of each time step is jointly guided by the layout information and the text information features, and the generation process not only retains the temporal rationality but also ensures the accurate control of the content structure and semantics.

[0071] Figure 3 is an application diagram of the layout-controllable video generation method provided by an example embodiment of the present disclosure. As shown in Figure 3 , it includes:

[0072] The video generation model includes an interpolation module, an initialization module, an adapter module, and a video generation module. The adapter module includes n first network layers, the video generation module includes n second network layers and n third network layers, and the n first network layers correspond to the n second network layers one by one.

[0073] A preset noise, text information, and a sparse layout image sequence are obtained. Image features of each layout image in the sparse layout image sequence are extracted to obtain image features of the sparse layout image sequence, and the image features of the sparse layout image sequence are input into an encoder of a 3D VAE. The encoder of the 3D VAE outputs hidden space features of the sparse layout image sequence, and the hidden space features of the sparse layout image sequence constitute a sparse guidance condition. The text information is input into a text encoder, and the text encoder outputs text information features. The text encoder uses a CLIP model. The preset noise is input into the encoder of the 3D VAE, and the encoder of the 3D VAE outputs hidden space features of the preset noise.

[0074] The sparse layout guidance condition, the text information features, and the hidden space features of the preset noise are input into the video generation model for denoising processing of a preset time step to obtain a target video.

[0075] In the denoising processing of the mth time step: the sparse layout guidance condition is input into the interpolation module, the interpolation module performs interpolation processing on the sparse layout guidance condition to obtain a dense layout guidance condition, the dense layout guidance condition includes hidden space features of a dense layout image sequence, the dense layout image sequence includes the sparse layout image sequence and an interpolation image obtained by interpolation, the hidden space features of the dense layout guidance condition are processed by feature fusion to obtain layout fusion features, and based on noise information of the mth (current) time step in the video generation model, the layout fusion features and the current features in the video generation module are processed by feature alignment to obtain first layout features;

[0076] The first layout features are input into the adapter module, and the n first network layers in the adapter module output first vectors respectively;

[0077] The first vectors are input into an initialization module, and the data distribution of each first vector is adjusted by the initialization module to align the data distribution of each first vector with the data distribution in the video generation module.

[0078] Then, for the i-th second network layer, the first vector output by the i-th first network layer is input into the i-th second network layer, the i-th second network layer fuses the input first vector, the preset noise hidden space feature and the text information feature, outputs a second vector, and the second vector output by the i-th second network layer is determined as the preset noise hidden space feature of the i+1-th second network layer. The above operation is repeated until the n-th second network layer, and the second vector output by the n-th second network layer is determined as the intermediate fusion feature. The intermediate fusion feature is determined as the input information input into the 1st third network layer, and the input information is denoised by the 1st third network layer based on the text information feature to obtain an intermediate result. The intermediate result is used as the input information of the 2nd third network layer, and the operation in the 1st third network layer is repeated until the n-th third network layer. The intermediate result output by the n-th third network layer is determined as the intermediate feature of the m-th time step.

[0079] The intermediate feature of the m-th time step is determined as the preset noise hidden space feature of the m+1-th time step, and then the denoising processing of the m+1-th time step is performed until the denoising processing of the last time step. The intermediate feature of the last time step is input into the decoder of the 3D VAE, and the intermediate feature is decoded by the decoder of the 3D VAE to obtain the target video.

[0080] Figure 4 is a flowchart of a layout-controllable video generation method provided by another exemplary embodiment of the present disclosure. In some optional embodiments, as shown in Figure 4 The video generation model can be obtained in the following manner:

[0081] In step S200, a to-be-trained model and sample data are obtained.

[0082] The sample data includes a plurality of sample videos, and for each sample video, a label noise, a hidden space feature and a label text information feature corresponding to the sample video. The label noise of each sample video is obtained by adding noise to the sample video.

[0083] For each sample video, the sample video can be input into an encoder of the 3D VAE, and the encoder of the 3D VAE outputs a latent space feature of the sample video. The sample video can be input into a vision-language model (VLM), and the vision-language model outputs label text information corresponding to the sample video, and the label text information is output to a CLIP model, and the CLIP model outputs a label text information feature. The sample video can be added with Gaussian noise to obtain label noise.

[0084] The to-be-trained model includes an interpolation module, a to-be-trained adapter module, an initialization module, and a video generation module. The structures of the interpolation module, the to-be-trained adapter module, the initialization module, and the video generation module in the to-be-trained model are the same as those of the interpolation module, the adapter module, the initialization module, and the video generation module in the video generation module, which will not be described herein.

[0085] In step S210, for a plurality of sample videos, based on difference information between adjacent two frame video images in the sample video, a label sparse layout image sequence is obtained from the sample video.

[0086] The label sparse layout image sequence includes a plurality of label layout images with a time sequence relationship. The difference information between adjacent two frame images can include a difference measurement value between the adjacent two frame video images.

[0087] In an embodiment, a preset number of continuous video images can be extracted as initial images in each time period of the sample video. For example, 16 continuous video images (a preset number of frames) can be extracted as initial images in each second (each time period) of the sample video, and the plurality of initial images can be denoted as F_1, F_2, …, F_n.

[0088] For each initial image, a pixel difference value between the initial image and a previous initial image of the initial image is calculated, and the pixel difference value is normalized to obtain a difference measurement value between the initial image and the previous initial image. The normalization of the pixel difference value can be averaging or summing the pixel difference value and the pixel of the gray image corresponding to the initial image. For example, the pixel difference value can be obtained by an absolute difference graph D_t = |F_t - F_{t-1}|, where D_t is the pixel difference value, F_t is the pixel value of the initial image, and F_{t-1} is the pixel value of the previous initial image.

[0089] Determine whether the initial image is a label layout image based on the preset threshold and the difference measure value. For example, when Delta_t (difference measure value) > T (preset threshold), it indicates that the difference between the initial image and the previous frame initial image is significant, and the initial image is determined as a label layout image. When Delta_t≤T, it indicates that the difference between the initial image and the previous frame initial image is not significant, and the initial image is not determined as a label layout image.

[0090] In step S220, a label sparse layout guide condition is determined based on the label sparse layout image sequence.

[0091] The label sparse layout guide condition includes the latent space features of each label layout image in the label sparse layout image sequence.

[0092] The way to obtain the latent space features of the label layout image is the same as the way to obtain the latent space features of the layout image, which will not be repeated here.

[0093] In step S230, the label noise corresponding to the sample video, the latent space features, the label text information features, and the label sparse layout guide condition are input into the to-be-trained model for noise addition processing at a preset time step, to obtain a predicted noise.

[0094] In each time step of the noise addition processing: the interpolation module performs interpolation processing and feature fusion processing on the label sparse layout guide condition to obtain a label first layout feature, the to-be-trained adapter module performs guide regulation processing on the label first layout feature to output a label second layout feature, the initialization module adjusts the data distribution of the label second layout feature, and the video generation module processes the label text information features, the label second layout feature after data distribution adjustment, and the latent space features of the sample video to obtain an intermediate feature. Based on the noise addition processing at each time step and the label intermediate feature at each time step, the preset noise is determined.

[0095] In each time step of the noise addition processing: the interpolation module performs interpolation processing and feature fusion processing on the label sparse layout guide condition to obtain a label first layout feature, the to-be-trained adapter module performs guide regulation processing on the label first layout feature to output a label second layout feature, the initialization module adjusts the data distribution of the label second layout feature, and the video generation module processes the label text information features, the label second layout feature after data distribution adjustment, and the latent space features of the sample video to obtain an intermediate feature. Based on the noise addition processing at each time step and the label intermediate feature at each time step, the preset noise is determined.

[0096] The manner in which the initialization module processes the label second layout feature can be seen with reference to the manner in which the initialization module processes the second layout feature, and will not be described again here. The manner in which the video generation module performs the forward process (adding noise) on the latent space feature of the sample video based on the label text information feature and the label second layout feature after adjustment of the data distribution can be seen with reference to the manner in which the video generation module processes the preset noise based on the text information feature and the second layout feature in the reverse process (denoising), and only the denoising at each time step is replaced by adding noise, and will not be described again here.

[0097] At step S240, the parameters of the adapter module to be trained are adjusted according to each predicted noise and the corresponding label noise until a preset training end condition is met, and a video generation model is obtained from the model to be trained.

[0098] In the model training stage, the parameters of the frame insertion module, the initialization module and the video generation module can be frozen, and the parameters of the input layer of the video generation module are used as the initialization parameters of the adapter module to be trained.

[0099] Specifically, the training process is to iteratively adjust the parameters of the adapter module, and the parameters of the frame insertion module, the initialization module and the video generation module are fixed and unchanged.

[0100] In an embodiment, based on the difference between the predicted noise and the label noise corresponding to each sample video, a preset loss function can be used to determine the loss function value. The preset loss function may, for example, include but is not limited to a cross-entropy error function or a mean square error function. The operation of determining the loss function value can be iteratively performed, and the parameters of the adapter network module to be trained are iteratively adjusted to continuously reduce the loss function value until the loss function value converges, and the training of the model to be trained is completed when the preset training end condition is met. The trained model to be trained is used as the video generation model.

[0101] A parameter optimizer such as Stochastic Gradient Descent (SGD), Adagrad, Adaptive Moment Estimation (Adam), Root Mean Square Prop (RMSprop) and the like can be used to adjust the parameters of the adapter network module to be trained. For example, the parameter optimizer can be used to calculate the gradient of each parameter of the adapter network module to be trained, and the parameter is adjusted in the direction of the gradient, where the gradient represents the direction in which the loss function value decreases the most. The operation of determining the loss function value is iteratively determined until the loss function value no longer decreases, and the training of the model to be trained is completed to obtain the video generation model.

[0102] In the embodiments of the present disclosure, only the parameters of the adapter module to be trained need to be adjusted in the model training process, thereby reducing the model training difficulty and improving the model training efficiency.

[0103] Figure 5 A structural schematic diagram of an embodiment of a video generation device based on layout controllability is provided in the present disclosure. As shown in the figure, the device of this embodiment can include: Figure 5

[0104] The first data acquisition unit 300 is configured to acquire a sparse layout guide condition corresponding to a target video, a text information feature, and an implicit space feature of a preset noise, wherein the sparse layout guide condition includes an implicit space feature of a sparse layout image sequence corresponding to the target video, and the sparse layout image sequence includes layout information of the target video.

[0105] The video generation unit 310 is configured to input the sparse layout guide condition, the text information feature, and the implicit space feature of the preset noise into a video generation model for denoising processing at a preset time step to obtain the target video. The video generation model includes an interpolation module, an adapter module, and a video generation module. In the denoising processing at each time step: the interpolation module performs interpolation processing on the sparse layout guide condition to obtain a dense layout guide condition, and performs fusion processing on the dense layout guide condition to obtain a first layout feature, wherein the dense layout guide condition includes an implicit space feature of a dense layout image sequence, and the dense layout image sequence includes the sparse layout image sequence and an interpolation image obtained through interpolation processing; the adapter module performs guide regulation processing on the first layout feature to output a second layout feature; and the video generation module obtains an intermediate feature based on the text information feature, the second layout feature, and the implicit space feature of the preset noise. The target video is determined based on the denoising processing at each time step and the intermediate feature at each time step.

[0106] In some possible implementation manners of the present disclosure, the interpolation module in the embodiments of the present disclosure performs interpolation processing on the sparse layout guide condition, and is further configured to: perform interpolation processing on the sparse layout guide condition based on a frame rate of the target video to obtain the dense layout guide condition, wherein the frame rate of the dense layout image sequence is the same as the frame rate of the target video.

[0107] In some possible implementation manners of the present disclosure, the sparse layout guide condition is acquired in the embodiments of the present disclosure, and is further configured to: perform image feature extraction on the sparse layout image sequence to obtain an image feature corresponding to the sparse layout image sequence; and perform implicit space mapping processing on the image feature corresponding to the sparse layout image sequence to obtain the sparse layout guide condition.​

[0108] In some possible implementation manners of the present disclosure, the text information feature in the embodiment of the present disclosure is further used to: obtain text information corresponding to the target video; and perform feature extraction on the text information by using a text encoder to obtain the text information feature.

[0109] In some possible implementation manners of the present disclosure, the fusion processing of the dense layout guide condition to obtain the first layout feature in the embodiment of the present disclosure is further used to: perform feature fusion processing on the latent space feature of the dense layout guide condition to obtain a layout fusion feature; and perform feature alignment processing on the layout fusion feature and the current feature in the video generation module based on the noise information of the current time step in the video generation model to obtain the first layout feature.

[0110] In some possible implementation manners of the present disclosure, the adapter module in the embodiment of the present disclosure includes n first network layers, the video generation module includes n second network layers and n third network layers, and the n first network layers and the n second network layers correspond to each other in a one-to-one manner.

[0111] The first layout feature is guided and regulated by the adapter module to output a second layout feature, and the video generation module obtains an intermediate feature based on the text information feature, the second layout feature and the latent space feature of the preset noise, and the method further includes:

[0112] Each first network layer performs feature extraction processing on the first layout feature in sequence to obtain a second layout feature, and the second layout feature includes a first vector output by each first network layer.

[0113] For each second network layer, the first vector output by the first network layer corresponding to the second network layer is input into the second network layer, the second network layer performs feature fusion based on the input first vector, the latent space feature of the preset noise and the text information feature to obtain a second vector, and determines the intermediate fusion feature based on the second vector output by the nth second network layer.

[0114] The n third network layers determine the intermediate feature based on the intermediate fusion feature output by the nth second network layer.

[0115] In some possible implementation manners of the present disclosure, the video generation model in the embodiment of the present disclosure further includes an initialization module.

[0116] Before the first vector output by the first network layer corresponding to the second network layer is input into the second network layer, the method further includes:

[0117] The first vectors are input into the initialization module, and data distribution of the first vectors is adjusted by the initialization module to align the data distribution of the first vectors with the data distribution in the video generation module.

[0118] In some possible implementation manners of the present disclosure, the layout-controllable video generation apparatus in the embodiments of the present disclosure further includes:

[0119] The second sample obtaining unit is configured to obtain a to-be-trained model and sample data, wherein the sample data includes a plurality of sample videos, and label noise, latent space features and label text information features corresponding to each sample video, the label noise of each sample video is obtained by adding noise to the sample video; and the to-be-trained model includes an interpolation module, a to-be-trained adapter module, an initialization module and a video generation module.

[0120] The layout image selecting unit is configured to obtain a label sparse layout image sequence from the sample video based on difference information between adjacent two video images in the sample video.

[0121] The guidance condition determining unit is configured to determine a label sparse layout guidance condition based on the label sparse layout image sequence, wherein the label sparse layout guidance condition includes latent space features of each label layout image in the label sparse layout image sequence.

[0122] The first training unit is configured to input the label noise, the latent space features, the label text information features and the label sparse layout guidance condition corresponding to the sample video into the to-be-trained model for noise adding processing at a preset time step to obtain a predicted noise, wherein in the noise adding processing at each time step: the interpolation module performs interpolation processing and feature fusion processing on the label sparse layout guidance condition to obtain label first layout features, the to-be-trained adapter module performs guidance regulation processing on the label first layout features to output label second layout features, the initialization module adjusts data distribution of the label second layout features, and the video generation module processes the label text information features, the label second layout features after data distribution adjustment and the latent space features of the sample video to obtain label intermediate features, so as to determine the preset noise based on the noise adding processing at each time step and the label intermediate features at each time step.

[0123] The second training unit is configured to adjust parameters of the to-be-trained adapter module according to each predicted noise and a corresponding label noise until a preset training end condition is met, and obtain the video generation model by the to-be-trained model.

[0124] The layout-controllable video generation apparatus of the embodiments of the present disclosure corresponds to the embodiments of the layout-controllable video generation method of the present disclosure, and the related content can be referred to each other, which will not be repeated here.

[0125] The exemplary embodiments of the layout-controllable video generation apparatus of the embodiments of the present disclosure can have the corresponding beneficial technical effects as described above in the corresponding exemplary method part, which will not be repeated here.

[0126] In addition, the embodiments of the present disclosure also provide an electronic device, comprising:

[0127] a memory configured to store a computer program;

[0128] a processor configured to execute the computer program stored in the memory, and when the computer program is executed, the layout-controllable video generation method according to any one of the embodiments of the present disclosure is implemented.

[0129] Figure 6 The structure schematic diagram of an application embodiment of the electronic device of the present disclosure is shown in the following figure. Figure 6 The electronic device according to the embodiments of the present disclosure will be described below with reference to

[0130] As shown in Figure 6 , the electronic device comprises one or more processors and a memory.

[0131] The processor can be a central processing unit (CPU) or other forms of processing units having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0132] The memory can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor can run the program instructions to implement the layout-controllable video generation method of the embodiments of the present disclosure described above and / or other desired functions.

[0133] In one example, the electronic device can further include an input device and an output device, which components are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0134] In addition, the input device can further include, for example, a keyboard, a mouse, and the like.

[0135] The output device can output various information including the determined distance information, direction information, and the like to the outside. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0136] Of course, in order to simplify, Figure 6 In FIG. 1, only some of the components of the electronic device related to the present disclosure are illustrated, and components such as a bus, an input / output interface, and the like are omitted. In addition thereto, the electronic device can further include any other appropriate components according to a specific application.

[0137] In addition to the above-described method and device, an embodiment of the present disclosure can be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the layout-controllable video generation method according to various embodiments of the present disclosure described in the above parts of the specification.

[0138] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0139] In addition, an embodiment of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform steps of the layout-controllable video generation method according to various embodiments of the present disclosure described in the above parts of the specification.

[0140] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0141] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0142] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0143] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0144] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0145] The methods and apparatus of this disclosure can be implemented in a number of ways. For example, the methods and apparatus of this disclosure can be implemented using software, hardware, firmware, or any combination of these methods. The order of any steps described above is merely exemplary and the steps of the methods of this disclosure need not be performed in the order described unless otherwise specified. Furthermore, any steps described above can be combined, reordered, or otherwise altered unless otherwise specified. Moreover, in some embodiments, the disclosure can also be implemented as a program for running on a computer or other programmable apparatus to implement the methods as described above. Thus, the disclosure also covers record mediums storing programs for implementing the methods of this disclosure.

[0146] It is also important to note that the methods of this disclosure can be implemented by a computer or other programmable apparatus as described above. Accordingly, the disclosure also covers computer programs stored on computer-readable media, which, when executed, perform the methods of this disclosure.

[0147] The above description of disclosed aspects is intended to be illustrative, and not restrictive. Many embodiments of the present disclosure will be readily apparent to those skilled in the art having the benefit of this disclosure before them. Embodiments of the present disclosure can be implemented in the absence of any element not specifically disclosed. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is therefore intended that the scope of the present disclosure should not be limited to the particular examples disclosed herein, but should be understood to cover all possible embodiments falling within the scope of the present disclosure.

[0148] The above description has been presented for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of skill in the art will recognize, or be able to ascertain using no more than routine experimentation, numerous modifications, changes, substitutions, equivalents, and alternative aspects and embodiments of the provisioned aspects.

Claims

1. A layout-controllable video generation method, characterized in that: include: Obtaining a sparse layout guiding condition, text information features, and latent space features of preset noise corresponding to a target video, wherein the sparse layout guiding condition includes latent space features of a sparse layout image sequence corresponding to the target video, and the sparse layout image sequence includes layout information of the target video; Inputting the sparse layout guiding condition, the text information feature, and the latent space feature of the preset noise into a video generation model for denoising at a preset time step to obtain the target video; the video generation model includes: an insertion module, an adapter module, and a video generation module; In the denoising process at each time step: the interpolation module performs interpolation processing on the sparse layout guiding condition to obtain a dense layout guiding condition, and performs fusion processing on the dense layout guiding condition to obtain a first layout feature, wherein the dense layout guiding condition includes the latent space feature of the dense layout image sequence, the dense layout image sequence includes the sparse layout image sequence and the interpolation image obtained by interpolation processing, the interpolation module inserts the latent space feature of the interpolation image into the latent space features of the multiple layout images included in the sparse layout image sequence; the adapter module performs interpolation processing on the first layout The features are guided and regulated to output a second layout feature; the video generation module obtains an intermediate feature based on the text information feature, the second layout feature and the latent space feature of the preset noise, wherein the video generation module denoises the latent space feature of the preset noise based on the text information feature and the second layout feature to obtain the intermediate feature, and the intermediate feature of each time step is used as the latent space feature of the preset noise of the next time step to perform denoising processing on the next time step; based on the denoising processing of each time step and the intermediate feature of each time step, the target video is determined.

2. The method according to claim 1, characterized in that The interpolation module performs interpolation processing on the sparse layout guiding condition, including: The interpolation module performs interpolation processing on the sparse layout guiding condition based on the frame rate of the target video to obtain the dense layout guiding condition, wherein the frame rate of the dense layout image sequence is the same as the frame rate of the target video.

3. The method according to claim 1, characterized in that Obtaining the sparse layout guiding condition includes: Performing image feature extraction on the sparse layout image sequence to obtain image features corresponding to the sparse layout image sequence; Performing latent space mapping processing on image features corresponding to the sparse layout image sequence to obtain the sparse layout guiding condition.

4. The method according to claim 1, wherein Acquiring the text information features includes: Obtaining text information corresponding to the target video; A text encoder is used to extract features from the text information to obtain features of the text information.

5. The method according to claim 1, wherein The fusing process of the dense layout guiding conditions to obtain a first layout feature includes: Performing feature fusion processing on the latent space features of the dense layout guiding condition to obtain layout fusion features; Based on the noise information of the current time step in the video generation model, the layout fusion feature is aligned with the current feature in the video generation module to obtain the first layout feature, and the current feature represents the feature corresponding to the latent space feature of the preset noise after denoising processing at the current time step.

6. The method according to any one of claims 1 to 5, characterized in that: The adapter module includes n first network layers, the video generation module includes n second network layers and n third network layers, and the n first network layers correspond one to one with the n second network layers; The adapter module guides and regulates the first layout feature and outputs a second layout feature; The video generation module obtains intermediate features based on the text information feature, the second layout feature, and the latent space feature of the preset noise, including: Each first network layer sequentially performs feature extraction processing on the first layout feature to obtain a second layout feature, wherein the second layout feature includes a first vector outputted by each first network layer; For each second network layer, inputting the first vector output by the first network layer corresponding to the second network layer into the second network layer, performing feature fusion on the second network layer based on the input first vector, the latent space feature of the preset noise, and the text information feature to obtain a second vector, and determining an intermediate fusion feature based on the second vector output by the nth second network layer; The n third network layers determine the intermediate features based on the intermediate fusion features output by the nth second network layer.

7. The method according to claim 6, characterized in that The video generation model also includes an initialization module; Before inputting the first vector output by the first network layer corresponding to the second network layer into the second network layer, the method further includes: The first vectors are input into the initialization module, which adjusts the data distribution of the first vectors so that the data distribution of the first vectors is aligned with the data distribution in the video generation module, where the data distribution is used to describe the dispersion of data in different value ranges.

8. The method according to claim 1, characterized in that The video generation model is obtained in the following way: Obtaining a model to be trained and sample data, wherein the sample data includes multiple sample videos, and label noise, latent space features, and label text information features corresponding to each sample video, wherein the label noise of each sample video is obtained by adding noise to the sample video; the model to be trained includes an interpolation module, an adapter module to be trained, an initialization module, and a video generation module; Based on difference information between two adjacent frames of video images in the sample video, obtaining a label sparse layout image sequence from the sample video; Determining a label sparse layout guiding condition based on the label sparse layout image sequence, wherein the label sparse layout guiding condition includes latent space features of each label layout image in the label sparse layout image sequence; The label noise, latent space features, label text information features and label sparse layout guiding conditions corresponding to the sample video are input into the model to be trained for noise addition processing at a preset time step to obtain predicted noise, wherein, in the noise addition processing at each time step: the interpolation module performs interpolation processing and feature fusion processing on the label sparse layout guiding conditions to obtain a first label layout feature, the adapter module to be trained performs guidance and regulation processing on the first label layout feature to output a second label layout feature, the initialization module adjusts the data distribution of the second label layout feature, and the video generation module processes the label text information features, the second label layout features after data distribution adjustment and the latent space features of the sample video to obtain a label intermediate feature, so as to determine the predicted noise based on the noise addition processing at each time step and the label intermediate feature at each time step; The parameters of the adapter module to be trained are adjusted according to each predicted noise and the corresponding label noise until a preset training end condition is met, and the video generation model is obtained from the model to be trained.

9. A layout-controllable video generation device, characterized in that: include: a first data acquisition unit, configured to acquire a sparse layout guidance condition, text information features, and latent space features of preset noise corresponding to a target video, wherein the sparse layout guidance condition includes latent space features of a sparse layout image sequence corresponding to the target video, and the sparse layout image sequence includes layout information of the target video; A video generation unit is used to input the sparse layout guiding condition, the text information feature and the latent space feature of the preset noise into a video generation model for denoising at a preset time step to obtain the target video; the video generation model includes: an interpolation module, an adapter module and a video generation module; in the denoising process at each time step: the interpolation module interpolates the sparse layout guiding condition to obtain a dense layout guiding condition, and performs fusion processing on the dense layout guiding condition to obtain a first layout feature, wherein the dense layout guiding condition includes the latent space feature of a dense layout image sequence, the dense layout image sequence includes the sparse layout image sequence and the interpolation image obtained by interpolation processing, and the interpolation module performs interpolation processing on the sparse layout guiding condition to obtain a dense layout guiding condition, and performs fusion processing on the dense layout guiding condition to obtain a first layout feature, wherein the dense layout guiding condition includes the latent space feature of a dense layout image sequence, the dense layout image sequence includes the sparse layout image sequence and the interpolation image obtained by interpolation processing, and the interpolation module performs interpolation processing on the sparse layout guiding condition to obtain a dense layout guiding condition. The latent space features of the interpolated image are inserted into the latent space features of multiple layout images included in the layout image sequence; the adapter module guides and regulates the first layout feature and outputs the second layout feature; the video generation module obtains the intermediate feature based on the text information feature, the second layout feature and the latent space feature of the preset noise, wherein the video generation module denoises the latent space feature of the preset noise based on the text information feature and the second layout feature to obtain the intermediate feature, and the intermediate feature of each time step is used as the latent space feature of the preset noise of the next time step to perform denoising processing on the next time step; based on the denoising processing of each time step and the intermediate feature of each time step, the target video is determined.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images