Video generation method and device, electronic equipment, storage medium and product

Abstract

Embodiments of the present disclosure disclose a video generation method and device, electronic equipment, storage medium and product, wherein the method comprises: obtaining condition information for guiding video generation; processing the condition information based on a first adapter model and a video model to obtain a second video segment corresponding to the condition information. Embodiments of the present disclosure can generate a video segment with high matching degree to the condition information for guiding video generation.

Description

Technical Field

[0001] This disclosure relates to video generation technology, and in particular to a method, apparatus, electronic device, storage medium, and product for generating video. Background Technology

[0002] With the rapid development of video generation model technology, video generation models have shone brightly in the field of Artificial Intelligence Generated Content (AIGC).

[0003] In related technologies, natural language processing is performed on the text description information input by the user to obtain corresponding feature information. A video generation model is then used to process this feature information to generate a video corresponding to the text description. However, because the text description input by the user cannot fully express the specific requirements for generating the video, or because the accuracy of the video generation model in generating the corresponding video based on the text description is insufficient, the videos generated by the video generation model often fail to meet the user's needs.

[0004] How to generate videos that highly match the text description information is a problem that urgently needs to be solved. Summary of the Invention

[0005] This disclosure provides a video generation method, apparatus, electronic device, storage medium, and product to solve the above-mentioned problems.

[0006] A first aspect of this disclosure provides a video generation method, comprising:

[0007] Obtain condition information for guiding video generation, wherein the condition information includes at least: a first image sequence corresponding to a first video segment, the first image sequence including: at least one depth map corresponding to at least one video frame of the first video segment, and pixel value processing images corresponding to the remaining video frames in the first video segment other than the at least one video frame;

[0008] The conditional information is processed based on the first adapter model and the video model to obtain a second video segment corresponding to the conditional information, wherein the first adapter model is obtained by joint training based on the second adapter model and the video model.

[0009] In some embodiments of this disclosure, obtaining the conditional information for generating the guiding video includes:

[0010] Depth processing is performed on each video frame in the first video segment to obtain multiple depth maps corresponding to each video frame;

[0011] At least one depth map is selected from the plurality of depth maps for pixel value processing to obtain an image with at least one pixel value processed;

[0012] Based on the temporal relationship of each video frame in the first video segment, the at least one depth map and the at least one pixel value processed image are sorted to obtain the first image sequence.

[0013] In some embodiments of this disclosure, selecting at least one depth map from the plurality of depth maps for pixel value processing includes:

[0014] Select at least one depth map from the plurality of depth maps;

[0015] Adjust all pixel values ​​of all depth maps in the at least one depth map to preset pixel values.

[0016] In some embodiments of this disclosure, the condition information for guiding video generation further includes: text description information corresponding to the second video segment.

[0017] In some embodiments of this disclosure, the second adapter model and the video model have the same input layer structure;

[0018] The process of processing the conditional information based on the first adapter model and the video model to obtain the second video segment corresponding to the conditional information includes:

[0019] The first image sequence is processed based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model;

[0020] At each time step, the output vectors of each downsampling layer of the first adapter model are weighted and calculated with the output vectors of the corresponding downsampling layer of the video model to obtain the weighted calculation results of the output vectors of each sampling layer. The weighted calculation results of the output vectors of each sampling layer are used as the input of the next sampling layer of the video model to obtain the second video segment.

[0021] In some embodiments of this disclosure, the step of weighting the output vectors of each downsampling layer of the first adapter model with the output vectors of the corresponding downsampling layer of the video model to obtain the weighted calculation result of the output vectors of each sampling layer, and using the weighted calculation result of the output vectors of each sampling layer as the input of the next sampling layer corresponding to the video model, includes:

[0022] Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is the output vector of the first downsampling layer of the video model;

[0023] The first video output vector and the output vector of the corresponding layer of the first adapter are weighted and calculated to obtain the first vector, wherein the output vector of the first video output vector of the corresponding layer of the first adapter is the output vector of the first downsampling layer of the first adapter model;

[0024] The first vector is used as the input to the second downsampling layer of the video model to obtain the second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model;

[0025] The Nth vector is used as the input of the (N+1)th downsampling layer of the video model to obtain the output vector of the (N+1)th downsampling layer of the video model. The Nth vector is calculated by weighting the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, where N is an integer greater than 1.

[0026] In some embodiments of this disclosure, before processing the condition information based on the first adapter model and the video model to obtain the second video segment corresponding to the condition information, the method further includes:

[0027] Obtain sample video clips and sample text descriptions;

[0028] The second image sequence corresponding to the sample video segment is processed based on the second adapter model to obtain the output vector of each downsampling layer of the second adapter model;

[0029] Based on the video model, the sample text description information, the output vectors of each intermediate layer of the second adapter model, and the second Gaussian noise matrix are processed to obtain the third video segment;

[0030] By fixing the parameters of the video model, and adjusting the parameters of the second adapter model based on the differences between the third video segment and the sample video segment, the first adapter model is obtained.

[0031] A second aspect of this disclosure provides a video generation apparatus, comprising:

[0032] An information acquisition module is used to guide the condition information for video generation, wherein the condition information includes at least: a first image sequence corresponding to a first video segment, the first image sequence including: at least one depth map corresponding to at least one video frame of the first video segment, and pixel value processing images corresponding to the remaining video frames in the first video segment other than the at least one video frame;

[0033] The video generation module is used to process the conditional information based on the first adapter model and the video model to obtain a second video segment corresponding to the conditional information, wherein the first adapter model is obtained by jointly training the second adapter model and the video model.

[0034] In some embodiments of this disclosure, the information acquisition module is used to perform depth processing on each video frame in the first video segment to obtain multiple depth maps corresponding to each video frame;

[0035] The information acquisition module is also used to select at least one depth map from the plurality of depth maps for pixel value processing to obtain at least one pixel value processed image;

[0036] The information acquisition module is further configured to sort the at least one depth map and the at least one pixel value processed image based on the temporal relationship of each video frame in the first video segment to obtain the first image sequence.

[0037] In some embodiments of this disclosure, the information acquisition module is used to select at least one depth map from the plurality of depth maps;

[0038] The information acquisition module is also used to adjust all pixel values ​​of all depth maps in the at least one depth map to preset pixel values.

[0039] In some embodiments of this disclosure, the condition information for guiding video generation further includes: text description information corresponding to the second video segment.

[0040] In some embodiments of this disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of both the first adapter model and the video model includes multiple downsampling layers;

[0041] The video generation module is used to process the first image sequence based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model;

[0042] The video generation module is further configured to, at each time step, perform weighted calculations on the output vectors of each downsampling layer of the first adapter model and the output vectors of the corresponding downsampling layer of the video model to obtain the weighted calculation results of the output vectors of each sampling layer, and use the weighted calculation results of the output vectors of each sampling layer as the input of the next sampling layer of the video model to obtain the second video segment.

[0043] In some embodiments of this disclosure, the video generation module is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is the output vector of the first downsampling layer of the video model;

[0044] The video generation module is further configured to perform a weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector, wherein the first adapter output vector is the output vector of the first downsampling layer of the first adapter model;

[0045] The video generation module is further configured to use the first vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model.

[0046] The video generation module is further configured to use the Nth vector as the input of the (N+1)th downsampling layer of the video model to obtain the output vector of the (N+1)th downsampling layer of the video model, wherein the Nth vector is calculated by weighting the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0047] In some embodiments of this disclosure, the apparatus further includes:

[0048] The model training module is used to acquire sample video clips and sample text description information;

[0049] The model training module is further configured to process the second image sequence corresponding to the sample video segment based on the second adapter model to obtain the output vector of each downsampling layer of the second adapter model. The second image sequence includes: at least one depth map corresponding to at least one video frame image of the sample video segment in time sequence, and at least one pixel value processing image corresponding to the remaining video frame images in the sample video segment in time sequence other than the at least one video frame image.

[0050] The model training module is also used to process the sample text description information, the output vectors of each intermediate layer of the second adapter model, and the second Gaussian noise matrix based on the video model to obtain a third video segment.

[0051] The model training module is also used to fix the parameters of the video model and adjust the parameters of the second adapter model based on the difference between the third video segment and the sample video segment to obtain the first adapter model.

[0052] A third aspect of this disclosure provides an electronic device, comprising:

[0053] Memory, used to store computer program products;

[0054] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0055] A fourth aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect above.

[0056] The video generation method, apparatus, electronic device, storage medium, and product of this disclosure, since the adapter model, in conjunction with the video model, can improve the matching degree between the generated video segments and the video generation conditions, uses the conditional information guiding video generation as model input. By processing the conditional information guiding video generation through the first adapter model and the video model, video segments with a high degree of matching with the conditional information guiding video generation can be generated. Since the conditional information guiding video generation includes a partial depth map with depth information and a pixel value processing image used to enhance the model's ability to generate video frames with depth information, the generated video segments possess depth information.

[0057] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0058] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0059] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0060] Figure 1 This is a flowchart of a video generation method in one embodiment of the present disclosure;

[0061] Figure 2 This is a partial flowchart of step S1 in some embodiments of this disclosure;

[0062] Figure 3 This is a flowchart illustrating step S2 in some embodiments of this disclosure;

[0063] Figure 4 This is a schematic diagram of the workflow of each time step in step S2-2 in some embodiments of this disclosure;

[0064] Figure 5This is a flowchart illustrating step S0 in some embodiments of this disclosure;

[0065] Figure 6 This is a structural block diagram of a video generation apparatus in one embodiment of the present disclosure;

[0066] Figure 7 This is a structural block diagram of a video generation apparatus in another embodiment of the present disclosure;

[0067] Figure 8 This is a structural block diagram of an electronic device in one embodiment of the present disclosure. Detailed Implementation

[0068] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0069] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0070] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0071] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0072] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0073] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0074] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0075] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0076] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0077] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0078] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0079] Figure 1 This is a flowchart of a video generation method in one embodiment of this disclosure. Figure 1 As shown, the video generation method may include the following steps:

[0080] S1: Obtain the condition information for generating the guide video.

[0081] The conditional information guiding video generation includes at least: a first image sequence corresponding to the first video segment. The first image sequence includes: at least one depth map corresponding to at least one video frame of the first video segment, and pixel value processed images corresponding to the remaining video frames in the first video segment excluding the at least one video frame.

[0082] In optional embodiments of this disclosure, the user-input video generation conditions may include a first video segment. A first image sequence can be obtained by preprocessing the first video segment, and this first image sequence can then be used as at least one piece of information in the conditional information guiding video generation.

[0083] In optional embodiments of this disclosure, the process of preprocessing the first video segment to obtain the first image sequence may include:

[0084] A depth map generation model is used to transform the depth maps of each video frame in the first video segment, resulting in a first image sequence that corresponds one-to-one with the temporal relationship of each video frame in the first video segment. Each depth map in the first image sequence contains information related to the distance to the surface of scene objects from the viewpoint, directly reflecting the geometry of the visible surfaces of scene objects. Depth map visualization clearly demonstrates the operating principle of the visual attention mechanism. In this embodiment, using the first image sequence corresponding to the first video segment as conditional information or one of the types of information guiding video generation can better guide the video model to generate video segments that meet user expectations.

[0085] At least one depth image is determined from the first image sequence, and the remaining depth images in the first image sequence other than the at least one depth image are processed for pixel values ​​to obtain the corresponding pixel-value-processed images. Alternatively, at least one depth image can be randomly selected from the first image sequence for pixel value processing to obtain the pixel-value-processed image corresponding to the at least one depth image, while the remaining at least one depth image is not processed.

[0086] S2: Based on the first adapter model and the video model, the conditional information is processed to obtain a second video segment corresponding to the conditional information. The second video segment has visual content of at least one depth map corresponding to the first video segment.

[0087] The first adapter model is obtained through joint training of the second adapter model and the video model. The second adapter model is an untrained adapter model, also known as the initial adapter model. The first adapter model is the adapter model obtained through joint training, also known as the final adapter model. Optionally, in the process of obtaining the first adapter model through joint training of the second adapter model and the video model, the second adapter model and the video model are first obtained. The network layer structure of the second adapter model is the same as the network structure of the input layer of the video model. Each network layer of the second adapter model corresponds one-to-one with a network layer of the input layer of the video model, that is, the number of network layers in the second adapter model is the same as the number of network layers in the input layer of the video model, and the network parameters of each network layer of the second adapter model are the same as those of the corresponding network layer in the input layer of the video model. Then, the network parameters of each network layer of the second adapter model are adjusted using the loss function of the video model to obtain the first adapter model.

[0088] In this embodiment of the invention, the first adapter model is obtained by jointly training the second adapter model and the video model. This avoids directly adjusting the parameters of the video model (which typically has a large number of parameters, such as billions or tens of billions). This not only reduces the number of parameters required for fine-tuning but also reduces the computational resource requirements during training, thus accelerating the training process. Furthermore, when the video model performs a specific task, the trained first adapter model is used to fine-tune the output vector of the input layer of the lower video model. This allows for efficient and accurate completion of specific tasks without requiring large-scale overall training of the video model. Therefore, by jointly training the second adapter model and the video model to obtain the first adapter model, high-quality second video segments with visual content corresponding to the first image sequence are quickly generated using AIGC technology, based on conditional information including the first image sequence corresponding to the first video segment, without affecting the versatility of the video model.

[0089] The video model can be a neural network model based on a diffusion model, used to generate corresponding videos based on input data, including text, images, or audio. Audio can be converted into text through speech recognition and input into the video model. The video model can be pre-trained, for example, based on input samples including input text descriptions and input videos.

[0090] Furthermore, the diffusion model is a generative model based on deep learning algorithms. It generates high-quality content through a process of progressively adding and removing noise, demonstrating excellent performance in generating images, videos, audio, and other high-dimensional data, as well as strong performance in generation tasks. In this embodiment, the diffusion model is a pre-trained model. After the encoder maps the conditional information to the latent space representation, guided by the latent space representation of the conditional information, it performs denoising processing on random noise for T time steps, i.e., downsampling and upsampling processing at each time step, to obtain the latent space representation of the final second video segment. Then, the decoder maps the latent space representation of the target video to the pixel space to obtain the final second video segment. Here, the latent space representation of the conditional information is the latent space representation obtained after encoding the conditional information using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space, obtaining an implicit and continuous representation, which makes the model computation more efficient. The decoder can reconstruct the video from this latent space representation, mapping the generation result of the diffusion model from the latent space to the pixel space. In some implementations, the encoder and decoder used in the diffusion model of this disclosure can both be variational autoencoders (VAEs). The latent space of a VAE is continuous, which is suitable for generating continuous data, so that the generated target videos or images are correlated.

[0091] The adapter model is a neural network-based model used to fine-tune the parameters of the generated video based on the video generation task. It achieves this by adding small, trainable modules, rather than retraining the video model.

[0092] The conditional information for guiding video generation can be processed using a first adapter model to obtain the output vector of the first adapter model. The output vector of the first adapter model is then processed using a video model to obtain a second video segment. This second video segment contains visual content from at least one depth map corresponding to the first video segment, which facilitates its application in scenarios such as games or movies.

[0093] In this embodiment, since the adapter model, in conjunction with the video model, can improve the matching degree between the generated video clips and the video generation conditions, the conditional information guiding video generation is used as the model input. By processing this conditional information through the first adapter model and the video model, video clips with a high degree of matching with the conditional information guiding video generation can be generated. Because the conditional information guiding video generation includes a partial depth map with depth information—that is, the first image sequence contains depth maps of sparsely distributed video frames—this not only enhances the model's ability to generate visual content corresponding to depth information but also enables the generation of rich video products using sparsely distributed depth maps, meeting user creative needs. Therefore, users can adaptively generate high-quality second video clips using any video frame in the first video clip and the corresponding depth map information.

[0094] Figure 2 This is a partial flowchart illustrating step S1 in some embodiments of this disclosure. For example... Figure 2 As shown, step S1 may include the following steps:

[0095] S1-1: Perform depth processing on each video frame in the first video segment to obtain multiple depth maps corresponding to each video frame.

[0096] Based on the intrinsic and extrinsic parameters of the video image acquisition device in each video frame of the first video segment, depth estimation can be performed on each object in each video frame of the first video segment, and multiple depth maps corresponding to each video frame can be generated based on the depth estimation results of each object.

[0097] S1-2: Select at least one depth map from multiple depth maps and process its pixel values ​​to obtain an image with at least one processed pixel value.

[0098] A subset of depth maps can be selected randomly from multiple depth maps. For example, a preset probability value can be used to determine whether to process the pixel values ​​of each depth map to obtain an image with at least one processed pixel value. For instance, the preset probability value can be a value between 10% and 20%.

[0099] For all randomly selected depth maps, pixel values ​​can be processed using the same pixel processing method. For example, the pixel values ​​of pixels at predetermined coordinate positions in all randomly selected depth maps can be adjusted to the pixel values ​​corresponding to black, RGB(0,0,0).

[0100] S1-3: Based on the temporal relationship of each video frame in the first video segment, sort at least one depth map and at least one pixel value processed image to obtain the first image sequence.

[0101] Each video frame can be numbered according to its temporal relationship within the first video segment. For example, if the first video segment is 3 seconds long and each second of video consists of 16 video frames, then the video frame number could be: the first frame of the first second of the video could be numbered NA. 1-1 The second frame of the first second of the video is labeled NA. 1-2 ..., the first frame of the 2-second video is labeled NA. 2-1 The image at 3 seconds and 16 frames is labeled NA. 3-16 .

[0102] The depth map or pixel value processing image corresponding to each video frame can be labeled accordingly. For example, the depth map or pixel value processing image corresponding to the first frame of the first second of the video is numbered NB. 1-1 The image corresponding to the second frame of the first second of the video is labeled NB, which is the depth map or pixel value processed image. 1-2 ..., the image corresponding to the first frame of the 2nd second of the video, with the depth map or pixel value processing image numbered NB. 2-1 The image corresponding to the depth map or pixel value processing image at the 3rd second and 16th frame is labeled NB. 3-16 .

[0103] The first image sequence can be obtained by sorting at least one depth map and at least one pixel value processed image according to the number of the depth map or pixel value processed image corresponding to each video frame.

[0104] In this embodiment, depth processing, pixel value processing, and sorting are performed on each video frame in the first video segment to quickly and accurately obtain a first image sequence containing a sparsely distributed depth map. This improves the applicability of users to generate high-quality videos using the depth map information corresponding to any video frame in the first video segment, expands the application field of AIGC technology, and enhances the fun of AIGC technology applications.

[0105] In some embodiments of this disclosure, selecting at least one depth map from a plurality of depth maps for pixel value processing may include: selecting at least one depth map from a plurality of depth maps; and adjusting all pixel values ​​of all depth maps in the at least one depth map to preset pixel values.

[0106] For each depth map in the at least one depth map, the pixel values ​​of all pixels in each depth map can be adjusted to the pixel values ​​corresponding to black, RGB(0,0,0).

[0107] In this embodiment, all pixel values ​​can be processed quickly using the same pixel value adjustment method.

[0108] In some embodiments of this disclosure, the conditional information guiding the generation of the video also includes: text description information corresponding to the second video segment.

[0109] Accordingly, the output vector of the first adapter model and the text description information can be used as input to the video model, and the video model can be used to output a second video segment. The text description information is used to characterize the visual content information of the final generated video. Furthermore, the text description information can be used to guide the processing operations of the visual content in the first image sequence. For example, if the first image sequence includes the depth map of a video frame of a cat and the depth map of a video frame of a dog, the text description information could be "The cat and the dog ran together towards a grassy area."

[0110] In this embodiment, textual description information can be used as the basic condition for generating the video, and the first video segment or the first image sequence can be used as an auxiliary constraint for generating the video. A second video segment with a high degree of matching between the first video segment or the first image sequence and the textual description information is generated using a first adapter model and a video model. As a general video generation model, the video model has relatively strong text understanding capabilities. In this embodiment, by using textual description information as input to the video model, it can be ensured that the generated second video segment better meets the user's expectations.

[0111] In some embodiments of this disclosure, the input layer structures of the first adapter model and the video model are identical. Specifically, both the input layer of the first adapter model and the input layer of the video model include multiple downsampling layers. By using input layers with identical network structures, feature vectors of the same dimension can be extracted based on the conditional information guiding video generation.

[0112] Figure 3 This is a flowchart illustrating step S2 in some embodiments of this disclosure. For example... Figure 3 As shown, step S2 includes the following steps:

[0113] S2-1: Process the first image sequence based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model.

[0114] The first adapter model may include N+1 downsampling layers, where N is an integer greater than 1. The output vectors of the first downsampling layer to the (N+1)th downsampling layer are stored in order from 1 to N+1, denoted as vector VA1 to vector VA2. N+1 .

[0115] S2-2: At each time step, the output vectors of each downsampling layer of the first adapter model are weighted and calculated with the output vectors of the corresponding downsampling layer of the video model to obtain the weighted calculation result of the output vectors of each sampling layer. The weighted calculation result of the output vectors of each sampling layer is used as the input of the next sampling layer of the video model to obtain the second video segment.

[0116] Noise can be added to the video model, and the video model can be used to gradually denoise based on the output vectors of each downsampling layer of the first adapter model, and finally generate the second video segment.

[0117] In each denoising step, the output vectors of each downsampling layer of the first adapter model are weighted and calculated with the output vectors of the corresponding downsampling layers of the video model to obtain the output vector of the video model's input layer. The video model includes an input layer and an output layer. The input layer of the video model comprises multiple downsampling layers, each mapped one-to-one with a corresponding downsampling layer of the first adapter model, with each group of corresponding downsampling layers having the same structure. The output vector of each downsampling layer of the first adapter model is used to perform a weighted calculation with the output vector of the mapped downsampling layer of the video model, serving as the input vector for the next downsampling layer of the video model.

[0118] It can be understood that one round of computation between the input and output layers of the above video model constitutes one time step of noise prediction, i.e., one time step of denoising. The video model performs T rounds of iterative denoising to obtain the final target video. After the output layer of the video model outputs the output vector of the first round, it is used as the initial input vector for the second round and input back into the input layer of the video model. Following the above process, after T rounds of denoising are completed, the decoder decodes the final output vector of the video model's output layer, mapping it from the latent space to the pixel space to obtain the second video segment.

[0119] In this embodiment, since the input layer structure of the first adapter model and the video model is the same, feature vectors of the same dimension can be extracted from the input information of their respective models, and then a second video segment that highly matches the condition information of the guiding video generation can be obtained by gradually denoising.

[0120] Figure 4 This is a flowchart illustrating step S2-2 in some embodiments of this disclosure. For example... Figure 4 As shown, step S2-2 may further include the following steps:

[0121] S2-2-1: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain the first video output vector.

[0122] By extracting features from the text description information and the first Gaussian noise matrix using the video model, a vector of predetermined dimensions can be obtained as the output vector of the first downsampling layer in the input layer of the video model, denoted as the first video output vector, and represented by vector VB1.

[0123] S2-2-2: The first video output vector and the output vector of the corresponding layer of the first adapter are weighted and calculated to obtain the first vector.

[0124] If the first video output vector is the output vector of the first downsampling layer in the input layer of the video model, then the output vector of the first adapter corresponding to the first video output vector is the output vector of the first downsampling layer in the input layer of the first adapter.

[0125] For S2-2-2, the output vector of the first downsampling layer of the first adapter model is extracted from the output vectors of each downsampling layer of the first adapter model, and denoted by vector VA1. A weighted calculation is performed on vectors VA1 and VB1, for example, by weighted summation, to obtain the first vector.

[0126] S2-2-3: The first vector is used as the input to the second downsampling layer of the video model to obtain the second video output vector. Furthermore, the second video output vector and the output vector of the second downsampling layer of the first adapter are weighted and calculated to obtain the second vector.

[0127] The first vector is used as the input to the second downsampling layer of the video model to obtain the output vector of the second downsampling layer, denoted as the second video output vector. The first and second downsampling layers of the video model are adjacent; that is, the second downsampling layer is the next downsampling layer adjacent to the first downsampling layer.

[0128] Based on processes S2-2-2 and S2-2-3, the output vector of the Nth downsampling layer of the video model, i.e., the Nth video output vector, can be obtained similarly, denoted as vector VB. N .

[0129] Extract the output vector (i.e., vector VA) of the Nth downsampling layer of the first adapter model from the output vectors of each intermediate layer of the first adapter model. N ), will vector VA N Sum of vectors VB N Perform weighted calculations, for example, for VA N Sum of vectors VB N We perform a weighted summation to obtain the Nth vector, where N is an integer greater than 1.

[0130] S2-2-4: Use the Nth vector as the input of the (N+1)th downsampling layer of the video model to obtain the output vector of the (N+1)th downsampling layer of the video model.

[0131] Using the Nth vector as the input to the (N+1)th downsampling layer of the video model, we obtain the output vector of the (N+1)th downsampling layer of the video model, such as the output vector of the last intermediate layer of the video model.

[0132] In this embodiment, since the input layer structures of the first adapter model and the video model are the same, and the dimensions of the input vectors of the downsampling layers of the first adapter model and the video model are the same, the first Gaussian noise matrix is ​​input to the video model, and the output vectors of each downsampling layer of the first adapter model and the video model are weighted and calculated to obtain a vector as the input of the next downsampling layer of the video model. This achieves denoising within a single time step, and through stepwise denoising, a second video segment that highly matches the conditions for generating the guiding video can finally be obtained.

[0133] In some embodiments of this disclosure, the following may be included before step S1:

[0134] Step S0: Based on the sample video segments and the corresponding text description information, jointly train the second adapter model and the video model to obtain the first adapter model.

[0135] Figure 5 This is a flowchart illustrating step S0 in some embodiments of this disclosure. For example... Figure 5 As shown, step S0 may include the following steps:

[0136] S0-1: Obtain sample video clips and sample text descriptions. The sample text descriptions correspond to the sample video clips. The sample text descriptions can be obtained by manually annotating the sample video clips.

[0137] S0-2: Based on the second adapter model, the second image sequence corresponding to the sample video segment is processed to obtain the output vector of each intermediate layer of the second adapter model.

[0138] A similar processing method to that used in step S1 to obtain the first image sequence can be employed to obtain the second image sequence corresponding to the sample video segment. The second image sequence includes: at least one depth map corresponding to at least one video frame of the sample video segment, and at least one pixel value processed image corresponding to the remaining video frames of the sample video segment excluding the at least one video frame.

[0139] A similar processing method as step S2-1 can be used to process the second image sequence using the first adapter model to obtain the output vectors of each intermediate layer of the first adapter model.

[0140] S0-3: Based on the video model, the sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix are processed to obtain the third video segment.

[0141] A similar approach to step S2-2 can be used to process the sample text description information, the output vectors of each intermediate layer of the first adapter model, and the second Gaussian noise matrix using the video model to obtain the third video segment.

[0142] S0-4: With the parameters of the fixed video model, the parameters of the second adapter model are adjusted based on the difference between the third video segment and the sample video segment to obtain the first adapter model.

[0143] Using a pre-defined loss function formula, the loss value is calculated based on the difference between the third video segment and the sample video segment.

[0144] With the parameters of the video model fixed, the parameters of the second adapter model are adjusted through backpropagation based on the loss value.

[0145] When the preset model training termination conditions are met, such as the number of iterations reaching a preset threshold, or the loss value calculated after a certain iteration being less than a preset loss threshold, the model training process ends, and the first adapter model is obtained.

[0146] In this embodiment, since the video model is a large model with many parameters, adjusting the parameters of the video model to improve the matching degree between the generated video and the text description information would require a lot of model training costs. Therefore, under the premise of fixing the parameters of the video model, the matching degree between the video generated by the video model and the text description information can be improved by training a matching adapter model, thereby significantly reducing the model training cost.

[0147] Figure 6 This is a structural block diagram of a video generation apparatus according to one embodiment of the present disclosure. Figure 6 As shown, the video generation apparatus includes:

[0148] The information acquisition module 100 is used to guide the condition information for video generation. The condition information includes at least: a first image sequence corresponding to the first video segment. The first image sequence includes: at least one depth map corresponding to at least one video frame of the first video segment, and pixel value processing images corresponding to the remaining video frames in the first video segment other than at least one video frame.

[0149] The video generation module 200 is used to process the conditional information based on the first adapter model and the video model to obtain the second video segment corresponding to the conditional information. The first adapter model is obtained by jointly training the second adapter model and the video model.

[0150] In some embodiments of this disclosure, the information acquisition module 100 is used to perform depth processing on each video frame in the first video segment to obtain multiple depth maps corresponding to each video frame.

[0151] The information acquisition module 100 is also used to select at least one depth map from multiple depth maps for pixel value processing to obtain at least one pixel value processed image;

[0152] The information acquisition module 100 is also used to sort at least one depth map and at least one pixel value processing image based on the temporal relationship of each video frame in the first video segment to obtain a first image sequence.

[0153] In some embodiments of this disclosure, the information acquisition module 100 is used to select at least one depth map from a plurality of depth maps;

[0154] The information acquisition module 100 is also used to adjust all pixel values ​​of all depth maps in at least one depth map to preset pixel values.

[0155] In some embodiments of this disclosure, the conditional information guiding the generation of the video also includes: text description information corresponding to the second video segment.

[0156] In some embodiments of this disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of both the first adapter model and the video model includes multiple downsampling layers.

[0157] The video generation module 200 is used to process the first image sequence based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model;

[0158] The video generation module 200 is also used to perform weighted calculations at each time step, by combining the output vectors of each downsampling layer of the first adapter model with the output vectors of the corresponding downsampling layer of the video model, to obtain the weighted calculation results of the output vectors of each sampling layer, and to use the weighted calculation results of the output vectors of each sampling layer as the input of the next sampling layer of the video model, so as to obtain the second video segment.

[0159] In some embodiments of this disclosure, the video generation module 200 is used to process text description information and a first Gaussian noise matrix based on a video model to obtain a first video output vector, wherein the first video output vector is the output vector of the first downsampling layer of the video model;

[0160] The video generation module 200 is also used to perform weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector, wherein the first adapter output vector is the output vector of the first downsampling layer of the first adapter model;

[0161] The video generation module 200 is also used to take the first vector as the input of the second downsampling layer of the video model to obtain the second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model.

[0162] The video generation module 200 is also used to take the Nth vector as the input of the (N+1)th downsampling layer of the video model to obtain the output vector of the (N+1)th downsampling layer of the video model. The Nth vector is calculated by weighting the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, where N is an integer greater than 1.

[0163] Figure 7 This is a structural block diagram of the video generation apparatus in other embodiments of this disclosure. For example... Figure 7 As shown, the video generation device also includes:

[0164] The model training module 300 is used to acquire sample video clips and sample text description information;

[0165] The model training module 300 is also used to process the second image sequence corresponding to the sample video segment based on the second adapter model to obtain the output vector of each downsampling layer of the second adapter model. The second image sequence includes: at least one depth map corresponding to at least one video frame image of the sample video segment in time sequence, and at least one pixel value processing image corresponding to the remaining video frame images in the sample video segment in time sequence except for at least one video frame image.

[0166] The model training module 300 is also used to process the sample text description information, the output vectors of each intermediate layer of the second adapter model, and the second Gaussian noise matrix based on the video model to obtain the third video segment.

[0167] The model training module 300 is also used to fix the parameters of the video model and adjust the parameters of the second adapter model to obtain the first adapter model based on the difference between the third video segment and the sample video segment.

[0168] It should be noted that the specific implementation of the video generation device in this disclosure is similar to the specific implementation of the video generation method in this disclosure, and the technical effects of the video generation device in this disclosure are similar to the technical effects of the video generation method in this disclosure. For details, please refer to the description of the video generation method section. In order to reduce redundancy, it will not be described again.

[0169] In addition, this disclosure also provides an electronic device, including:

[0170] Memory, used to store computer programs;

[0171] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the video generation method described in any of the above embodiments of the present disclosure.

[0172] Below, for reference Figure 8 To describe an electronic device according to embodiments of this disclosure. For example... Figure 8 As shown, the electronic device includes one or more processors and memory.

[0173] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0174] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the video generation methods of the various embodiments of this disclosure described above and / or other desired functions.

[0175] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0176] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0177] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0178] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0179] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the video generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0180] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0181] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the video generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0182] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0183] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0185] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0186] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0187] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0188] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0189] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating video, characterized in that, include: Obtain condition information for guiding video generation, wherein the condition information includes at least: a first image sequence corresponding to a first video segment, the first image sequence including: at least one sparsely distributed depth map corresponding to at least one video frame of the first video segment, and pixel value processing images corresponding to the remaining video frames in the first video segment other than the at least one video frame, wherein the at least one depth map is obtained by performing depth processing on the at least one video frame, and the at least one depth map is used to reflect the geometry of the visible surface of the scene object; The conditional information is processed based on the first adapter model and the video model to obtain a second video segment corresponding to the conditional information. The first adapter model is obtained by joint training of the second adapter model and the video model. The first adapter model is used to fine-tune the parameters of the generated video according to the video generation task based on the video model. The process of obtaining the first adapter model by jointly training the second adapter model and the video model includes: obtaining the second adapter model and the video model, wherein the network layer structure of the second adapter model is the same as the network structure of the input layer of the video model, and the network layers of the second adapter model correspond one-to-one with the network layers of the input layer of the video model; adjusting the network parameters of each network layer of the second adapter model through the loss function of the video model, thereby obtaining the first adapter model.

2. The method according to claim 1, characterized in that, The conditions for obtaining the guidance video generation include: Depth processing is performed on each video frame in the first video segment to obtain multiple depth maps corresponding to each video frame; At least one depth map is selected from the plurality of depth maps for pixel value processing to obtain an image with at least one pixel value processed; Based on the temporal relationship of each video frame in the first video segment, the at least one depth map and the at least one pixel value processed image are sorted to obtain the first image sequence.

3. The method according to claim 2, characterized in that, The conditional information for generating the guiding video also includes: text description information corresponding to the second video segment.

4. The method according to claim 3, characterized in that, The first adapter model and the video model have the same input layer structure; The process of processing the conditional information based on the first adapter model and the video model to obtain the second video segment corresponding to the conditional information includes: The first image sequence is processed based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model; At each time step, the output vectors of each downsampling layer of the first adapter model are weighted and calculated with the output vectors of the corresponding downsampling layer of the video model to obtain the weighted calculation results of the output vectors of each sampling layer. The weighted calculation results of the output vectors of each sampling layer are used as the input of the next sampling layer of the video model to obtain the second video segment.

5. The method according to claim 4, characterized in that, The step of weighting the output vectors of each downsampling layer of the first adapter model with the output vectors of the corresponding downsampling layer of the video model to obtain the weighted calculation result of the output vectors of each sampling layer, and using the weighted calculation result of the output vectors of each sampling layer as the input of the next sampling layer of the video model, includes: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is the output vector of the first downsampling layer of the video model; The first video output vector and the output vector of the corresponding layer of the first adapter are weighted and calculated to obtain the first vector, wherein the output vector of the first video output vector of the corresponding layer of the first adapter is the output vector of the first downsampling layer of the first adapter model; The first vector is used as the input to the second downsampling layer of the video model to obtain the second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The Nth vector is used as the input of the (N+1)th downsampling layer of the video model to obtain the output vector of the (N+1)th downsampling layer of the video model. The Nth vector is calculated by weighting the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, where N is an integer greater than 1.

6. The method according to claim 3, characterized in that, Before processing the condition information based on the first adapter model and the video model to obtain the second video segment corresponding to the condition information, the method further includes: Obtain sample video clips and sample text descriptions; The second image sequence corresponding to the sample video segment is processed based on the second adapter model to obtain the output vector of each downsampling layer of the second adapter model. The second image sequence includes: at least one depth map corresponding to at least one video frame image of the sample video segment in time sequence, and at least one pixel value processing image corresponding to the remaining video frame images in the sample video segment in time sequence other than the at least one video frame image. Based on the video model, the sample text description information, the output vectors of each intermediate layer of the second adapter model, and the second Gaussian noise matrix are processed to obtain the third video segment; By fixing the parameters of the video model, and adjusting the parameters of the second adapter model based on the differences between the third video segment and the sample video segment, the first adapter model is obtained.

7. A video generation apparatus, characterized in that, include: An information acquisition module is used to guide the condition information for video generation. The condition information includes at least: a first image sequence corresponding to a first video segment. The first image sequence includes: at least one sparsely distributed depth map corresponding to at least one video frame of the first video segment, and pixel value processing images corresponding to the remaining video frames in the first video segment other than the at least one video frame. The at least one depth map is obtained by performing depth processing on the at least one video frame. The at least one depth map is used to reflect the geometry of the visible surface of the scene object. The video generation module is used to process the conditional information based on the first adapter model and the video model to obtain a second video segment corresponding to the conditional information. The first adapter model is obtained by joint training based on the second adapter model and the video model. The first adapter model is used to fine-tune the parameters of the generated video according to the video generation task based on the video model. The process of obtaining the first adapter model by jointly training the second adapter model and the video model includes: obtaining the second adapter model and the video model, wherein the network layer structure of the second adapter model is the same as the network structure of the input layer of the video model, and the network layers of the second adapter model correspond one-to-one with the network layers of the input layer of the video model; adjusting the network parameters of each network layer of the second adapter model through the loss function of the video model, thereby obtaining the first adapter model.

8. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-6.

10. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-6.

Images