Video generation method and device, electronic equipment, storage medium and product

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: obtaining condition information for guiding the generation of a video; and processing the condition information based on a first adapter model and a video model to obtain a second video clip corresponding to the condition information. According to the embodiment of the invention, the video clip with high matching degree with the condition information for guiding video generation can be generated.

Description

Technical Field

[0001] The present disclosure relates to Vincent video technology, and in particular to a video generation method, device, electronic device, storage medium and product. Background Art

[0002] With the rapid development of video generation model technology, video generation models have shined in the field of artificial intelligence content generation (AIGC) technology.

[0003] Controlling the generation of video images based on the depth information of the scene required by the video can be applied to multiple scenarios. For example, for game developers, the depth information of the required scene can be used to generate a three-dimensional scene to enhance the visual effects and immersion of the game; for movie special effects, the depth information of the video frame image can be used to provide realistic visual effects for the post-production of the video frame image.

[0004] In the related art, the text description information input by the user is processed by natural language to obtain corresponding feature information, and the feature information is processed by the video generation model to generate a video corresponding to the text description information. Since the text description information input by the user cannot express the specific requirements for the video to be generated in detail, or the video generation model does not have enough accuracy to generate the corresponding video based on the text description information, the video generated by the video generation model is difficult to meet the user's needs.

[0005] How to obtain video clips that have a high degree of match with the conditional information guiding video generation is an urgent problem to be solved. Summary of the invention

[0006] The embodiments of the present disclosure provide a video generation method, device, electronic device, storage medium and product to solve the above problems.

[0007] According to a first aspect of the present disclosure, a video generation method is provided, including: Acquire condition information for guiding video generation, wherein the condition information at least includes: a first depth map sequence corresponding to the first video clip; The condition information is processed based on the first adapter model and the video model to obtain a second video clip corresponding to the condition information, wherein the first adapter model is obtained based on joint training of the second adapter model and the video model.

[0008] In some embodiments of the present disclosure, obtaining condition information for guiding video generation includes: Performing depth processing on each video frame image in the first video clip to obtain a plurality of depth maps corresponding to each video frame image; The multiple depth maps are sorted based on the temporal relationship between the video frame images in the first video segment to obtain the first depth map sequence.

[0009] In some embodiments of the present disclosure, the condition information guiding video generation further includes: text description information corresponding to the second video clip.

[0010] In some embodiments of the present disclosure, the input layer structures of the first adapter model and the video model are the same, and both the input layer of the first adapter model and the input layer of the video model include multiple downsampling layers; The step of processing the condition information based on the first adapter model and the video model to obtain a second video segment corresponding to the condition information includes: Processing the first depth map sequence based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; At each time step, the output vectors of each downsampling layer of the first adapter model are weightedly calculated with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each sampling layer, and the weighted calculation results of the output vectors of each sampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0011] In some embodiments of the present disclosure, performing weighted calculation on the output vectors of each downsampling layer of the first adapter model and the output of the downsampling layer corresponding to the video model to obtain the second video clip includes: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is an output vector of a first downsampling layer of the video model; Performing weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector, wherein the first adapter output vector is an output vector of a first downsampling layer of the first adapter model; Using the first vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The Nth vector is used as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth vector is calculated based on the weighted output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0012] In some embodiments of the present disclosure, before the step of processing the condition information based on the first adapter model and the video model to obtain a second video segment corresponding to the condition information, the step further includes: Obtain sample video clips and sample text description information; Processing a second depth map sequence corresponding to the sample video clip based on the second adapter model to obtain an output vector of each downsampling layer of the second adapter model; Based on the video model, the sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix are processed to obtain a third video clip; The parameters of the video model are fixed, and based on the difference between the third video segment and the sample video segment, the parameters of the second adapter model are adjusted to obtain the first adapter model.

[0013] According to a second aspect of the embodiments of the present disclosure, a video generating device is provided, including: An information acquisition module, configured to guide condition information for video generation, wherein the condition information at least includes: a first depth map sequence corresponding to a first video clip; A video generation module is used to process the condition information based on a first adapter model and a video model to obtain a second video clip corresponding to the condition information, wherein the first adapter model is obtained based on joint training of a second adapter model and the video model.

[0014] In some embodiments of the present disclosure, the information acquisition module is used to perform depth processing on each video frame image in the first video clip to obtain a plurality of depth maps corresponding to each video frame image; The information acquisition module is further configured to sort the multiple depth maps based on a temporal relationship between the video frame images in the first video segment to obtain the first depth map sequence.

[0015] In some embodiments of the present disclosure, the condition information guiding video generation further includes: text description information corresponding to the second video clip.

[0016] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of the first adapter model and the input layer of the video model both include multiple downsampling layers; The video generation module is used to process the first depth map sequence based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; The video generation module is also used to perform weighted calculations on the output vectors of each downsampling layer of the first adapter model and the output vectors of the downsampling layer corresponding to the video model at each time step to obtain weighted calculation results of the output vectors of each sampling layer, and use the weighted calculation results of the output vectors of each sampling layer as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0017] In some embodiments of the present disclosure, the video generation module is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is an output vector of the first downsampling layer of the video model; The video generation module is further used to perform weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector, wherein the first adapter output vector is an output vector of a first downsampling layer of the first adapter model; The video generation module is further used to use the first vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The video generation module is also used to use the Nth vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth vector is obtained based on the weighted calculation of the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0018] In some embodiments of the present disclosure, the device further includes: A model training module is used to obtain sample video clips and sample text description information; The model training module is further used to process the second depth map sequence corresponding to the sample video clip based on the second adapter model to obtain the output vector of each downsampling layer of the second adapter model; The model training module is also used to process the sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix based on the video model to obtain a third video clip; The model training module is also used to fix the parameters of the video model, and based on the difference between the third video segment and the sample video segment, adjust the parameters of the second adapter model to obtain the first adapter model.

[0019] According to a third aspect of the present disclosure, an electronic device is provided, including: A memory for storing a computer program product; The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0020] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0021] In the video generation method, device, electronic device, storage medium and product of the disclosed embodiments, since the adapter model cooperates with the video model to improve the matching degree between the generated video clip and the video generation condition, the condition information guiding the video generation is used as the model input, and the condition information guiding the video generation is processed by the first adapter model and the video model, so that a video clip with a high matching degree with the condition information guiding the video generation can be generated. Since the condition information guiding the video generation includes a depth map sequence with depth information, the generated video clip has depth information.

[0022] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 A flowchart of a video generation method in one embodiment of the present disclosure; Figure 2 is a partial flow chart of step S1 in some embodiments of the present disclosure; Figure 3 is a flowchart of step S2 in some embodiments of the present disclosure; Figure 4is a schematic diagram of the workflow of each time step in step S2-2 in some embodiments of the present disclosure; Figure 5 is a flowchart of step S0 in some embodiments of the present disclosure; Figure 6 is a structural block diagram of a video generating device in one embodiment of the present disclosure; Figure 7 It is a structural block diagram of a video generating device in another embodiment of the present disclosure; Figure 8 The structure block diagram of an electronic device in one embodiment of the present disclosure is shown in FIG. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0026] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0027] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0028] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0029] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0030] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0031] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0032] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0033] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0034] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0035] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0036] Figure 1 FIG. 1 is a flow chart of a video generation method in one embodiment of the present disclosure. Figure 1 As shown, the video generation method may include the following steps: S1: Obtain condition information for guiding video generation.

[0037] The condition information guiding video generation at least includes: a first depth map sequence corresponding to the first video segment. Each depth map in the first depth map sequence corresponds to each video frame in the first video segment. The first video segment includes visual content of the second video segment to be generated.

[0038] In an optional embodiment of the present disclosure, the video generation condition input by the user may include: a first video clip. A first depth map sequence may be obtained by depth processing the first video clip, so that the first depth map sequence may be used as at least one information in the condition information guiding video generation.

[0039] Each depth map in the first depth map sequence contains information related to the distance of the scene object surface from the viewpoint, which can directly reflect the geometric shape of the visible surface of the scene object. The depth map visualization can clearly demonstrate the operating principle of the visual attention mechanism. In the disclosed embodiment, the first depth map sequence corresponding to the first video clip is used as the condition information or one of the information to guide video generation, which can better guide the video model to generate video clips that meet the user's expectations.

[0040] S2: Process the condition information based on the first adapter model and the video model to obtain a second video segment corresponding to the condition information. The second video segment has visual content of the first depth map sequence corresponding to the first video segment.

[0041] Among them, the first adapter model is obtained based on the joint training of the second adapter model and the video model. The second adapter model is an untrained adapter model, and can also be called the initial adapter model. The first adapter model is an adapter model obtained through joint training, and can also be called the final adapter model. Optionally, in the process of obtaining the first adapter model based on the joint training of the second adapter model and the video model, the second adapter model and the video model are first obtained, wherein the network layer structure of the second adapter model is the same as the network structure of the input layer of the video model, and the network layers of each second adapter model correspond one-to-one to the network layers of the input layer of the video model, that is, the number of network layers of the second adapter model is the same as the number of network layers of the input layer of the video model, and the network parameters of each network layer of the second adapter model are the same as those of the network layers of the corresponding input layer of the video model. Secondly, the network parameters of each network layer of the second adapter model are adjusted through the loss function of the video model to obtain the first adapter model.

[0042] In an embodiment of the present invention, by obtaining the first adapter model based on joint training of the second adapter model and the video model, it is not necessary to directly adjust the parameters of the video model (the magnitude of the parameters of the video model is generally large, such as billions or tens of billions), which can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training, thereby speeding up the training speed. At the same time, when the video model performs a specific task, the output vector of the input layer of the lower video model is fine-tuned through the trained first adapter model, thereby achieving the specific task that can be completed efficiently and accurately without large-scale overall training of the video model. It can be seen from this that the first adapter model is obtained by joint training of the second adapter model and the video model, and the AIGC technology is used to quickly generate a high-quality second video clip with visual content of the first depth map sequence by including conditional information of the first depth map sequence corresponding to the first video clip, and the versatility of the video model is not affected.

[0043] The video model may be a neural network model based on a diffusion model, and is used to generate a corresponding video according to input data information, wherein the data information includes text information, image information, or voice information, etc., wherein the voice information may be converted into text information through voice information recognition and input into the video model. The video model may be pre-trained, for example, it may be trained according to input samples including input text descriptions and input videos.

[0044] Furthermore, the diffusion model is a generative model based on a deep learning algorithm, which generates high-quality content through a gradual process of adding noise and denoising, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generating tasks. In the disclosed embodiment, the diffusion model is a pre-trained model. After the conditional information is mapped to the latent space expression by the encoder, under the guidance of the latent space expression of the conditional information, the random noise is denoised for T time steps, that is, downsampling and upsampling are performed at each time step to obtain the final latent space expression of the second video clip, and then the latent space expression of the target video is further mapped to the pixel space by the decoder to obtain the final second video clip. Among them, the latent space expression of the conditional information is the latent space expression obtained after encoding the conditional information using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space (LatentSpace) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the video from this latent space expression and map the generation result of the diffusion model from the latent space to the pixel space. In some implementations, the encoder and decoder used in the diffusion model of the disclosed embodiments may both be variational autoencoders (VAE). The latent space of VAE is continuous and suitable for generating continuous data, that is, to make the generated target videos or images correlated.

[0045] The adapter model is a model based on neural network model technology, which is used to fine-tune the parameters of the generated video based on the video model according to the video generation task. It achieves this by adding small, trainable modules instead of retraining the video model.

[0046] The first adapter model can be used to process the condition information for guiding video generation to obtain an output vector of the first adapter model. The output vector of the first adapter model can be processed by the video model to obtain a second video clip. The second video clip is a video clip with depth information, which helps to apply the second video clip to scenes such as games or movies.

[0047] In this embodiment, since the adapter model cooperates with the video model to improve the matching degree between the generated video clip and the video generation condition, the condition information guiding the video generation is used as the model input, and the condition information guiding the video generation is processed by the first adapter model and the video model, so that a video clip with a high matching degree with the condition information guiding the video generation can be generated. Since the condition information guiding the video generation includes a depth map sequence with depth information, the generated video clip has depth information.

[0048] Figure 2 is a partial flow chart of step S1 in some embodiments of the present disclosure. Figure 2 As shown, step S1 may include the following steps: S1-1: Perform depth processing on each video frame image in the first video segment to obtain multiple depth maps corresponding to each video frame image.

[0049] Since each video frame of the first video clip may be captured by different video image acquisition devices, and the internal and external parameters of the video image acquisition devices corresponding to each video frame of the first video clip may be different, the depth of each object in each video frame in the first video clip can be estimated based on the internal and external parameters of the video image acquisition device that captured each video frame in the first video clip, and multiple depth maps corresponding to each video frame are generated based on the depth estimation results of each object. Through this embodiment, the depth information in the first video clip can be obtained more accurately, so as to generate high-quality video based on the first adapter model and the video model in the future.

[0050] S1-2: Sort the multiple depth maps based on the temporal relationship of the video frame images in the first video segment to obtain a first depth map sequence.

[0051] Each video frame image can be labeled according to the temporal relationship of each video frame image in the first video segment. For example, if the first video segment is a video segment with a duration of 3 seconds, and each second of video includes 16 video frame images, then the labels of each video frame image can be: the first frame image of the first second of video is labeled A 1-1 , the second frame of the first second video is labeled NA 1-2 ,…, the first frame of the second second video is labeled NA 2-1 ,…, the 16th frame image at the 3rd second is labeled NA 3-16 .

[0052] The depth map corresponding to each video frame image can be labeled accordingly. For example, the depth map corresponding to the first frame image of the first second of video is numbered NB 1-1 , the depth map corresponding to the second frame of the first second video is labeled NB 1-2 ,…, the depth map corresponding to the first frame of the second video is labeled NB 2-1 ,…, the depth map corresponding to the 16th frame image at the 3rd second is labeled NB 3-16 .

[0053] The multiple depth maps may be sorted according to the serial numbers of the depth maps corresponding to the video frame images to obtain a first depth map sequence.

[0054] In this embodiment, depth processing and sorting are performed on each video frame image in the first video segment, so as to obtain a first depth map sequence quickly and accurately.

[0055] In some embodiments of the present disclosure, the conditional information guiding the generation of the video also includes: text description information corresponding to the second video clip. The text description information is used to characterize the visual content information of the finally generated video. Further, the text description information can be used to guide the processing operation of the visual content in the first depth map sequence corresponding to the first video clip. For example, the first video clip includes a depth map sequence of video frames of a cat on the grass, and the text description information can be "the cat runs from the grass to fly in the air."

[0056] Accordingly, the output vector and text description information of the first adapter model may be used as inputs of the video model, and the video model may be used to output the second video segment.

[0057] In this embodiment, the text description information can be used as a basic condition for generating a video, and the first video clip or the first depth map sequence can be used as an auxiliary limiting condition for generating a video, and the first adapter model and the video model can be used to generate a second video clip that is highly matched with the first video clip or the first depth map sequence and the text description information. As a general video generation model, the video model has a relatively strong ability to understand text. In this embodiment, by using the text description information as the input of the video model, it can be ensured that the generated second video clip is more in line with the user's expectations.

[0058] In some embodiments of the present disclosure, the input layer structures of the first adapter model and the video model are the same. The input layer of the first adapter model and the input layer of the video model both include multiple downsampling layers, and feature vectors of the same dimension can be extracted for conditional information guiding video generation through input layers with the same network structure.

[0059] Figure 3 FIG. 1 is a flow chart of step S2 in some embodiments of the present disclosure. Figure 3 As shown, step S2 includes the following steps: S2-1: Process the first depth map sequence based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model.

[0060] The first adapter model may include N+1 downsampling layers, where N is an integer greater than 1. The output vector of the first downsampling layer to the output vector of the N+1th downsampling layer are stored in the order of 1 to N+1, denoted as vector VA1 to vector VA N+1 .

[0061] S2-2: At each time step, the output vectors of each downsampling layer of the first adapter model are weightedly calculated with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each sampling layer, and the weighted calculation results of the output vectors of each sampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0062] Noise can be added to the video model, and the video model is used to gradually denoise the output vectors of each downsampling layer of the first adapter model to finally generate a second video clip.

[0063] In each denoising process, the output vectors of each downsampling layer of the first adapter model are weightedly calculated with the output vectors of the corresponding downsampling layers of the video model to obtain the output vector of the input layer of the video model. The video model includes an input layer and an output layer, and the input layer of the video model includes multiple downsampling layers. The multiple downsampling layers of the video model input layer are mapped one-to-one with the multiple downsampling layers of the first adapter model, and the structure of each group of corresponding downsampling layers is the same. The output vector of each downsampling layer of the first adapter model is used for weighted calculation with the output vector of the mapped downsampling layer of the video model to serve as the input vector of the downsampling layer of the next layer of the video model.

[0064] It can be understood that the input layer and output layer of the above video model perform one round of calculation for noise prediction of one time step, that is, denoising processing of one time step. The video model can obtain the final target video by performing T rounds of iterative denoising. After the output layer of the video model outputs the output vector of the first round, it is input into the input layer of the video model again as the initial input vector of the second round. After the denoising is completed after T rounds of loop execution according to the above process, the final output vector of the output layer of the video model is decoded by the decoder, and it is mapped from the latent space to the pixel space to obtain the second video clip.

[0065] In this embodiment, since the input layer structures of the first adapter model and the video model are the same, feature vectors of the same dimension can be extracted for the input information of each model, and then a second video clip that highly matches the conditional information guiding video generation can be obtained through stepwise denoising.

[0066] Figure 4 is a schematic diagram of the workflow of each time step in step S2-2 in some embodiments of the present disclosure. Figure 4 As shown, step S2-2 may further include the following steps: S2-2-1: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector.

[0067] By performing feature extraction on the text description information and the first Gaussian noise matrix through the video model, a vector of a predetermined dimension can be obtained as the output vector of the first downsampling layer in the input layer of the video model, which is recorded as the first video output vector and represented by vector VB1.

[0068] S2-2-2: Perform weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector.

[0069] The first adapter output vector is the output vector of the first downsampling layer of the first adapter model.

[0070] For S2-2-2, the output vector of the first downsampling layer of the first adapter model is extracted from the output vectors of each downsampling layer of the first adapter model, and is represented by vector VA1. A weighted calculation is performed on vector VA1 and vector VB1, for example, a weighted sum is performed on vector VA1 and vector VB1 to obtain a first vector.

[0071] S2-2-3: Use the first vector as the input of the second downsampling layer of the video model to obtain a second video output vector.

[0072] The second video output vector is the output vector of the second downsampling layer of the video model.

[0073] The first vector is used as the input of the second downsampling layer of the video model to obtain the output vector of the second downsampling layer of the video model, which is recorded as the second video output vector. The first downsampling layer of the video model is adjacent to the second downsampling layer of the video model, that is, the second downsampling layer of the video model is the next downsampling layer adjacent to the first downsampling layer of the video model.

[0074] According to the process of S2-2-2 and S2-2-3, the output vector of the Nth downsampling layer of the video model can be obtained in the same way, that is, the Nth video output vector, which is recorded as vector VB N .

[0075] From the output vectors of each downsampling layer of the first adapter model, extract the output vector of the Nth downsampling layer of the first adapter model (i.e., vector VA N ), the vector VA N and vector VB N Perform weighted calculations, such as VA N and vector VB N Perform weighted summation to obtain the Nth vector, where N is an integer greater than 1.

[0076] S2-2-4: Use the Nth vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model.

[0077] The Nth vector is used as an input of the N+1th downsampling layer of the video model to obtain an output vector of the N+1th downsampling layer of the video model, for example, an output vector of the last downsampling layer of the video model.

[0078] In this embodiment, since the input layer structures of the first adapter model and the video model are the same, and the dimensions of the corresponding downsampling layer input vectors of the first adapter model and the video model are the same, the first Gaussian noise matrix is ​​input into the video model, and the output vectors of each downsampling layer of the first adapter model and the video model are weighted to obtain a vector as the input of the next downsampling layer of the video model, thereby achieving denoising within a single time step, and finally obtaining a second video clip that highly matches the guidance video generation conditions through step-by-step denoising.

[0079] In some embodiments of the present disclosure, before step S1, the following steps may also be included: Step S0: Based on the sample video clip and the sample text description information corresponding to the sample video clip, the second adapter model and the video model are jointly trained to obtain the first adapter model.

[0080] Figure 5 FIG. 1 is a flow chart of step S0 in some embodiments of the present disclosure. Figure 5 As shown, step S0 may include the following steps: S0-1: Obtain sample video clips and sample text description information. In an optional embodiment, the sample text description information corresponds to the sample video clip. The sample text description information can be obtained by manually annotating the sample video clip.

[0081] S0-2: Processing a second depth map sequence corresponding to the sample video clip based on the second adapter model to obtain output vectors of each downsampling layer of the first adapter model.

[0082] The second depth map sequence corresponding to the sample video clip may be obtained in a manner similar to that of obtaining the first depth map sequence in step S1.

[0083] The second depth map sequence may be processed using the second adapter model in a manner similar to step S2-1 to obtain output vectors of each downsampling layer of the second adapter model.

[0084] S0-3: Based on the video model, the sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix are processed to obtain a third video clip.

[0085] The sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix can be processed using the video model in a manner similar to step S2-2 to obtain a third video clip.

[0086] S0-4: Fix the parameters of the video model, and adjust the parameters of the second adapter model based on the difference between the third video segment and the sample video segment to obtain the first adapter model.

[0087] The loss value is calculated according to the difference between the third video segment and the sample video segment using a preset loss function calculation formula.

[0088] When the parameters of the video model are fixed, the parameters of the second adapter model are adjusted according to the loss value by back propagation.

[0089] When the preset model training end conditions are met, for example, the number of iterations reaches a preset number threshold, or the loss value calculated after a certain iteration is less than a preset loss threshold, the model training process is ended to obtain the first adapter model.

[0090] In this embodiment, since the video model has many parameters as a large model, adjusting the parameters of the video model to improve the matching degree between the generated video and the text description information requires a lot of model training costs. Therefore, under the premise of fixing the video model parameters, the matching degree between the video generated by the video model and the conditional information guiding the generation of the video can be improved by training a matching adapter model, thereby significantly reducing the model training cost.

[0091] Figure 6 FIG. 1 is a structural block diagram of a video generating device in one embodiment of the present disclosure. Figure 6 As shown, the video generating device comprises: The information acquisition module 100 is used to guide the condition information of video generation, wherein the condition information at least includes: a first depth map sequence corresponding to the first video clip; The video generation module 200 is used to process the condition information based on the first adapter model and the video model to obtain a second video clip corresponding to the condition information, wherein the first adapter model is obtained based on the joint training of the second adapter model and the video model.

[0092] In some embodiments of the present disclosure, the information acquisition module 100 is used to perform depth processing on each video frame image in the first video clip to obtain a plurality of depth maps corresponding to each video frame image; The information acquisition module 100 is further configured to sort the multiple depth maps based on the temporal relationship of the video frame images in the first video segment to obtain a first depth map sequence.

[0093] In some embodiments of the present disclosure, the condition information guiding the generation of the video further includes: text description information corresponding to the second video clip.

[0094] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of the first adapter model and the input layer of the video model both include multiple downsampling layers; The video generation module 200 is used to process the first depth map sequence based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; The video generation module 200 is also used to perform weighted calculation on the output vectors of each downsampling layer of the first adapter model and the output of the downsampling layer corresponding to the video model at each time step to obtain a second video clip.

[0095] In some embodiments of the present disclosure, the video generation module 200 is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is an output vector of the first downsampling layer of the video model; The video generation module 200 is further used to perform weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector, wherein the first adapter output vector is an output vector of the first downsampling layer of the first adapter model; The video generation module 200 is further configured to use the first vector as an input of a second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is an output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The video generation module 200 is also used to use the Nth vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth vector is obtained based on the weighted calculation of the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0096] Figure 7 FIG. 1 is a structural block diagram of a video generating device in some other embodiments of the present disclosure. Figure 7 As shown, the video generating device further includes: Model training module 300, used to obtain sample video clips and sample text description information; The model training module 300 is further used to process the second depth map sequence corresponding to the sample video clip based on the second adapter model to obtain the output vector of each downsampling layer of the second adapter model; The model training module 300 is also used to process the sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix based on the video model to obtain a third video clip; The model training module 300 is also used to fix the parameters of the video model, and based on the difference between the third video segment and the sample video segment, adjust the parameters of the second adapter model to obtain the first adapter model.

[0097] It should be noted that the specific implementation of the video generating device of the embodiment of the present disclosure is similar to the specific implementation of the video generating method of the embodiment of the present disclosure, and the technical effect of the video generating device of the embodiment of the present disclosure is similar to the technical effect of the video generating method of the embodiment of the present disclosure. Please refer to the description of the video generating method part for details. In order to reduce redundancy, it is not repeated here.

[0098] In addition, an embodiment of the present disclosure further provides an electronic device, including: Memory for storing computer programs; The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the video generation method described in any of the above embodiments of the present disclosure is implemented.

[0099] Below, reference Figure 8 To describe the electronic device according to the embodiment of the present disclosure. Figure 8 As shown, the electronic device includes one or more processors and memory.

[0100] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0101] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the video generation method of each embodiment of the present disclosure described above and / or other desired functions.

[0102] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0103] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0104] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0105] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0106] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0107] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0108] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0109] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0111] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0113] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0114] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0115] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0116] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video generation method, characterized in that: include: Acquire condition information for guiding video generation, wherein the condition information at least includes: a first depth map sequence corresponding to the first video clip; The condition information is processed based on the first adapter model and the video model to obtain a second video clip corresponding to the condition information, wherein the first adapter model is obtained based on joint training of the second adapter model and the video model.

2. The method according to claim 1, characterized in that The step of obtaining condition information for guiding video generation includes: Performing depth processing on each video frame image in the first video clip to obtain a plurality of depth maps corresponding to each video frame image; The multiple depth maps are sorted based on the temporal relationship between the video frame images in the first video segment to obtain the first depth map sequence.

3. The method according to claim 1, characterized in that The condition information guiding video generation further includes: text description information corresponding to the second video segment.

4. The method according to claim 3, characterized in that The input layer structures of the first adapter model and the video model are the same, and both the input layer of the first adapter model and the input layer of the video model include multiple downsampling layers; The step of processing the condition information based on the first adapter model and the video model to obtain a second video segment corresponding to the condition information includes: Processing the first depth map sequence based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; At each time step, the output vectors of each downsampling layer of the first adapter model are weightedly calculated with the output vector of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and the weighted calculation results of the output vectors of each downsampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

5. The method according to claim 4, characterized in that The step of performing weighted calculation on the output vectors of each downsampling layer of the first adapter model and the output vectors of the downsampling layer corresponding to the video model to obtain weighted calculation results of the output vectors of each sampling layer, and using the weighted calculation results of the output vectors of each sampling layer as input of the next sampling layer corresponding to the video model, comprises: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is an output vector of a first downsampling layer of the video model; Performing weighted calculation on the first video output vector and the first adapter output vector to obtain a first vector, wherein the first adapter output vector is an output vector of a first downsampling layer of the first adapter model; Using the first vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The Nth vector is used as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth vector is calculated based on the weighted output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

6. The method according to claim 3, characterized in that Before the step of processing the condition information based on the first adapter model and the video model to obtain a second video segment corresponding to the condition information, the step further includes: Obtain sample video clips and sample text description information; Processing a second depth map sequence corresponding to the sample video clip based on the second adapter model to obtain an output vector of each downsampling layer of the second adapter model; Based on the video model, the sample text description information, the output vectors of each downsampling layer of the second adapter model, and the second Gaussian noise matrix are processed to obtain a third video clip; The parameters of the video model are fixed, and based on the difference between the third video segment and the sample video segment, the parameters of the second adapter model are adjusted to obtain the first adapter model.

7. A video generating device, characterized in that: include: An information acquisition module, configured to guide condition information for video generation, wherein the condition information at least includes: a first depth map sequence corresponding to a first video clip; A video generation module is used to process the condition information based on a first adapter model and a video model to obtain a second video clip corresponding to the condition information, wherein the first adapter model is obtained based on joint training of a second adapter model and the video model.

8. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the method according to any one of claims 1 to 6.

Images