Method, device, equipment, and medium for generating video based on multiple video segments

Abstract

The disclosed embodiment discloses a method, device, equipment, and medium for generating a video based on multiple video clips. The method includes: setting multiple target videos corresponding to the video to be generated into a preset template video to obtain an initial video; inputting preset noise information, the initial video, and text information corresponding to the video to be generated into a pre-trained video generation model, and processing the initial video by an adapter network module in the video generation model to obtain a first vector output by each first network layer in the adapter network module; and processing multiple first vectors, preset noise information, and text information by a video generation network module in the video generation model to obtain the video to be generated. As a result, the generated video to be generated not only conforms to the description of the text information but also includes multiple target videos, thereby improving the matching degree between the generated video to be generated and the video required by the user, and improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of video generation technology, and in particular to a method, device, equipment, and medium for generating a video based on multiple video clips. Background Art

[0002] In recent years, with the rapid development of Vincent video models, Vincent video models have made great achievements in the field of AIGC (Artificial Intelligence Generated Content). In related technologies, Vincent video models usually generate corresponding videos based on the text descriptions corresponding to the videos. However, in practical applications, the generated videos often need to include some specified video clips, but the Vincent video model cannot generate videos containing the specified video clips, which reduces the matching degree between the generated videos and the videos required by users, affecting the user experience. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a method, an apparatus, a device, and a medium for generating a video based on multiple video clips.

[0004] One aspect of an embodiment of the present disclosure provides a method for generating a video based on multiple video clips, including: obtaining multiple target videos and text information corresponding to the video to be generated; setting the multiple target videos to a preset template video to obtain an initial video; inputting preset noise information, the initial video and the text information into a pre-trained video generation model, and processing the initial video by an adapter network module in the video generation model to obtain a first vector output by each first network layer of the adapter network module; and processing the multiple first vectors, the preset noise information and the text information by a video generation network module in the video generation model to obtain the video to be generated, wherein the video to be generated includes the multiple target videos.

[0005] Another aspect of the embodiments of the present disclosure provides a device for generating videos based on multiple video clips, including: an information acquisition module, used to acquire multiple target videos and text information corresponding to the video to be generated; a first video generation module, used to set the multiple target videos to a preset template video to obtain an initial video; a second video generation module, used to input preset noise information, the initial video and the text information into a pre-trained video generation model, an adapter network module in the video generation model processes the initial video to obtain a first vector output by each first network layer of the adapter network module, and a video generation network module in the video generation model processes the multiple first vectors, the preset noise information and the text information to obtain the video to be generated, wherein the video to be generated includes the multiple target videos.

[0006] Another aspect of the embodiments of the present disclosure provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, the above-mentioned method of generating a video based on multiple video segments is implemented.

[0007] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a video based on multiple video segments is implemented.

[0008] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the above-mentioned method for generating a video based on multiple video segments.

[0009] In the disclosed embodiment, the video generation model is designed to include an adapter network module and a video generation network module. The adapter network module is used to process an initial video including multiple target videos to obtain multiple first vectors. Then, the video generation network module is used to process the multiple first vectors, text information and preset noise information, so that the video generation network module takes into account multiple target videos and text information when generating a video to be generated, thereby generating a video to be generated that conforms to the description of the text information and includes multiple target videos, thereby improving the matching degree between the generated video to be generated and the video required by the user, and improving the user experience.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0013] Figure 1 is a flowchart of a method for generating a video based on multiple video clips provided by an exemplary embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure;

[0015] Figure 3 is a flowchart of a method for generating a video based on multiple video clips provided by another exemplary embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of the structure of an embodiment of the apparatus for generating a video based on multiple video segments disclosed in the present invention;

[0017] Figure 5 A schematic diagram of the structure of another embodiment of the apparatus for generating a video based on multiple video segments disclosed in the present invention;

[0018] Figure 6 The figure is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0020] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0021] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0022] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0023] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0024] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0025] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0026] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0027] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0028] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0029] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0030] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0031] In the process of implementing the present disclosure, the inventors found that in practical applications, in application scenarios such as video restoration or video modification, it is usually necessary to obtain a video that includes some specified video clips, but the Vincent video model cannot generate a video that includes the specified video clips. This makes the video generated by the Vincent video model have a low match with the video required by the user, affecting the user experience.

[0032] Figure 1 FIG. 1 is a flow chart of a method for generating a video based on multiple video clips provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0033] Step S100, obtaining multiple target videos and text information corresponding to the video to be generated.

[0034] Among them, the text information is used to describe the screen content in the video to be generated. The text information may include content description information of multiple target videos, content description information of the connection part between multiple target videos, or content description information of some video frames of the video to be generated, etc., wherein the partial video frames of the video to be generated may be video frames of the connection part between multiple target videos, or the first frame of the video to be generated, or the tail frame of the video to be generated, or the video frames corresponding to the specified position and specified duration of the video to be generated. Exemplarily, there are three target videos A, B, and C, the video content of target video A is "a girl in a skirt running on a wide lawn", the video content of target video B is "the girl sitting at the dining table slowly turns back", and the video content of target video C is "a group of people sunbathing on the beach", and the text information may include: a girl in a skirt running on a wide lawn, and then the picture is the girl sitting at the dining table after taking a bath, and then the picture is the girl sitting at the dining table slowly turns back, then the picture is a vast sea, and then the picture is a group of people sunbathing on the beach. Through this text information, the video generation model can be better guided to generate the video content expected by the user.

[0035] The target video is used to generate the video to be generated. In one embodiment, the total duration of the multiple target videos is less than the duration of the video to be generated.

[0036] Step S110, setting multiple target videos into a preset template video to obtain an initial video.

[0037] The preset template video may be a content-free video of a certain length. Exemplarily, the preset template video may be a video of 10 seconds in length, in which each frame image is a solid color image. Preferably, each frame image in the preset template video is a pure black image. Obtaining the initial video through the preset template video is conducive to the versatility of the pre-trained video generation model in performing video generation based on the initial video, that is, it ensures that in any scenario (the user can provide a target video of any length), high-quality videos to be generated can be adaptively generated.

[0038] In one implementation, the multiple target videos may be replaced into the preset template video respectively, so as to set the multiple target videos in the preset template video.

[0039] Step S120, input the preset noise information, the initial video and the text information into a pre-trained video generation model, the adapter network module in the video generation model processes the initial video to obtain the first vector output by each first network layer of the adapter network module, the video generation network module in the video generation model processes the multiple first vectors, the preset noise information and the text information to obtain the video to be generated.

[0040] The video generation model may include: an adapter network module and a video generation network module. The adapter network module and the video generation network module may be pre-trained deep learning models.

[0041] In this embodiment, the network layer in the adapter network module can be referred to as a first network layer, and the adapter network module can include multiple first network layers. The input layer of the video generation network module includes multiple second network layers, and the multiple first network layers of the adapter network module correspond to the multiple second network layers of the video generation network module in a one-to-one mapping manner, wherein the output vector of each first network layer is used to be superimposed with the output vector of the corresponding second network layer to serve as the input vector of the next second network layer.

[0042] Furthermore, in some embodiments, the structure of each first network layer is the same as the structure of the corresponding second network layer, and thus the dimension of the output vector of each first network layer is the same as the dimension of the output vector of the corresponding second network layer, thereby facilitating the same-dimensional superposition of the output vector of the first network layer and the output vector of the corresponding second network layer, so as to achieve comprehensive fine-tuning of the video generation network module through the adapter network module.

[0043] In one embodiment, the adapter network module may adopt an adapter model; the video generation network module may be a diffusion model structure, wherein the diffusion module is a generation model based on a deep learning algorithm, which generates high-quality content through a gradual denoising and denoising process, performs well in generating images, videos, audios, and other high-dimensional data, and has strong performance in generation tasks. In the disclosed embodiment, a video generation network module can be obtained through training, and the video generation network module has universality in the field of video generation, but the matching effect between the content of the generated video and the input may not be very good. In the disclosed embodiment, by combining the adapter network module with the video generation network module, it is possible to efficiently generate a video of higher quality (i.e., the matching effect between the content of the generated video and the input content is good) without changing the universality of the video generation network module when the input includes one or more videos.

[0044] The preset noise information may include Gaussian noise. Exemplarily, a Gaussian noise matrix may be randomly generated in advance, and the Gaussian noise matrix may be determined as the preset noise information.

[0045] In one embodiment, the video generation network module is a diffusion model structure, and the video generation network module mainly includes two processes: noise addition and denoising: in the noise addition process, the original data is converted into an expression form of latent space through a variational autoencoder, so that the model calculation can be more efficient, and noisy content is obtained after T (T≥1) steps of noise addition; in the denoising process, noise is expected for the given noisy content, and conditional guidance is received during the process, which includes the initial video and text information, and finally the variational autoencoder is used to restore the latent space to the pixel space to obtain a video that meets the expectations. In this embodiment, the information of multiple target videos and text information can be better referenced during the video generation process, so that the generated video and the input content match better.

[0046] In one embodiment, when the preset noise information, the initial video and the text information are input into the video generation model, the adapter network module processes the initial video and records the first vector output by each first network layer; then the video generation network module processes the first vector output by each first network layer, the text information and the preset noise information to obtain the video to be generated, and outputs the video to be generated. The video to be generated includes multiple target videos, and the video content of the video to be generated matches the text information.

[0047] In the disclosed embodiment, the video generation model is designed to include an adapter network module and a video generation network module. The adapter network module is used to process an initial video including multiple target videos to obtain multiple first vectors. Then, the video generation network module is used to process the multiple first vectors, text information and preset noise information, so that the video generation network module takes into account multiple target videos and text information when generating a video to be generated, thereby generating a video to be generated that conforms to the description of the text information and includes multiple target videos, thereby improving the matching degree between the generated video to be generated and the video required by the user, and improving the user experience.

[0048] In some optional implementations, in the disclosed embodiment, the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module, respectively, and n is an integer greater than or equal to 1. Among them, the video generation network module includes an input layer and an output layer. At each time step, the video generation network module adds or removes noise for one time step through noise prediction of the input layer and the output layer, and then obtains the latent space expression of the final generated video after performing noise prediction for T (T≥1) time steps. In this embodiment, the network layer in the input layer of the video generation network module can be called a second network layer, and any second network layer corresponds to a first network layer.

[0049] Among them, the network structure of the input layer of the video generation network module is the same as the network structure of the adapter network module, that is, the adapter network module includes n first network layers, and the first network layers with the same sequence number correspond to the second network layers. For example, the first first network layer corresponds to the first second network layer, and so on, the nth first network layer corresponds to the nth second network layer. The shape (dimension) of the data output by the first network layer is the same as the shape of the data output by the second network layer.

[0050] Exemplarily, both the first network layer and the second network layer can adopt the UNet network. U-Net can denoise random noise to achieve the generation from noise to video. In this embodiment, the input layer of the video generation network module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The output layer of the video generation network module gradually restores the spatial resolution of the video through deconvolution (or transposed convolution) and upsampling operations; through the jump connection structure of the UNet network, the output layer can combine the low-level features in the input layer (such as edge, texture and other detail information) with high-level features (such as the semantics of text information), and the fusion of features helps to reconstruct the details of the video; finally, the target video with the same resolution as the input video is output. In the present disclosure, based on the multi-layer second network layer structure in the input layer of the video generation network module, an independent adapter network module is designed to fine-tune the vector generated by the input layer downsampling process. That is to say, the adapter network module adapts to new tasks by inserting additional parameters into the pre-trained video generation network module, rather than directly adjusting the parameters of the entire video generation network module (the parameters of the video generation network module are very large, such as billions or tens of billions of parameters). This can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training and speed up training. At the same time, when the video generation network module performs a specific task, the trained adapter network module is used to fine-tune the output vectors of each layer of the output layer downsampled, so that the video generation network module does not need to be trained on a large scale as a whole, and the specific task can be completed efficiently and accurately. It can be seen that through the combination and joint training of the adapter network module and the video generation network module, it is possible to quickly generate high-quality videos based on AIGC technology by including multiple target videos, and it will not affect the versatility of the video generation network module.

[0051] Correspondingly, in the embodiment of the present disclosure, the video generation network module can process multiple first vectors, preset noise information and text information in the following manner: in the denoising process of each time step of the video generation network module, based on the preset noise information and text information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the video to be generated.

[0052] It can be understood that the input layer and output layer of the video generation network module perform one round of calculation as a noise prediction for one time step, that is, a denoising process for one time step. The video generation network module performs T rounds of iterative denoising to obtain the final generated video. After the output layer outputs the output vector of the first round, it is input into the input layer again as the input vector of the second round. After T rounds of loop execution according to the above process, the output layer outputs the output vector of the Tth round to input the decoder to obtain the final generated target video. The decoder can reconstruct the video from the latent space expression, map the generation result of the video generation network module from the latent space to the pixel space, and generate the final video.

[0053] In the embodiment of the present disclosure, the adapter network module is set to include n first network layers, and the video network module is set to include n second network layers, so that each second network layer can process the first vector output by the corresponding first network layer, thereby enabling the video network module to comprehensively consider the target video when generating the video to be generated, so that the generated video to be generated includes the target video.

[0054] In some optional implementations, in the embodiment of the present disclosure, using n second network layers to process the plurality of first vectors, preset noise information, and text information may include:

[0055] For the denoising process at each time step, the first second network layer processes the preset noise information based on the text information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the text information to obtain the second vector output by the second network layer; based on the second vector output by the nth second network layer in the denoising process at the last time step, the video to be generated is determined.

[0056] Among them, in the 2nd second network layer to the nth second network layer, the third vector is obtained by adding the second vector output by the previous second network layer based on the second network layer and the first vector output by the first network layer corresponding to the previous second network layer. In one embodiment, the video to be generated is determined based on the second vector output by the nth second network layer and the first vector output by the nth first network layer in the denoising process of the last time step (i.e., the Tth round). Figure 2 FIG. 1 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure. Figure 2As shown, for the denoising process at each time step, the text information and the preset noise information are input into the first second network layer to obtain a second vector output by the first second network layer, and the second vector is added to the first vector output by the first first network layer to obtain a third vector, which can be used as input data of the second second network layer;

[0057] A third vector obtained by adding the second vector output by the i-1th second network layer to the first vector output by the i-1th first network layer and the text information are input into the i-th second network layer to obtain the second vector output by the i-th second network layer, and the second vector is added to the first vector output by the i-th first network layer to obtain a third vector, which can be used as input data of the i+1th second network layer, wherein 2≤i≤n;

[0058] The result of adding the second vector output by the nth second network layer in the denoising process of the last time step to the first vector output by the nth first network layer is input to the image decoder, and the image decoder outputs the video to be generated. The image decoder may be, for example, an AutoEncoder Decoder.

[0059] In the embodiment of the present disclosure, the third vector obtained by adding the second vector output by each second network layer to the first vector output by the corresponding first network layer is used as the input data of the next second network layer, thereby enabling the video generation network module to take the target video into consideration when generating the video to be generated, so that the generated video to be generated includes the target video.

[0060] In some optional implementations, in the embodiments of the present disclosure, step S110 may include: for each target video, according to preset position information, replacing the target video with a preset template video.

[0061] Wherein, each frame image in the preset template video is a completely black image, and the initial video includes multiple target videos. The preset position information can be used to indicate the appearance position of the target video in the preset template video. Exemplarily, the preset position information may include: the appearance time of each target video in the preset template video, or the appearance order of each target video and the interval between each target video.

[0062] In one embodiment, for each target video, video editing software such as Hand Brake, Freemake Video Converter or Quick Clip can be used to replace the target video with the video content (video clip) of the same length in the preset template video according to the preset position information, so as to replace the target video with the preset template video.

[0063] In the disclosed embodiment, each target video is replaced with a preset template video according to preset position information, so that the position where the target video appears can be set according to actual needs, thereby improving the flexibility of video generation.

[0064] In an optional implementation manner, before step S110 in the embodiment of the present disclosure, the following may also be included: adjusting the video parameters of the plurality of target videos to the video parameters of the preset template video respectively.

[0065] The video parameters may be indicators used to describe video quality and technical characteristics. The video parameters may include, for example, frame rate and / or resolution.

[0066] In a specific implementation, the resolutions of the multiple target videos are adjusted to the resolutions of the preset template video, and the frame rates of the multiple target videos are adjusted to the frame rate of the preset template video.

[0067] You can use video editing software such as Micro Video Editor or Quick Clip to adjust the resolution of multiple target videos to the resolution of the preset template video; you can use frame rate conversion software such as SmoothVideo Project, Quick Video or Alight Motion to adjust the frame rate of each target video to the frame rate of the preset template video.

[0068] For example, it is assumed that there are three target videos for generating the video to be generated, and the durations of the three target videos are 1 second, 2 seconds, and 3 seconds respectively. The preset template video is a video with a duration of 10 seconds and each frame image is a completely black image. The three target videos are replaced with the preset template video respectively to obtain an initial video, which includes the three target videos.

[0069] In the embodiment of the present disclosure, the video parameters of multiple target videos are adjusted to the video parameters of a preset template video, so that the multiple target videos can be set in the preset template video and the video parameters of each video frame of the obtained initial video are made consistent, thereby ensuring that high-quality video to be generated is generated based on the initial video.

[0070] Exemplarily, the three target videos are represented by 1, 11, and 111, respectively, and the preset template video is represented by 000000000000000000000, wherein each 1 represents a video frame of each target video, and each 0 represents a completely black video frame. If the preset positions of the three target videos in the preset template video (i.e., the temporal positions in the preset template video) are respectively the 3rd frame, the 7th to the 8th frame, and the 14th to the 16th frame, then the three target videos are set to the preset template video to obtain the initial video: 001000110000011100000. The preset noise information, the initial video and the text information are input into the pre-trained video generation model, and the final video to be generated is: 2212221122222211122222, wherein 2 represents a video frame of brand-new visual content generated by the pre-trained video generation model, and the video to be generated includes the visual content of three target videos, and the temporal positions of the three target videos in the video to be generated are consistent with the temporal positions of the three target videos in the preset template video. Therefore, through the embodiment of the present invention, in a scenario where there is a demand for the temporal positions of multiple target videos in the final generated video, a high-quality video with smooth lens transitions including the visual content of multiple target videos is generated based on an adapter network module and a video generation network module, thereby improving the fun of video generation in the field of AIGC technology.

[0071] Figure 3 FIG. 1 is a flow chart of a method for generating a video based on multiple video clips provided by another exemplary embodiment of the present disclosure. Figure 3 As shown, the video generation model can be obtained as follows:

[0072] Step S200, obtaining the model to be trained and sample data.

[0073] The model to be trained includes: a video generation network module and an adapter network module to be trained. The adapter network module to be trained includes n first network layers. The video generation network module in the model to be trained can be a trained video generation network module.

[0074] The sample data includes multiple sample videos, and label videos and label text information corresponding to the multiple sample videos, each of the multiple sample videos includes at least one frame of a completely black image. The label text information corresponding to each sample video can be used to describe the content of the sample video, and the label text information can include, for example, video content information of the sample video. In one embodiment, the video parameters of each sample video are the same, and the duration of each sample video is the same.

[0075] In an optional implementation, the sample video may be obtained in the following manner: a plurality of tagged videos are obtained, and for the plurality of tagged videos, at least one frame of image in the tagged video is replaced with a completely black image to obtain a sample video corresponding to the tagged video.

[0076] Exemplarily, a complete video can be cropped into multiple labeled videos with preset resolutions, preset frame rates, and preset lengths. For each labeled video, assuming that the frame rate of the labeled video is 16FPS and the length is 10s, the labeled video includes 160 frames of images. 15% of the total number of frames in the labeled video can be replaced with completely black images to obtain a sample video corresponding to the labeled video. The above-mentioned operation of replacing the completely black image is performed on each labeled video to obtain multiple sample videos.

[0077] When there is an initial sample video among multiple sample videos whose video parameters and / or duration are different from those of other sample videos, the video parameters and / or duration of the sample video can be adjusted to make the video parameters and / or duration of each sample video in the multiple sample videos the same.

[0078] In an optional implementation, the label text information corresponding to each sample video can be obtained in the following manner: the label video is input into a visual language model (Vision-Language Models, VLMs), the visual language model outputs the text information corresponding to the label video, and the text information is used as the label text information corresponding to the sample video obtained based on the label video.

[0079] Step S210, for multiple sample videos, the sample videos, and the label videos and label text information corresponding to the sample videos are input into the model to be trained, the sample videos are processed by the adapter network module to be trained, and the fourth vectors output by each first network layer in the adapter network module to be trained are obtained, and the video generation network module processes the multiple fourth vectors and label text information to obtain a predicted video.

[0080] Among them, multiple sample videos, and label text information and label videos corresponding to the multiple sample videos are sequentially input into the model to be trained, and the model to be trained outputs the predicted videos corresponding to each sample video.

[0081] Step S220, adjusting the parameters of the adapter network module to be trained according to the predicted videos and the corresponding label videos corresponding to each sample video until the preset training end condition is met, and obtaining the video generation model from the model to be trained.

[0082] Among them, in the model training stage, the parameters of the video generation network module can be frozen, and the parameters of the input layer of the video generation network module can be used as the initialization parameters of the adapter network module to be trained. Specifically, the training process is to iteratively adjust the network parameters of the adapter network module, and the network parameters of each layer of the video generation network module remain unchanged. Based on this, according to the loss function of the video generation network module, the network parameters of each first network layer of the adapter network module are adjusted by back propagation. In this way, according to the loop iteration from S210 to S220 until the loss function converges, a trained adapter network module can be obtained, and each first network layer has a trained network parameter.

[0083] The training method disclosed in the present invention can form sample data from a sample video including at least one frame of completely black image and label text information, and due to the sparse content characteristics of the sample video, an external adapter network module is used during the downsampling process of a pre-trained video generation network module to implement training of the adapter network module, thereby efficiently and at low cost obtaining a video generation model that can restore or reconstruct video content.

[0084] In one embodiment, the loss function value can be determined based on the difference between the label video and the predicted video corresponding to each sample video using a preset loss function. The preset loss function may include, but is not limited to, a cross entropy error function or a mean square error function. The operation of determining the loss function value may be iteratively performed, and the parameters of the adapter network module to be trained may be iteratively adjusted to continuously reduce the loss function value until the loss function value converges, and it is determined that the preset training end condition is met, and the training of the model to be trained is completed, and the trained model to be trained is used as the video generation model.

[0085] Parameter optimizers such as Stochastic Gradient Descent (SGD), Adaptive Gradient Algorithm (Adagrad), Adaptive Moment Estimation (Adam), Root Mean Square Prop (RMSprop) can be used to adjust the parameters of the adapter network module to be trained. For example, the parameter optimizer can be used to calculate the gradient of the parameters of the adapter network module to be trained, and the parameters can be adjusted along the direction of the gradient, where the gradient represents the direction in which the loss function value is reduced the most, and the operation of iteratively determining the loss function value is performed until the loss function value no longer decreases, and the training of the model to be trained is completed, and a video generation model is obtained.

[0086] In the disclosed embodiment, a model to be trained, including a video generation network module and an adapter network module to be trained, is trained using multiple sample videos, as well as label videos and label text information corresponding to the multiple sample videos. During the model training process, only the parameters of the adapter network module to be trained need to be adjusted, which not only reduces the difficulty of model training and improves the efficiency of model training, but also enables the trained video model to generate videos including corresponding video clips.

[0087] In some optional implementations, in the embodiments of the present disclosure, the video generation network module in the model to be trained may process the plurality of fourth vectors and label text information in the following manner:

[0088] For the denoising process at each time step, the first second network layer in the video generation network module processes the label text information to obtain a fifth vector output by the first second network layer;

[0089] For the 2nd to nth second network layers in the video generation network module, the second network layer processes the label text information and the sixth vector to obtain the fifth vector output by the second network layer, and determines the predicted video based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

[0090] The sixth vector is obtained by adding the fifth vector output by the previous second network layer of the second network layer and the fourth vector output by the first network layer corresponding to the previous second network layer.

[0091] In one embodiment, the network structure of the input layer of the video generation network module is the same as the network structure of the adapter network module to be trained, that is, the input layer of the adapter network module to be trained also includes n first network layers. In the model to be trained, the first network layer with the same sequence number corresponds to the second network layer, and the shape of the data output by the first network layer is the same as the shape of the data output by the second network layer.

[0092] In the forward propagation process in the video generation network module, for the denoising process at each time step, the label text information is input into the first second network layer, the first second network layer outputs a fifth vector, the fifth vector is added to the fourth vector output by the first first network layer, and a sixth vector is obtained, and the sixth vector is used as input data of the second second network layer;

[0093] The sixth vector obtained by adding the fifth vector output by the i-1th second network layer and the fourth vector output by the i-1th first network layer and the label text information are input into the i-th second network layer, and the fifth vector is output by the i-th second network layer, and the fifth vector is added to the fourth vector output by the i-th first network layer to obtain a sixth vector, and the sixth vector is used as the input data of the i+1th second network layer;

[0094] The predicted video is determined according to the result of adding the fifth vector output by the nth second network layer and the fourth vector output by the nth first network layer in the denoising process of the last time step.

[0095] In the disclosed embodiment, the fifth vector output by each second network layer and the fourth vector output by the corresponding first network layer are added to determine the input data of the next second network layer. Thus, when the video generation network module generates a predicted video, the data output by the adapter network module to be trained can be taken into account, so that the adapter network module to be trained can learn better.

[0096] Figure 4 FIG. 1 is a schematic diagram of a structure of an embodiment of a device for generating a video based on multiple video clips disclosed in the present invention. Figure 4 As shown, the device of this embodiment may include: an information acquisition module 300 , a first video generation module 310 , and a second video generation module 320 .

[0097] The information acquisition module 300 is used to acquire multiple target videos and text information corresponding to the video to be generated;

[0098] A first video generation module 310, configured to set the plurality of target videos into a preset template video to obtain an initial video;

[0099] The second video generation module 320 is used to input the preset noise information, the initial video and the text information into a pre-trained video generation model, and the adapter network module in the video generation model processes the initial video to obtain the first vector output by each first network layer of the adapter network module, and the video generation network module in the video generation model processes multiple first vectors, the preset noise information and the text information to obtain the video to be generated, wherein the video to be generated includes the multiple target videos.

[0100] In some possible implementations of the present disclosure, the adapter network module in the embodiment of the present disclosure includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively;

[0101] The processing of the plurality of first vectors, the preset noise information and the text information by the video generation network module in the video generation model includes:

[0102] In the denoising process of each time step of the video generation network module, based on the preset noise information and the text information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the video to be generated.

[0103] In some possible implementations of the present disclosure, the n second network layers in the embodiment of the present disclosure process the multiple first vectors, the preset noise information and the text information to obtain second vectors respectively output by the n second network layers, and determine the video to be generated based on the second vectors respectively output by the n second network layers, further used to:

[0104] For the denoising process at each time step, the first second network layer processes the preset noise information based on the text information to obtain a second vector output by the first second network layer;

[0105] For the second second network layer to the nth second network layer, the second network layer processes the third vector based on the text information to obtain a second vector output by the second network layer, and the third vector is obtained by adding the second vector output by the second network layer immediately preceding the second network layer to the first vector output by the first network layer corresponding to the immediately preceding second network layer;

[0106] The video to be generated is determined based on the second vector output by the nth second network layer in the denoising process of the last time step.

[0107] In some possible implementations of the present disclosure, the first video generation module 310 in the embodiment of the present disclosure is specifically used to replace the target video with the preset template video according to preset position information for each target video, wherein each frame image in the preset template video is a completely black image.

[0108] Figure 5 FIG. 1 is a schematic diagram of another embodiment of the apparatus for generating a video based on multiple video segments of the present disclosure. In some possible implementations of the present disclosure, for example, Figure 5 As shown, the apparatus for generating a video based on multiple video segments also includes:

[0109] The first parameter adjustment module 330 is used to adjust the video parameters of the multiple target videos to the video parameters of the preset template video respectively, wherein the video parameters include frame rate and / or resolution.

[0110] In some possible implementations of the present disclosure, the apparatus for generating a complete video using multiple video segments in an embodiment of the present disclosure further includes:

[0111] A training data acquisition module 340 is used to acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes a plurality of sample videos, and label videos and label text information corresponding to the plurality of sample videos, respectively, and each of the plurality of sample videos includes at least one frame of a completely black image;

[0112] The training module 350 is used for inputting the sample videos, the label videos and the label text information corresponding to the sample videos into the model to be trained, processing the sample videos by the adapter network module to obtain the fourth vectors output by each first network layer in the adapter network module to be trained, and processing the plurality of fourth vectors and the label text information by the video generation network module to obtain a predicted video;

[0113] The second parameter adjustment module 360 ​​is used to adjust the parameters of the adapter network module to be trained according to each predicted video and the corresponding label video until the preset training end condition is met, and the video generation model is obtained from the model to be trained.

[0114] In some possible implementations of the present disclosure, the video generation network module in the embodiment of the present disclosure processes the plurality of fourth vectors and the label text information, and is further used to:

[0115] For the denoising process at each time step, the first second network layer in the video generation network module processes the label text information to obtain a fifth vector output by the first second network layer;

[0116] For the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label text information and the sixth vector to obtain a fifth vector output by the second network layer, wherein the sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the above second network layer;

[0117] The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

[0118] The device for generating a video based on multiple video segments in the embodiment of the present disclosure corresponds to the embodiment of the method for generating a video based on multiple video segments in the present disclosure, and the relevant contents can be referenced to each other and will not be repeated here.

[0119] The beneficial technical effects corresponding to the exemplary embodiment of the device for generating video based on multiple video segments in the embodiments of the present disclosure can be found in the corresponding beneficial technical effects of the above-mentioned corresponding exemplary method part, which will not be repeated here.

[0120] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0121] Memory for storing computer programs;

[0122] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method for generating a video based on multiple video segments described in any one of the above embodiments of the present disclosure is implemented.

[0123] Figure 6 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 6 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0124] like Figure 6 As shown, the electronic device includes one or more processors and memory.

[0125] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0126] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the method for generating a video based on multiple video segments and / or other desired functions of the various embodiments of the present disclosure described above.

[0127] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0128] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0129] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0130] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0131] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for generating videos based on multiple video fragments according to various embodiments of the present disclosure described in the above part of this specification.

[0132] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0133] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for generating videos based on multiple video fragments according to various embodiments of the present disclosure described in the above part of this specification.

[0134] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0136] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0137] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0138] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0139] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0140] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0141] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0142] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating a video based on multiple video clips, characterized in that: include: Obtain multiple target videos and text information corresponding to the video to be generated; The multiple target videos are set into a preset template video to obtain an initial video, wherein the duration of the initial video is greater than the total duration of the multiple target videos; Input the preset noise information, the initial video and the text information into a pre-trained video generation model, and the adapter network module in the video generation model processes the initial video to obtain the first vector output by each first network layer of the adapter network module, and the video generation network module in the video generation model processes multiple first vectors, the preset noise information and the text information to obtain the video to be generated, wherein the video to be generated includes the multiple target videos; the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; Using n second network layers to process the multiple first vectors, the preset noise information and the text information includes: for the denoising process of each time step, the first second network layer processes the preset noise information based on the text information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the text information to obtain the second vector output by the second network layer, and the third vector is obtained by adding the second vector output by the previous second network layer based on the second network layer and the first vector output by the first network layer corresponding to the previous second network layer; based on the second vector output by the nth second network layer in the denoising process of the last time step, the video to be generated is determined.

2. The method according to claim 1, characterized in that The processing of the plurality of first vectors, the preset noise information and the text information by the video generation network module in the video generation model includes: In the denoising process of each time step of the video generation network module, based on the preset noise information and the text information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the video to be generated.

3. The method according to claim 1, characterized in that The step of setting the plurality of target videos into a preset template video to obtain an initial video includes: For each of the target videos, the target video is replaced with the preset template video according to the preset position information, wherein each frame image in the preset template video is a completely black image.

4. The method according to any one of claims 1 to 3, characterized in that: Before setting the multiple target videos into the preset template video, the method further includes: The video parameters of the multiple target videos are respectively adjusted to the video parameters of the preset template video, wherein the video parameters include frame rate and / or resolution.

5. The method according to claim 1, characterized in that The video generation model is obtained in the following way: Acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes a plurality of sample videos, and label videos and label text information respectively corresponding to the plurality of sample videos, and each of the plurality of sample videos includes at least one frame of a completely black image; For the multiple sample videos, the sample videos, and the label videos and label text information corresponding to the sample videos are input into the model to be trained, the adapter network module to be trained processes the sample videos to obtain fourth vectors output by each first network layer in the adapter network module to be trained, and the video generation network module processes the multiple fourth vectors and the label text information to obtain a predicted video; The parameters of the adapter network module to be trained are adjusted according to each predicted video and the corresponding label video until a preset training end condition is met, and the video generation model is obtained from the model to be trained.

6. The method according to claim 5, characterized in that The processing of the plurality of fourth vectors and the label text information by the video generation network module includes: The first second network layer in the video generation network module processes the label text information to obtain a fifth vector output by the first second network layer; For the denoising process at each time step, for the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label text information and the sixth vector to obtain a fifth vector output by the second network layer; wherein the sixth vector is obtained by adding the fifth vector output by the second network layer immediately preceding the second network layer to the fourth vector output by the first network layer corresponding to the immediately preceding second network layer; The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

7. A device for generating a video based on multiple video clips, characterized in that: include: An information acquisition module, used to acquire multiple target videos and text information corresponding to the video to be generated; A first video generation module, used for setting the multiple target videos into a preset template video to obtain an initial video, wherein the duration of the initial video is greater than the total duration of the multiple target videos; A second video generation module is used to input the preset noise information, the initial video and the text information into a pre-trained video generation model, and the adapter network module in the video generation model processes the initial video to obtain the first vector output by each first network layer of the adapter network module, and the video generation network module in the video generation model processes multiple first vectors, the preset noise information and the text information to obtain the video to be generated, wherein the video to be generated includes the multiple target videos; the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; Using n second network layers to process the multiple first vectors, the preset noise information and the text information includes: for the denoising process of each time step, the first second network layer processes the preset noise information based on the text information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the text information to obtain the second vector output by the second network layer, and the third vector is obtained by adding the second vector output by the previous second network layer based on the second network layer and the first vector output by the first network layer corresponding to the previous second network layer; based on the second vector output by the nth second network layer in the denoising process of the last time step, the video to be generated is determined.

8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method for generating a video based on multiple video segments described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a video based on multiple video segments as described in any one of claims 1 to 6 is implemented.

Images