Video extension method and device

Abstract

The disclosed embodiment discloses a video extension method and device, the method comprising: extracting second video data from first video data, constructing third video data with the second video data as the starting point of video data, inputting preset noise information, third video data and video description information into a pre-trained video generation model, and obtaining target video data including extended video data of the first video data. Thus, it is possible to extend the video using the existing video data (first video data) while ensuring the continuity of the video content of the extended complete video data (target video data), thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video generation, and in particular to a video extension method and device. Background Art

[0002] Video extension usually extends an existing video based on its content. In the related art, when extending an existing video, a video that continues the content of the existing video is usually produced manually based on the content of the existing video to extend the existing video. However, this video extension method requires a lot of manpower and the video extension efficiency is low. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a video extension method and device.

[0004] One aspect of an embodiment of the present disclosure provides a video extension method, including: obtaining video description information and first video data corresponding to target video data; extracting second video data from the first video data, wherein the second video data includes a last frame image in the first video data; constructing third video data using the second video data as the starting point of video data; inputting preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain the target video data, wherein the target video data includes extended video data of the first video data.

[0005] Another aspect of the embodiments of the present disclosure provides a video extension device, including: an information acquisition module, used to acquire video description information and first video data corresponding to target video data; a video extraction module, used to extract second video data from the first video data, wherein the second video data includes a tail frame image in the first video data; a video determination module, used to construct third video data using the second video data as the starting point of video data; and a video extension module, used to input preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain the target video data, wherein the target video data includes extended video data of the first video data.

[0006] Another aspect of the embodiments of the present disclosure provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, the above-mentioned video extension method is implemented.

[0007] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video extension method is implemented.

[0008] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, which implement the above-mentioned video extension method when executed by a processor.

[0009] In the disclosed embodiment, the third video data is constructed starting from the second video data including the tail frame of the first video data, so that the video generation model can take into account the video content and the video description information of the second video data when generating the target video data according to the third video data, preset noise information and video description information. This not only ensures the efficient generation of the target video data without the need for manual generation, but also ensures that the generated target video data includes the extended video data of the first video data, and the video content of the extended video data is continuous with the video content of the first video data, thereby effectively ensuring the video quality.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0013] Figure 1 is a flowchart of a video extension method provided by an exemplary embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of step S120 provided by an exemplary embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of a video extension method provided by another exemplary embodiment of the present disclosure;

[0017] Figure 5 It is a structural schematic diagram of an embodiment of the video extension device disclosed in the present invention;

[0018] Figure 6 It is a structural schematic diagram of another embodiment of the video extension device disclosed in the present invention;

[0019] Figure 7 The figure is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0021] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0022] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0023] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0024] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0025] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0026] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0027] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0028] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0029] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0030] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0031] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0032] In the process of implementing the present disclosure, the inventors found that in actual applications, when extending a video, it is often necessary to manually create the video based on the existing video content, which leads to the inefficiency of video extension and requires a lot of manpower.

[0033] Figure 1 FIG. 1 is a flow chart of a video extension method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0034] Step S100: Obtain video description information and first video data corresponding to target video data.

[0035] The video description information may be used to describe the content of the target video data, and the video description information may be text information, for example. The first video data includes multiple frames of images arranged in time sequence. The video file format of the first video data may include, for example, mp4, rm, wmv, or avi, etc. In this embodiment, the video file format of the first video data is not specifically limited.

[0036] Step S110: extracting second video data from the first video data.

[0037] The second video data includes the last frame image in the first video data. In one embodiment, the second video data includes multiple frame images arranged in time sequence.

[0038] In one embodiment, the second video data can be obtained in the following manner: video data of a preset duration and including the last frame image (last frame image) of the first video data can be extracted from the first video data as the second video data. Exemplarily, assuming that the preset duration of the first video data is 3s and the preset duration is 1s, the video data corresponding to the last 1s (preset duration) of the first video data can be extracted as the second video data; or, when the first video data is 1s, the first video data can be directly used as the second video data.

[0039] The second video data can also be obtained in the following manner: split the video data from the mth frame image in the first video data to obtain the second video data. The second video data includes the image from the mth frame to the last frame image in the first video data, and m is an integer greater than or equal to 1. When m is 1, the first video data can be used as the second video data; when m>1, the image from the mth frame to the last frame (last frame) in the first video data can be extracted as the second video data.

[0040] Before extracting the second video data from the first video data, the first video data may be subjected to denoising and other processing to improve the image quality of the obtained second video data.

[0041] Step S120: construct third video data using the second video data as the start of video data.

[0042] The third video data includes the second video data, that is, the third video data includes each frame image in the second video data. In the third video data, the second video data is located at the starting position of the third video data.

[0043] In one embodiment, the third video data may further include multiple frames of contentless images, which are sequentially located after each frame of images in the second video data in the third video data, wherein the contentless images may be blank images or completely black images, for example.

[0044] Step S130: input the preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain target video data.

[0045] The target video data includes extended video data of the first video data. In one embodiment, the target video data also includes second video data, and in the target video data, the video data that is later in time sequence than the second video data is the extended video data of the first video data. The video generation model can be a deep learning model for extending a video according to the video data and the text description information.

[0046] In the disclosed embodiment, the third video data is constructed starting from the second video data including the tail frame of the first video data, so that the video generation model can take into account the video content and the video description information of the second video data when generating the target video data according to the third video data, preset noise information and video description information. This not only ensures the efficient generation of the target video data without the need for manual generation, but also ensures that the generated target video data includes the extended video data of the first video data, and the video content of the extended video data is continuous with the video content of the first video data, thereby effectively ensuring the video quality.

[0047] In some optional implementations, in the disclosed embodiments, the pre-trained video generation model includes an adapter network module and a video generation network module.

[0048] In one embodiment, the video generation model may include: an adapter network module and a video generation network module. The adapter network module and the video generation network module may be pre-trained deep learning models. In this embodiment, the pre-trained video generation model is obtained by joint training of the video generation network module and the adapter network module to be trained. Optionally, in the process of joint training based on the video generation network module and the adapter network module to be trained, the video generation network module and the adapter network module to be trained are first obtained, wherein the network layer structure of the adapter network module to be trained is the same as the network structure of the input layer of the video generation network module, and each network layer of the adapter network module to be trained corresponds to the network layer of the input layer of the video generation network module, that is, the number of layers of the network layer of the adapter network module to be trained is the same as the number of layers of the network layer of the input layer of the video generation network module, and each network layer of the adapter network module to be trained has the same network parameters as the network layer of the corresponding input layer of the video generation network module. Secondly, the network parameters of each network layer of the adapter network module to be trained are adjusted through the loss function of the video generation network module, so as to obtain the adapter network module in the video generation model.

[0049] In this embodiment, the network layer in the adapter network module can be referred to as the first network layer, and the adapter network module can include multiple first network layers. The input layer of the video generation network module includes multiple second network layers, and the multiple first network layers of the adapter network module correspond to the multiple second network layers of the video generation network module in a one-to-one mapping manner, wherein the first vector of the output of each first network layer is used to be superimposed with the second vector of the output of the corresponding second network layer to serve as the input vector of the next second network layer.

[0050] Furthermore, in some embodiments, the structure of each first network layer is the same as that of the corresponding second network layer, and thus the dimension of the output vector of each first network layer is the same as that of the output vector of the corresponding second network layer, thereby facilitating the same-dimensional superposition of the output vector of the first network layer and the output vector of the corresponding second network layer, so as to achieve comprehensive fine-tuning of the video generation network module through the adapter network module.

[0051] Exemplarily, the adapter network module can adopt an adapter model; the video generation network module is a diffusion model structure. Among them, the diffusion model is a generation model based on a deep learning algorithm, which generates high-quality content through a gradual denoising and denoising process, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generation tasks. In the embodiment of the present disclosure, the video generation network module can be obtained by training the diffusion model. The video generation network module has universality in the field of video generation, but the matching effect between the content of the generated video and the input is not necessarily very good. In the embodiment of the present disclosure, by combining the adapter network module with the video generation network module, it can be based on the universality of the video generation network module. When the input includes one or more videos, it can efficiently generate a high-quality video (that is, the matching effect between the content of the generated video and the input content is good).

[0052] The preset noise information may include Gaussian noise. Exemplarily, a Gaussian noise matrix may be randomly generated in advance, and the Gaussian noise matrix may be determined as the preset noise information.

[0053] In one embodiment, the video generation network module is a diffusion model structure, and the diffusion model mainly includes two processes: noise addition and denoising: in the noise addition process, the original data is converted into an expression form of latent space through a variational autoencoder, so that the model calculation can be more efficient, and noisy content is obtained after T (T≥1) steps of noise addition; in the denoising process, noise expectation is performed for the given noisy content, and conditional guidance is received during the process, which includes the third video data and video description information, and finally restored from the latent space to the pixel space through the variational autoencoder to obtain a video that meets the expectations. In this embodiment, the third video data and video description information can be better referenced in the video generation process, so that the generated target video data and the input content match better.

[0054] Correspondingly, in the embodiment of the present disclosure, the target video data can be obtained in the following manner: the adapter network module processes the third video data to obtain the first vectors output by each first network layer in the adapter network module, and the video generation network module processes multiple first vectors, preset noise information and video description information to obtain the target video data.

[0055] In one embodiment, when the preset noise information, the third video data and the video description information are input into the video generation model, the adapter network module processes the third video data and records the first vector output by each first network layer; then the video generation network module processes the first vector output by each first network layer, the video description information and the preset noise information to obtain the target video data, and outputs the target video data. The target video data includes the second video data, and the second video data is located at the starting position of the target video data.

[0056] In the disclosed embodiment, the video generation model is designed to include an adapter network module and a video generation network module. The third video data including the second video data is processed by the adapter network module to obtain multiple first vectors. Then, the multiple first vectors, video description information and preset noise information are processed by the video generation network module, so that when the video generation network module generates the target video data, it can take into account both the video content and the video description information of the second video data, so that the video content of the generated extended video data is continuous with the video content of the first video data, thereby achieving the goal of extending the video through the existing video data (first video data) while ensuring the continuity of the video content of the extended complete video data (target video data), thereby improving the user experience.

[0057] Figure 2 is a flow chart of step S120 provided by an exemplary embodiment of the present disclosure. In some optional implementations, such as Figure 2As shown, step S120 may include the following steps:

[0058] Step S121, obtaining preset video data.

[0059] Wherein, each frame image in the preset video data is a completely black image.

[0060] In one embodiment, video generation software may be used to generate preset video data of a preset duration (eg, 10 seconds) in which each frame of image is a completely black image.

[0061] Step S122: determining third video data based on the second video data and the preset video data.

[0062] The third video data includes each frame image in the second video data and at least one frame of all-black image. In the third video data, the timing of at least one frame of all-black image is located after each frame image in the second video data.

[0063] In one embodiment, the third video data can be obtained by replacing the first frame of the preset video data with each frame of the image according to the time sequence of each frame of the second video data. For example, assuming that the duration of the second video data is 1s and the duration of the preset video data is 10s, the frames of the image corresponding to the first second in the preset video data can be replaced with the frames of the second video data; or

[0064] Video data of a preset length is intercepted from the preset video data as the video data to be spliced, and then the video data to be spliced ​​is connected to the second video data to splice the second video data with the video data to be spliced ​​to obtain the third video data.

[0065] Exemplarily, assuming that the duration of the preset video data is 10s, the duration of the second video data is 1s, and the frame rates corresponding to the preset video data and the third video data are 16 FPS, that is, the preset video data includes 160 frames of images, and the 160 frames of images are all completely black images; the second video data is used to replace the video data of the 1s in the preset video data to obtain the third video data, in which the 16 frames of images in the 1s are the images corresponding to the second video data, and the remaining 144 frames of images in the 9s are completely black images.

[0066] In the disclosed embodiment, third video data are constructed using preset video data, both of which are all completely black images, and second video data, so that the third video data obtained are all completely black images except for each frame image in the second video data, thereby facilitating subsequent video generation models to process the third video data.

[0067] In some optional implementations, the video extension method in the embodiment of the present disclosure further includes: splicing the first video data and the target video data to extend the first video data.

[0068] In one embodiment, the first video data and the target video data may be connected (joined) in an order in which the timing of each frame image in the first video data is before the timing of each frame image in the target video data to obtain the target video data.

[0069] In the embodiment of the present disclosure, by splicing the first video data with the target video data, it is achieved to obtain complete video data including the first video data and the target video data.

[0070] In some optional embodiments, in the presently disclosed embodiment, the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module, respectively.

[0071] The video generation network module includes an input layer and an output layer. At each time step, the video generation network module adds or removes noise for one time step through noise prediction of the input layer and the output layer, and then obtains the latent space expression of the final generated video after performing noise prediction for T (T≥1) time steps. In this embodiment, the network layer in the input layer of the video generation network module can be called a second network layer, and any second network layer corresponds to a first network layer.

[0072] The input layer of the video generation network module includes n second network layers, where n is an integer greater than or equal to 1. The network structure of the input layer is the same as the network structure of the adapter network module, that is, the adapter network module includes n first network layers, and the first network layers with the same sequence number correspond to the second network layers. For example, the first first network layer corresponds to the first second network layer, and so on, the nth first network layer corresponds to the nth second network layer. The shape (dimension) of the data output by the first network layer is the same as the shape of the data output by the second network layer.

[0073] Exemplarily, both the first network layer and the second network layer can adopt the UNet network. UNet can denoise random noise to achieve the generation from noise to video. In this embodiment, the input layer of the video generation network module is used to gradually downsample the input noise, extract features and reduce the spatial resolution of the image. The output layer of the video generation network module gradually restores the spatial resolution of the video through deconvolution (or transposed convolution) and upsampling operations; through the jump connection structure of the UNet network, the output layer can combine the low-level features in the input layer (such as edge, texture and other detail information) with high-level features (such as semantic information of video description information), and the fusion of features helps to reconstruct the details of the video; finally, the target video data with the same resolution as the input video is output. In the present disclosure, based on the multi-layer second network layer structure in the input layer of the video generation network module, an independent adapter network module is designed to fine-tune the vector generated by the input layer downsampling process. That is to say, the adapter network module adapts to new tasks by inserting additional parameters into the pre-trained video generation network module, rather than directly adjusting the parameters of the entire video generation network module (the parameter magnitude of the video generation network module is very large, such as billions or tens of billions of parameters), which can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training and speed up training. At the same time, when the video generation network module performs a specific task, the trained adapter network module is used to fine-tune the output vectors of each layer of the output layer downsampled, so that the video generation network module does not need to be trained on a large scale as a whole, and the specific task can be completed efficiently and accurately. It can be seen that through the combination and joint training of the adapter network module and the video generation network module, the AIGC (Artificial Intelligence Generated Content) technology is realized, and high-quality videos are quickly generated based on video data, without affecting the versatility of the video generation network module.

[0074] Correspondingly, in the embodiment of the present disclosure, the video generation network module can process multiple first vectors, preset noise information and video description information in the following manner: in the denoising process of each time step of the video generation network module, based on the preset noise information and video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the target video data.

[0075] The n second network layers respectively process the first vector, preset noise information and video description information output by the corresponding first network layer to obtain n second vectors.

[0076] In the embodiment of the present disclosure, the adapter network module is set to include n first network layers, and the video network module is set to include n second network layers, so that each second network layer can process the first vector output by the corresponding first network layer, thereby enabling the video network module to comprehensively consider the target video data when generating target data.

[0077] In some optional implementations, in the embodiment of the present disclosure, using n second network layers to process the multiple first vectors, the preset noise information, and the video description information may include:

[0078] For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain the second vector output by the second network layer; based on the second vector output by the nth second network layer in the denoising process at the last time step, the target video data is determined.

[0079] The third vector is obtained by adding a second vector output by a second network layer previous to the second network layer to a first vector output by a first network layer corresponding to the previous second network layer. In one embodiment, the target video data is determined based on the second vector output by the nth second network layer and the first vector output by the nth first network layer.

[0080] For example, Figure 3 FIG. 1 is a schematic diagram of an input layer in an adapter network module and a video generation network module provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, when the video description information, the preset noise information and the third video data are input into the video generation model, the third video data can be encoded using a variational auto-encoder (VAE), and then the encoded third video data is input into the adapter network module, and the first vectors output by each first network layer in the adapter network module are recorded;

[0081] The VAE can be used to encode the video description information and the preset noise information. For the denoising process at each time step, the encoded video description information and the preset noise information are input into the first second network layer to obtain a second vector output by the first second network layer. The second vector is added to the first vector output by the first first network layer to obtain a third vector. The third vector can be used as input data of the second second network layer.

[0082] A third vector obtained by adding the second vector output by the i-1th second network layer to the first vector output by the i-1th first network layer and the encoded video description information are input into the i-th second network layer to obtain the second vector output by the i-th second network layer, and the second vector is added to the first vector output by the i-th first network layer to obtain a third vector, which can be used as input data of the i+1th second network layer, wherein 2≤i≤n;

[0083] The result of adding the second vector output by the nth second network layer in the denoising process of the last time step to the first vector output by the nth first network layer is input into the VAE for decoding processing to obtain the target video data.

[0084] In the embodiment of the present disclosure, a third vector obtained by adding the second vector output by each second network layer to the first vector output by the corresponding first network layer is used as input data for the next second network layer, so that the video generation network module can take the second video data into consideration when generating target video data, thereby ensuring that the target video data includes the second video data and extended video data of the first video data, and that the video content of the target video data is continuous.

[0085] In some optional implementations, in the embodiment of the present disclosure, before step S122, the method further includes: adjusting the video parameters corresponding to the second video data so that the video parameters corresponding to the second video data are the same as the video parameters corresponding to the preset video data.

[0086] The video parameters include: frame rate and / or resolution.

[0087] In a specific implementation, the resolution of the second video data is adjusted to the resolution of the preset video data; and the frame rate of the second video data is adjusted to the frame rate of the preset video data.

[0088] The resolution of the second video data can be adjusted to the resolution of the preset video data using video editing software such as Micro Video Editing Editor or Quick Clip; the frame rate of the second video data can be adjusted to the frame rate of the preset video data using frame rate conversion software such as Smooth Video Project, Quick Video or Alight Motion.

[0089] In the embodiment of the present disclosure, the video parameters corresponding to the second video data are adjusted to the video parameters corresponding to the preset video data, thereby facilitating the generation of the third video data based on the second video data and the preset video data.

[0090] Figure 4 FIG. 1 is a flow chart of a video extension method provided by another exemplary embodiment of the present disclosure. In some optional implementations, such as Figure 4 As shown, in the embodiment of the present disclosure, the video generation model can be obtained in the following manner:

[0091] Step S200, obtaining the model to be trained and sample data.

[0092] The model to be trained includes: a video generation network module and an adapter network module to be trained. The sample data includes a plurality of sample video data, and label videos and label video description information corresponding to the plurality of sample video data. For the plurality of sample video data, the sample video data includes a plurality of first frames of images with content and a plurality of second frames of images that are all black, and the timing of the plurality of first frames of images is located before the plurality of second frames of images, that is, the plurality of first frames of images are located at the starting position of the sample video data.

[0093] In an optional implementation, sample video data may be obtained in the following manner: obtaining multiple tagged videos; for the multiple tagged videos, replacing the image after a preset frame image in the tagged video with a completely black image, to obtain sample video data corresponding to the tagged video.

[0094] The preset frame images are images other than the first frame image in the label video.

[0095] Exemplarily, a complete video can be cropped into multiple labeled videos with preset resolutions, preset frame rates, and preset durations. For each labeled video, assuming that the frame rate of the labeled video is 16FPS and the duration is 10s, the labeled video includes 160 frames of images. The 16 frames of images in the 1st second of the video data of the labeled video can be left unprocessed, and the 144 frames of images in the last 9s of the video data of the labeled video can be replaced with all-black images to obtain sample video data corresponding to the labeled video. The above-mentioned operation of replacing the all-black image is performed on each labeled video to obtain multiple sample video data.

[0096] In an optional embodiment, the labeled video description information corresponding to each sample video data can be obtained in the following manner: the labeled video is input into a visual language model (Vision-Language Models, VLMs), the visual language model outputs the video description information corresponding to the labeled video, and the video description information is used as the labeled video description information corresponding to the sample video data obtained based on the labeled video.

[0097] Step S210, for multiple sample video data, the sample video data, and the label video and label video description information corresponding to the sample video data are input into the model to be trained, the sample video data is processed by the adapter network module to be trained, and the fourth vectors output by each first network layer in the adapter network module to be trained are obtained, and the video generation network module processes the multiple fourth vectors and the label video description information to obtain the predicted video.

[0098] Among them, multiple sample video data, and label videos and label video description information corresponding to the multiple sample video data are sequentially input into the model to be trained, and the model to be trained outputs the predicted video corresponding to each sample video data.

[0099] Step S220, adjusting the parameters of the adapter network module to be trained according to the predicted videos and label videos corresponding to each sample video data, until the preset training end condition is met, and obtaining the video generation model from the model to be trained.

[0100] In the model training stage, the parameters of the video generation network module can be frozen, and the parameters of the input layer of the video generation network module can be used as the initialization parameters of the adapter network module to be trained.

[0101] Specifically, the training process is to iteratively adjust the network parameters of the adapter network module, and the network parameters of each layer of the video generation network module remain unchanged. Based on this, according to the loss function of the video generation network module, the network parameters of each first network layer of the adapter network module are adjusted by back propagation. In this way, according to S210 to S220, the loop iteration is repeated until the loss function converges, and a trained adapter network module can be obtained, and each first network layer has a trained network parameter.

[0102] The training method disclosed in the present invention can form sample data consisting of sample video data, label video description information and label video, and due to the sparse content characteristics of the sample video data, an external adapter network module can be used in the downsampling process of the pre-trained video generation network module to realize the training of the adapter network module, thereby efficiently and low-costly obtaining a video generation model that can restore or reconstruct video content.

[0103] In one embodiment, the loss function value can be determined based on the difference between the label video and the predicted video corresponding to each sample video data using a preset loss function. The preset loss function may include, but is not limited to, a cross entropy error function or a mean square error function. The operation of determining the loss function value may be iteratively performed, and the parameters of the adapter network module to be trained may be iteratively adjusted to continuously reduce the loss function value until the loss function value converges, and it is determined that the preset training end condition is met, and the training of the model to be trained is completed, and the trained model to be trained is used as the video generation model.

[0104] Parameter optimizers such as Stochastic Gradient Descent (SGD), Adaptive Gradient Algorithm (Adagrad), Adaptive Moment Estimation (Adam), Root Mean Square Prop (RMSprop) can be used to adjust the parameters of the adapter network module to be trained. For example, the parameter optimizer can be used to calculate the gradient of the parameters of the adapter network module to be trained, and the parameters can be adjusted along the direction of the gradient, where the gradient represents the direction in which the loss function value is reduced the most, and the operation of iteratively determining the loss function value is performed until the loss function value no longer decreases, and the training of the model to be trained is completed, and a video generation model is obtained.

[0105] In the disclosed embodiment, a model to be trained, including a video generation network module and an adapter network module to be trained, is trained using multiple sample video data, as well as label videos and label video description information corresponding to the multiple sample video data. During the model training process, only the parameters of the adapter network module to be trained need to be adjusted, which not only reduces the difficulty of model training and improves the efficiency of model training, but also enables the trained video model to extend the video according to the video.

[0106] In some optional embodiments, in the embodiments of the present disclosure, the video generation network module can process multiple fourth vectors and label video description information in the following manner: for the denoising process of each time step, the first second network layer in the video generation network module processes the label video description information to obtain the fifth vector output by the first second network layer; for the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain the fifth vector output by the second network layer; based on the fifth vector output by the nth second network layer in the denoising process of the last time step, determine the predicted video.

[0107] The sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the second network layer above the second network layer.

[0108] In one embodiment, the network structure of the input layer of the video generation network module is the same as the network structure of the adapter network module to be trained, that is, the input layer of the adapter network module to be trained also includes n first network layers. In the model to be trained, the first network layer with the same sequence number corresponds to the second network layer, and the shape of the data output by the first network layer is the same as the shape of the data output by the second network layer. For example, the first network layer and the second network layer can adopt the UNet network.

[0109] In the forward propagation process of the video generation network module, for the denoising process of each time step, the label video description information is input into the first second network layer, the first second network layer outputs a fifth vector, the fifth vector is added to the fourth vector output by the first first network layer, and a sixth vector is obtained, and the sixth vector is used as input data of the second second network layer;

[0110] The sixth vector obtained by adding the fifth vector output by the i-1th second network layer and the fourth vector output by the i-1th first network layer and the label video description information are input into the i-th second network layer, the i-th second network layer outputs the fifth vector, the fifth vector is added to the fourth vector output by the i-th first network layer, and the sixth vector is obtained as the input data of the i+1th second network layer;

[0111] The predicted video is determined based on the result of adding the fifth vector output by the nth second network layer and the fourth vector output by the nth first network layer in the denoising process of the last time step.

[0112] In the disclosed embodiment, the fifth vector output by each second network layer and the fourth vector output by the corresponding first network layer are added to determine the input data of the next second network layer. Thus, when the video generation network module generates a predicted video, the data output by the adapter network module to be trained can be taken into account, so that the adapter network module to be trained can learn better.

[0113] Figure 5 FIG. 1 is a schematic diagram of the structure of an embodiment of the video extension device disclosed in the present invention. Figure 5 As shown, the video extension device of this embodiment may include: an information acquisition module 300 , a video extraction module 310 , a video determination module 320 and a video extension module 330 .

[0114] The information acquisition module 300 is used to acquire the video description information and the first video data corresponding to the target video data;

[0115] The video extraction module 310 is used to extract second video data from the first video data, where the second video data includes a last frame image in the first video data;

[0116] A video determination module 320, configured to construct third video data using the second video data as the start of video data;

[0117] The video extension module 330 is used to input preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain the target video data, wherein the target video data includes the extended video data of the first video data.

[0118] In some possible implementations of the present disclosure, in an embodiment of the present disclosure, the pre-trained video generation model includes an adapter network module and a video generation network module;

[0119] The inputting the preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain the target video data is further used for:

[0120] The adapter network module processes the third video data to obtain the first vectors output by each first network layer in the adapter network module, and the video generation network module processes multiple first vectors, the preset noise information and the video description information to obtain the target video data.

[0121] Figure 6 FIG. 1 is a schematic diagram of another embodiment of the video extension device disclosed in the present invention. In some possible implementations of the present invention, for example, Figure 6 As shown, the video determination module 320 in the embodiment of the present disclosure includes:

[0122] The data acquisition submodule 321 is used to acquire preset video data, in which each frame image is a completely black image;

[0123] The video creation submodule 322 is used to determine the third video data based on the second video data and the preset video data, and the third video data includes each frame image in the second video data and at least one frame of all-black image, and the timing of the at least one frame of all-black image is located after each frame image of the second video data.

[0124] In some possible implementations of the present disclosure, the video extension device in the embodiment of the present disclosure further includes:

[0125] The splicing module 340 is used to splice the first video data and the target video data to extend the first video data.

[0126] In some possible implementations of the present disclosure, the adapter network module in the embodiment of the present disclosure includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively;

[0127] The processing of the plurality of first vectors, the preset noise information and the video description information by the video generation network module is further used to:

[0128] In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the target video data.

[0129] In some possible implementations of the present disclosure, in the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the target video data, which is further used to:

[0130] For the denoising process at each time step, the first second network layer processes the preset noise information based on the video description information to obtain a second vector output by the first second network layer;

[0131] For the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain a second vector output by the second network layer, wherein the third vector is obtained by adding a second vector output by a second network layer immediately preceding the second network layer to a first vector output by a first network layer corresponding to the immediately preceding second network layer;

[0132] The target video data is determined based on the second vector output by the nth second network layer in the denoising process of the last time step.

[0133] In some possible implementations of the present disclosure, the video extension device in the embodiment of the present disclosure further includes:

[0134] The parameter adjustment module 350 is used to adjust the video parameters corresponding to the second video data so that the video parameters corresponding to the second video data are the same as the video parameters corresponding to the preset video data, and the video parameters include frame rate and / or resolution.

[0135] In some possible implementations of the present disclosure, the video extension device in the embodiment of the present disclosure further includes:

[0136] The sample data acquisition module 360 ​​is used to acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes a plurality of sample video data, and label videos and label video description information respectively corresponding to the plurality of sample video data, wherein the plurality of sample video data includes a plurality of first frames of images with content and a plurality of second frames of images that are all black, and the plurality of first frames of images are located at the starting position of the sample video data;

[0137] The first model training module 370 is used for inputting the sample video data, the label video corresponding to the sample video data and the label video description information into the model to be trained, the adapter network module to be trained processes the sample video data to obtain the fourth vector output by each first network layer in the adapter network module to be trained, and the video generation network module processes the plurality of fourth vectors and the label video description information to obtain a predicted video;

[0138] The second model training module 380 is used to adjust the parameters of the adapter network module to be trained according to each predicted video and the corresponding label video until the preset training end condition is met, and the video generation model is obtained from the model to be trained.

[0139] In some possible implementations of the present disclosure, in an embodiment of the present disclosure, the processing of the plurality of fourth vectors and the label video description information by the video generation network module is further used to:

[0140] For the denoising process at each time step, the first second network layer in the video generation network module processes the label video description information to obtain a fifth vector output by the first second network layer;

[0141] For the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain a fifth vector output by the second network layer, and the sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the above second network layer;

[0142] The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

[0143] In some possible implementations of the present disclosure, the sample data acquisition module 360 ​​in the embodiment of the present disclosure is specifically used to obtain multiple labeled videos; for the multiple labeled videos, the image after the preset frame image in the labeled video is replaced with a completely black image to obtain sample video data corresponding to the labeled video, and the preset frame image is the image in the labeled video except the first frame image.

[0144] The video extension device of the embodiment of the present disclosure corresponds to the embodiment of the video extension method of the present disclosure, and the relevant contents can be referenced to each other and will not be repeated here.

[0145] The beneficial technical effects corresponding to the exemplary embodiments of the video extension device of the disclosed embodiments can be found in the corresponding beneficial technical effects of the above-mentioned corresponding exemplary method part, which will not be repeated here.

[0146] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0147] Memory for storing computer programs;

[0148] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the video extension method described in any of the above embodiments of the present disclosure is implemented.

[0149] Figure 7 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 7 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0150] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0151] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0152] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the video extension method of each embodiment of the present disclosure described above and / or other desired functions.

[0153] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0154] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0155] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0156] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0157] In addition to the above methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the video extension method according to various embodiments of the present disclosure described in the above part of this specification.

[0158] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the video extension method according to various embodiments of the present disclosure described in the above part of this specification.

[0160] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0161] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0162] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0163] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0164] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0165] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0166] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0167] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0168] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video extension method, characterized in that: include: Obtaining video description information and first video data corresponding to the target video data; Extracting second video data from the first video data, wherein the second video data includes a last frame image in the first video data; Constructing third video data with the second video data as the start of video data, wherein the third video data includes each frame image in the second video data and at least one frame of all-black image, and the timing of the at least one frame of all-black image is located after each frame image in the second video data; Inputting preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain the target video data, wherein the target video data includes extended video data of the first video data and the second video data; The video generation model includes an adapter network module and a video generation network module, the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; Using the video generation model to process the preset noise information, the third video data and the video description information includes: for the denoising process of each time step, the first second network layer processes the preset noise information based on the video description information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain the second vector output by the second network layer, the third vector being obtained by adding the second vector output by the previous second network layer of the second network layer to the first vector output by the first network layer corresponding to the previous second network layer; based on the second vector output by the nth second network layer in the denoising process of the last time step, determining the target video data, wherein each first vector is obtained by processing the third video data by the adapter network module.

2. The method according to claim 1, characterized in that The step of inputting the preset noise information, the third video data, and the video description information into a pre-trained video generation model to obtain the target video data includes: The adapter network module processes the third video data to obtain the first vectors output by each first network layer in the adapter network module, and the video generation network module processes multiple first vectors, the preset noise information and the video description information to obtain the target video data.

3. The method according to claim 1 or 2, characterized in that: The step of constructing the third video data by taking the second video data as the starting point of the video data comprises: Acquire preset video data, wherein each frame image in the preset video data is a completely black image; Based on the second video data and the preset video data, the third video data is determined, and the third video data includes each frame image in the second video data and at least one frame of all-black image, and the timing of the at least one frame of all-black image is located after each frame image of the second video data.

4. The method according to claim 3, characterized in that Also includes: The first video data and the target video data are spliced ​​to extend the first video data.

5. The method according to claim 2, characterized in that: The processing of the plurality of first vectors, the preset noise information and the video description information by the video generation network module includes: In the denoising process of each time step of the video generation network module, based on the preset noise information and the video description information, the n first vectors output by the n first network layers are respectively calculated with the corresponding n second vectors output by the n second network layers to obtain the target video data.

6. The method according to claim 4, characterized in that Before determining the third video data based on the second video data and the preset video data, the method further includes: The video parameters corresponding to the second video data are adjusted so that the video parameters corresponding to the second video data are the same as the video parameters corresponding to the preset video data, the video parameters including frame rate and / or resolution.

7. The method according to claim 1, characterized in that: The video generation model is obtained in the following way: Acquire a model to be trained and sample data, wherein the model to be trained includes a video generation network module and an adapter network module to be trained, and the sample data includes a plurality of sample video data, and label videos and label video description information respectively corresponding to the plurality of sample video data, wherein the plurality of sample video data includes a plurality of first frames of images with content and a plurality of second frames of images that are all black, and the timing of the plurality of first frames of images are all located before the plurality of second frames of images; For the plurality of sample video data, the sample video data, and the label video and label video description information corresponding to the sample video data are input into the model to be trained, the adapter network module to be trained processes the sample video data to obtain fourth vectors output by each first network layer in the adapter network module to be trained, and the video generation network module processes the plurality of fourth vectors and the label video description information to obtain a predicted video; The parameters of the adapter network module to be trained are adjusted according to each predicted video and the corresponding label video until a preset training end condition is met, and the video generation model is obtained from the model to be trained.

8. The method according to claim 7, characterized in that The processing of the plurality of fourth vectors and the label video description information by the video generation network module includes: For the denoising process at each time step, the first second network layer in the video generation network module processes the label video description information to obtain a fifth vector output by the first second network layer; For the second second network layer to the nth second network layer in the video generation network module, the second network layer processes the label video description information and the sixth vector to obtain a fifth vector output by the second network layer, and the sixth vector is obtained by adding the fifth vector output by the second network layer above the second network layer and the fourth vector output by the first network layer corresponding to the above second network layer; The predicted video is determined based on the fifth vector output by the nth second network layer in the denoising process of the last time step.

9. A video extension device, characterized in that: include: An information acquisition module, used to acquire video description information and first video data corresponding to target video data; A video extraction module, used to extract second video data from the first video data, wherein the second video data includes a last frame image in the first video data; a video determination module, configured to construct third video data using the second video data as the start of video data, wherein the third video data includes each frame image in the second video data and at least one frame of all-black image, and the timing of the at least one frame of all-black image is located after each frame image in the second video data; A video extension module, used for inputting preset noise information, the third video data and the video description information into a pre-trained video generation model to obtain the target video data, wherein the target video data includes the extended video data of the first video data and the second video data; The video generation model includes an adapter network module and a video generation network module, the adapter network module includes n first network layers, the input layer of the video generation network module includes n second network layers, and the n first network layers of the adapter network module correspond one-to-one to the n second network layers of the video generation network module respectively; Using the video generation model to process the preset noise information, the third video data and the video description information includes: for the denoising process of each time step, the first second network layer processes the preset noise information based on the video description information to obtain the second vector output by the first second network layer; for the second second network layer to the nth second network layer, the second network layer processes the third vector based on the video description information to obtain the second vector output by the second network layer, the third vector being obtained by adding the second vector output by the previous second network layer of the second network layer to the first vector output by the first network layer corresponding to the previous second network layer; based on the second vector output by the nth second network layer in the denoising process of the last time step, determining the target video data, wherein each first vector is obtained by processing the third video data by the adapter network module.

Images