Video generation model training method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses a video generation model training method and device, equipment and a medium, and the method comprises the steps: inputting a video sample into a pre-trained spatial variational automatic encoder for spatial coding, and obtaining a video spatial coding result; inputting the video space coding result into a pre-trained time variation automatic encoder for time coding to obtain a video space-time coding result; training a to-be-trained video generation model based on the video space-time coding result to obtain a target video generation model; wherein the to-be-trained video generation model is a diffusion model. According to the technical scheme provided by the embodiment of the invention, the problem that the training calculation load of the current video generation model is large is solved, the video sample can be coded in space and time dimensions during model training, the calculation resource demand in the model training process is reduced, and the model training efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a video generation model training method, device, equipment, medium and product. Background Art

[0002] Most of the Wensheng video models are based on the architecture of diffusion model and variational autoencoder. The training video encoding is first reduced to a smaller latent space through variational autoencoder, and the diffusion model is trained in this latent space. During inference, the diffusion model is used to generate video in the latent space, and then decoded to the regular video space through variational autoencoder.

[0003] However, current variational autoencoders only perform spatial dimensionality reduction on videos, which imposes a heavy burden on the training cost of the diffusion model. Summary of the invention

[0004] Embodiments of the present invention provide a video generation model training method, apparatus, equipment, medium and product, which can encode video samples in spatial and temporal dimensions during model training, reduce the computing resource requirements during model training, and improve model training efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a video generation model training method, the method comprising:

[0006] Input the video sample into the pre-trained spatial variational autoencoder for spatial encoding to obtain the video spatial encoding result;

[0007] The video spatial encoding result is input into the pre-trained temporal variational autoencoder for temporal encoding to obtain the video spatiotemporal encoding result;

[0008] The video generation model to be trained is trained based on the video spatiotemporal coding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0009] In a second aspect, an embodiment of the present invention further provides a video generation model training device, the device comprising:

[0010] The first encoding module is used to input the video sample into a pre-trained spatial variational autoencoder for spatial encoding to obtain a video spatial encoding result;

[0011] The second encoding module is used to input the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result;

[0012] The model training module is used to train the video generation model to be trained based on the video spatiotemporal encoding results to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0014] one or more processors;

[0015] A memory for storing one or more programs;

[0016] When one or more programs are executed by one or more processors, the one or more processors implement the video generation model training method provided by any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a video generation model training method as provided in any embodiment of the present invention.

[0018] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements the video generation model training method provided by any embodiment of the present invention.

[0019] The embodiments of the above invention have the following advantages or beneficial effects:

[0020] The embodiment of the present invention obtains a video spatial encoding result by inputting a video sample into a pre-trained spatial variational autoencoder for spatial encoding; the video spatial encoding result is input into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; a video generation model to be trained is trained based on the video spatiotemporal encoding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model. The technical solution of the embodiment of the present invention solves the problem of heavy computational load in current video generation model training, and can encode video samples in spatial and temporal dimensions during model training, thereby reducing the computational resource requirements during model training and improving model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a video generation model training method provided by an embodiment of the present invention;

[0022] Figure 2 is a flow chart of a video generation model training method provided by an embodiment of the present invention;

[0023] Figure 3 is a flow chart of a video generation model training method provided by an embodiment of the present invention;

[0024] Figure 4 It is a structural schematic diagram of a video generation model training device provided by an embodiment of the present invention;

[0025] Figure 5 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0027] Figure 1 A flowchart of a video generation model training method provided in an embodiment of the present invention. This embodiment can be applied to scenarios where a neural network model with video generation capabilities is trained. The device can be implemented by software and / or hardware and integrated into a computer device with application development capabilities.

[0028] like Figure 1 As shown, the video generation model training method of this embodiment includes the following steps:

[0029] S110, inputting the video sample into a pre-trained spatial variational autoencoder for spatial encoding to obtain a video spatial encoding result.

[0030] Among them, the spatial variational autoencoder can be a variational autoencoder commonly used at present, which is mainly composed of an encoder and a decoder. The encoder maps input data (such as images, texts, etc.) to a latent space representation. The decoder restores the representation of the latent space back to the original data space to obtain reconstructed data.

[0031] In this step, the video samples used to train the video generation model are input into the pre-trained spatial variational autoencoder for spatial encoding, that is, the video data is spatially encoded through the encoder of the spatial variational autoencoder to obtain the video spatial encoding result.

[0032] S120, inputting the video spatial coding result into a pre-trained temporal variational autoencoder for temporal coding to obtain a video spatiotemporal coding result.

[0033] Normally, when training a video generation model, the video data is only encoded spatially, and the video spatial encoding result is directly input into the video generation model to be trained for model training. In this embodiment, based on the video spatial encoding result, another encoding is performed to further reduce the dimension of the video data. That is, the temporal dimension is encoded by a pre-trained temporal variational autoencoder to obtain the video spatiotemporal encoding result.

[0034] S130, training a video generation model to be trained based on the video spatiotemporal coding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0035] Training the video generation model to be trained based on the video spatiotemporal coding results can be to perform noise processing on the video spatiotemporal coding results and then input them into the video generation model to be trained to obtain the output results of the model, and then update the parameters of the video generation model to be trained based on the output results of the model to complete the training of the model.

[0036] Only performing spatial dimensionality reduction on video samples will impose a heavy burden on the training cost of the diffusion model. For example, assuming that 8x8 spatial dimensionality reduction can be performed on the video sample, a 720p 2-minute video, the default video frame rate is 30 frames per second, after the (spatial) variational autoencoder dimensionality reduction is performed and 2x2 blocks are cut into input to the diffusion model, then each video in the diffusion model is as long as 216,000 blocks. Training a single video model requires processing such a long sequence, and the diffusion model often needs to train hundreds of millions or even more videos, which will require huge computing resources. In this embodiment, after the spatial dimensionality reduction of the video sample, it will also be reduced in the time dimension, which greatly reduces the computational load of model training.

[0037] The technical solution of this embodiment is to obtain a video spatial encoding result by inputting the video sample into a pre-trained spatial variational autoencoder for spatial encoding; inputting the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; training the video generation model to be trained based on the video spatiotemporal encoding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model. The technical solution of the embodiment of the present invention solves the problem of high computational load in current video generation model training, and can encode video samples in spatial and temporal dimensions during model training, reduce the computational resource demand during model training, and improve model training efficiency.

[0038] Figure 2A flowchart of a video generation model training method provided in an embodiment of the present invention, this embodiment and the video generation model training method in the above embodiment belong to the same inventive concept, and further describes the training process of the video generation model to be trained. The method can be executed by a video generation model training device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0039] like Figure 2 As shown, the video generation model training method of this embodiment includes the following steps:

[0040] S210, inputting the video sample into a pre-trained spatial variational autoencoder for spatial encoding to obtain a video spatial encoding result.

[0041] S220, inputting the video spatial coding result into a pre-trained temporal variational autoencoder for temporal coding to obtain a video spatiotemporal coding result.

[0042] S230 , superimposing noise on the video spatiotemporal coding result to obtain a noise-processed video spatiotemporal coding result.

[0043] S240, inputting the spatiotemporal encoding result of the noise-processed video into the video generation model to be trained to obtain a model output result.

[0044] In another optional implementation, the noise-processed spatiotemporal encoding result of the video and at least one encoding feature data may be input into the video generation model to be trained to obtain a corresponding model output result.

[0045] Among them, the coding feature data can be at least one of text coding feature data, picture coding feature data, noise step coding feature data and frame rate coding feature data, as a control signal in the video generation process.

[0046] S250, calculating the model output loss according to the model output result and the video spatiotemporal encoding result, and updating the parameters of the video generation model to be trained based on the model output loss to obtain the target video generation model.

[0047] The model output loss can be the mean square error loss, and then the gradient of the loss with respect to the model parameters is calculated through the back propagation algorithm, and the optimizer (such as Adam) is used to update the model parameters so that the model can better predict noise. This process will be repeated in multiple training cycles (Epochs) to finally obtain the target video generation model.

[0048] The technical solution of this embodiment is to obtain a video spatial encoding result by inputting the video sample into a pre-trained spatial variational autoencoder for spatial encoding; inputting the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; superimposing noise on the video spatiotemporal encoding result to obtain a noise-processed video spatiotemporal encoding result; inputting the noise-processed video spatiotemporal encoding result into the video generation model to be trained to obtain a model output result; calculating the model output loss according to the model output result and the video spatiotemporal encoding result, and updating the parameters of the video generation model to be trained based on the model output loss to obtain a target video generation model. The technical solution of the embodiment of the present invention solves the problem of high computational load in the current video generation model training, and can encode video samples in the spatial and temporal dimensions during model training, reduce the amount of computational resources required during model training, and improve model training efficiency.

[0049] Figure 3 A flowchart of a video generation model training method provided in an embodiment of the present invention, this embodiment and the video generation model training method in the above embodiment belong to the same inventive concept, and further describes the process of training the spatial variational autoencoder and the temporal variational autoencoder. The method can be executed by a video generation model training device, which can be implemented by software and / or hardware and integrated in a computer device with application development function.

[0050] like Figure 3 As shown, the video generation model training method of this embodiment includes the following steps:

[0051] S310: Train the spatial variational autoencoder and the temporal variational autoencoder to determine weight parameters of the spatial variational autoencoder and the temporal variational autoencoder.

[0052] The spatial variational autoencoder and the temporal variational autoencoder to be trained may be convolutional networks including an attention-based mechanism.

[0053] Specifically, the process of training the spatial variational autoencoder and the temporal variational autoencoder includes: inputting a preset video sample into the spatial variational autoencoder to be trained for encoding to obtain a first video encoding result; inputting the first video encoding result into the temporal variational autoencoder to be trained for encoding to obtain a second video encoding result; inputting the second video encoding result into the temporal variational autoencoder to be trained for decoding to obtain a first video decoding result; inputting the first video decoding result into the spatial variational autoencoder to be trained for decoding to obtain a second video decoding result; calculating the model learning loss based on the second video decoding result and the preset video sample, and updating the parameters of the spatial variational autoencoder to be trained and the temporal variational autoencoder to be trained according to the model learning loss to complete the encoder training process.

[0054] The model learning loss usually includes multiple parts such as reconstruction loss, KL divergence loss, adversarial loss and perceptual loss. A total loss can be determined through multiple losses as the final model learning loss to determine the weight parameter update of the weight parameters of the spatial variational autoencoder and the temporal variational autoencoder, and finally freeze the weight parameters of the spatial variational autoencoder and the temporal variational autoencoder for use in the subsequent video generation model training process.

[0055] S320: Input the video sample into a pre-trained spatial variational autoencoder for spatial encoding to obtain a video spatial encoding result.

[0056] S330, inputting the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result.

[0057] S340, training the video generation model to be trained based on the video spatiotemporal coding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0058] The technical solution of this embodiment is to train the spatial variational autoencoder and the temporal variational autoencoder to determine the weight parameters of the spatial variational autoencoder and the temporal variational autoencoder; input the video sample into the pre-trained spatial variational autoencoder for spatial encoding to obtain the video spatial encoding result; input the video spatial encoding result into the pre-trained temporal variational autoencoder for temporal encoding to obtain the video spatiotemporal encoding result; train the video generation model to be trained based on the video spatiotemporal encoding result to obtain the target video generation model; wherein the video generation model to be trained is a diffusion model. The technical solution of the embodiment of the present invention solves the problem of high computational load in the current video generation model training, and can encode the video samples in the spatial and temporal dimensions during model training, reduce the computational resource demand in the model training process, and improve the model training efficiency.

[0059] Figure 4 A structural schematic diagram of a video generation model training device provided in an embodiment of the present invention. This embodiment is applicable to scenarios in which a neural network model with video generation capabilities is trained. The video generation model training device can be implemented by software and / or hardware and integrated into a computer terminal device with application development capabilities.

[0060] like Figure 4 The video generation model training device shown includes: a first encoding module 410, a second encoding module 420 and a model training module 430.

[0061] Among them, the first encoding module 410 is used to input the video sample into a pre-trained spatial variational autoencoder for spatial encoding to obtain a video spatial encoding result; the second encoding module 420 is used to input the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; the model training module 430 is used to train the video generation model to be trained based on the video spatiotemporal encoding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0062] The technical solution of this embodiment is to obtain a video spatial encoding result by inputting the video sample into a pre-trained spatial variational autoencoder for spatial encoding; inputting the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; training the video generation model to be trained based on the video spatiotemporal encoding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model. The technical solution of the embodiment of the present invention solves the problem of high computational load in current video generation model training, and can encode video samples in spatial and temporal dimensions during model training, reduce the computational resource demand during model training, and improve model training efficiency.

[0063] In an optional implementation, the model training module 430 is specifically used to:

[0064] Superimposing noise on the video spatiotemporal coding result to obtain a noise-processed video spatiotemporal coding result;

[0065] Input the spatiotemporal encoding result of the noise-processed video into the video generation model to be trained to obtain the model output result;

[0066] The model output loss is calculated according to the model output results and the video spatiotemporal encoding results, and the parameters of the video generation model to be trained are updated based on the model output loss to obtain the target video generation model.

[0067] In an optional implementation, the model training module 430 may also be specifically used for:

[0068] The noise-processed video spatiotemporal encoding result and at least one encoding feature data are input into a video generation model to be trained to obtain a model output result.

[0069] In an optional implementation, the video generation model training device further includes an encoder training module, which is used to:

[0070] Before training the video generation model, the spatial variational autoencoder and the temporal variational autoencoder are trained to determine weight parameters of the spatial variational autoencoder and the temporal variational autoencoder;

[0071] The process of training the spatial variational autoencoder and the temporal variational autoencoder includes:

[0072] Inputting a preset video sample into the spatial variational autoencoder to be trained for encoding to obtain a first video encoding result;

[0073] Inputting the first video encoding result into the time variation autoencoder to be trained for encoding to obtain a second video encoding result;

[0074] Inputting the second video encoding result into the time variation autoencoder to be trained for decoding to obtain the first video decoding result;

[0075] Inputting the first video decoding result into the spatial variational autoencoder to be trained for decoding to obtain a second video decoding result;

[0076] The model learning loss is calculated based on the second video decoding result and the preset video sample, and the parameters of the spatial variational autoencoder to be trained and the temporal variational autoencoder to be trained are updated according to the model learning loss to complete the encoder training process.

[0077] In an optional embodiment, the temporal variational autoencoder to be trained is a convolutional network including an attention mechanism.

[0078] The video generation model training device provided in the embodiment of the present invention can execute the video generation model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller and server, a mobile phone and other terminal devices.

[0080] like Figure 5 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0081] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0082] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0083] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0084] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0085] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0086] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the video generation model training method provided in the embodiment of the present invention, the method comprising:

[0087] Input the video sample into the pre-trained spatial variational autoencoder for spatial encoding to obtain the video spatial encoding result;

[0088] The video spatial encoding result is input into the pre-trained temporal variational autoencoder for temporal encoding to obtain the video spatiotemporal encoding result;

[0089] The video generation model to be trained is trained based on the video spatiotemporal coding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0090] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the video generation model training method provided by any embodiment of the present invention is implemented. The method includes:

[0091] Input the video sample into the pre-trained spatial variational autoencoder for spatial encoding to obtain the video spatial encoding result;

[0092] The video spatial encoding result is input into the pre-trained temporal variational autoencoder for temporal encoding to obtain the video spatiotemporal encoding result;

[0093] The video generation model to be trained is trained based on the video spatiotemporal coding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

[0094] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0095] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0096] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0097] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0099] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements the video generation model training method provided in any embodiment of the present disclosure.

[0100] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code for the disclosed operation, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0101] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video generation model training method, characterized in that: include: Input the video sample into the pre-trained spatial variational autoencoder for spatial encoding to obtain the video spatial encoding result; Inputting the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; The video generation model to be trained is trained based on the video spatiotemporal encoding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

2. The method according to claim 1, characterized in that The training of the to-be-trained video generation model based on the video spatiotemporal encoding result to obtain a target video generation model comprises: Superimposing noise on the video spatiotemporal coding result to obtain a noise-processed video spatiotemporal coding result; Inputting the noise-processed video spatiotemporal encoding result into the video generation model to be trained to obtain a model output result; The model output loss is calculated according to the model output result and the video spatiotemporal encoding result, and the parameters of the video generation model to be trained are updated based on the model output loss to obtain a target video generation model.

3. The method according to claim 2, characterized in that The step of training the to-be-trained video generation model based on the video spatiotemporal encoding result to obtain a target video generation model further includes: Inputting the noise-processed video spatiotemporal encoding result and at least one encoding feature data into a video generation model to be trained to obtain a model output result; The coding feature data is at least one of text coding feature data, picture coding feature data, noise step coding feature data and frame rate coding feature data.

4. The method according to any one of claims 1 to 3, characterized in that: Before performing video generation model training, the method further includes: Training the spatial variational autoencoder and the temporal variational autoencoder to determine weight parameters of the spatial variational autoencoder and the temporal variational autoencoder; The process of training the spatial variational autoencoder and the temporal variational autoencoder includes: Inputting a preset video sample into the spatial variational autoencoder to be trained for encoding to obtain a first video encoding result; Inputting the first video encoding result into a time variational autoencoder to be trained for encoding to obtain a second video encoding result; Inputting the second video encoding result into the to-be-trained temporal variational autoencoder for decoding to obtain a first video decoding result; Inputting the first video decoding result into the to-be-trained spatial variational autoencoder for decoding to obtain a second video decoding result; The model learning loss is calculated based on the second video decoding result and the preset video sample, and the parameters of the spatial variational autoencoder to be trained and the temporal variational autoencoder to be trained are updated according to the model learning loss to complete the encoder training process.

5. The method according to claim 4, characterized in that The temporal variational autoencoder to be trained is a convolutional network including an attention mechanism.

6. A video generation model training device, characterized in that: include: The first encoding module is used to input the video sample into a pre-trained spatial variational autoencoder for spatial encoding to obtain a video spatial encoding result; A second encoding module is used to input the video spatial encoding result into a pre-trained temporal variational autoencoder for temporal encoding to obtain a video spatiotemporal encoding result; The model training module is used to train the video generation model to be trained based on the video spatiotemporal encoding result to obtain a target video generation model; wherein the video generation model to be trained is a diffusion model.

7. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video generation model training method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video generation model training method as described in any one of claims 1-5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When executed by a processor, the computer program implements the video generation model training method as described in any one of claims 1 to 5.

Images