Video processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a video processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed video; a video to be processed is input into a pre-trained time-space variational auto-encoder to obtain a video encoding result, and the time-space variational auto-encoder comprises a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space up-sampling unit and a time-space down-sampling unit; the time-space three-dimensional convolution unit is used for carrying out convolution on the video to be processed from a time dimension and a space dimension; the time-space self-attention mechanism unit is used for capturing the space-time dependency relationship of the to-be-processed video from the time dimension and the space dimension; the time-space up-sampling unit is used for carrying out numerical interpolation on the video to be processed from a time dimension and a space dimension; and the time-space down-sampling unit is used for down-sampling the video to be processed through three-dimensional convolution from the time dimension and the space dimension.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video processing method, device, electronic equipment and storage medium. Background Art

[0002] Video generation refers to the training of artificial intelligence to automatically generate descriptive, high-fidelity video content based on given single-modal or multi-modal data such as text, images, and videos.

[0003] Video generation technology mainly relies on the diffusion model and the architecture of variational autoencoders. Currently, Black Forest Laboratory has open-sourced the spatial variational autoencoder of flux's Wensheng graph, which only has the ability to reduce the spatial dimension.

[0004] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art: the existing variational autoencoder has the problem of poor video dimensionality reduction effect. Summary of the invention

[0005] The present invention provides a video processing method, device, electronic device and storage medium to improve the video dimensionality reduction effect.

[0006] According to one aspect of the present invention, there is provided a video processing method, comprising:

[0007] Get the video to be processed;

[0008] Inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed;

[0009] The time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit;

[0010] The time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension;

[0011] The time-space self-attention mechanism unit is used to capture the time-space dependency relationship of the video to be processed from the time dimension and the space dimension;

[0012] The time-space upsampling unit is used to perform numerical interpolation on the video to be processed from the time dimension and the space dimension;

[0013] The time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution.

[0014] According to another aspect of the present invention, there is provided a video processing device, comprising:

[0015] A module for obtaining videos to be processed, used for obtaining videos to be processed;

[0016] A time-space variational autoencoder encoding module, used for inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed;

[0017] The time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit;

[0018] The time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension;

[0019] The time-space self-attention mechanism unit is used to capture the time-space dependency relationship of the video to be processed from the time dimension and the space dimension;

[0020] The time-space upsampling unit is used to perform numerical interpolation on the video to be processed from the time dimension and the space dimension;

[0021] The time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution.

[0022] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0023] at least one processor;

[0024] and a memory communicatively coupled to the at least one processor;

[0025] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video processing method described in any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video processing method described in any embodiment of the present invention when executed.

[0027] The technical solution of the embodiment of the present invention obtains the video to be processed, and then inputs the video to be processed into the pre-trained time-space variational autoencoder to obtain the video encoding result, wherein the time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit; the time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension; the time-space self-attention mechanism unit is used to capture the spatiotemporal dependency relationship of the video to be processed from the time dimension and the space dimension; the time-space upsampling unit is used to numerically interpolate the video to be processed from the time dimension and the space dimension; the time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution. The above technical solution realizes the video dimensionality reduction of the time dimension and the space dimension through the time-space variational autoencoder, which effectively improves the video dimensionality reduction effect.

[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flowchart of a video processing method provided according to Embodiment 1 of the present invention;

[0031] Figure 2 is a flowchart of a video processing method provided according to Embodiment 2 of the present invention;

[0032] Figure 3 is a schematic diagram of a two-dimensional convolution kernel expansion provided according to an embodiment of the present invention;

[0033] Figure 4 is a flowchart of a video processing method provided according to Embodiment 3 of the present invention;

[0034] Figure 5 is a structural schematic diagram of a video processing device provided according to a fourth embodiment of the present invention;

[0035] Figure 6 It is a structural schematic diagram of an electronic device for implementing the video processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0038] Embodiment 1

[0039] Figure 1 This is a flowchart of a video processing method provided in the first embodiment of the present invention. This embodiment is applicable to the case where a video is compressed in a video generation task. The method can be executed by a video processing device. The video processing device can be implemented in the form of hardware and / or software. The video processing device can be configured in electronic devices such as terminals and / or servers. Figure 1 As shown, the method includes:

[0040] S110: Obtain the video to be processed.

[0041] In the embodiment of the present invention, the video to be processed refers to a video to be subjected to dimensionality reduction processing, and may be composed of multiple video frames.

[0042] Exemplarily, the video to be processed may be read from a preset storage path of the electronic device, or may be acquired in real time through a camera or other video acquisition device.

[0043] S120, inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed.

[0044] Among them, the time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit; the time-space three-dimensional convolution unit is used to convolve the processed video from the time dimension and the space dimension; the time-space self-attention mechanism unit is used to capture the spatiotemporal dependency relationship of the processed video from the time dimension and the space dimension; the time-space upsampling unit is used to numerically interpolate the processed video from the time dimension and the space dimension; the time-space downsampling unit is used to downsample the processed video from the time dimension and the space dimension through three-dimensional convolution.

[0045] It should be noted that the time-space variational autoencoder in the embodiment of the present invention is a three-dimensional variational autoencoder improved on the basis of the two-dimensional space variational autoencoder of the flux graph. The difference between the time-space variational autoencoder and the space variational autoencoder lies in the convolution unit, the self-attention mechanism unit, the upsampling unit and the downsampling unit. In other words, the convolution unit, the self-attention mechanism unit, the upsampling unit and the downsampling unit in the time-space variational autoencoder all add the time dimension.

[0046] Specifically, the time-space 3D convolution unit is a 3D convolution that can be used to convolve the processed video from the time dimension and the space dimension. The time-space self-attention mechanism unit is a self-attention mechanism that is used to capture the spatiotemporal dependencies of the processed video from the time dimension and the space dimension. The time-space upsampling unit can perform numerical interpolation of the processed video from the time dimension and the space dimension. The time-space downsampling unit can downsample the processed video from the time dimension and the space dimension through 3D convolution.

[0047] The technical solution of the embodiment of the present invention obtains the video to be processed, and then inputs the video to be processed into the pre-trained time-space variational autoencoder to obtain the video encoding result, wherein the time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit; the time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension; the time-space self-attention mechanism unit is used to capture the spatiotemporal dependency relationship of the video to be processed from the time dimension and the space dimension; the time-space upsampling unit is used to numerically interpolate the video to be processed from the time dimension and the space dimension; the time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution. The above technical solution realizes the video dimensionality reduction of the time dimension and the space dimension through the time-space variational autoencoder, which effectively improves the video dimensionality reduction effect.

[0048] Embodiment 2

[0049] Figure 2 A flowchart of a video processing method provided for Embodiment 2 of the present invention, the method of this embodiment can be combined with the various optional schemes in the video processing method provided in the above embodiments. The video processing method provided in this embodiment has been further optimized. Optionally, before inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, it also includes: obtaining weight information of the spatial variational autoencoder; copying the weight information of the spatial variational autoencoder to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder; training the expanded time-space variational autoencoder to obtain a trained time-space variational autoencoder.

[0050] like Figure 2 As shown, the method includes:

[0051] S210, obtaining weight information of the spatial variational autoencoder.

[0052] The spatial variational autoencoder refers to a two-dimensional variational autoencoder, which may be a two-dimensional spatial variational autoencoder of a flux graph or other two-dimensional spatial variational autoencoders, etc., and is not specifically limited here. The weight information refers to the model parameters of the spatial variational autoencoder, which may include but is not limited to two-dimensional convolution kernel weights and linear projection weights.

[0053] S220. Copy the weight information of the spatial variational autoencoder to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder.

[0054] Among them, the initial time-space variational autoencoder is a pre-built three-dimensional variational autoencoder architecture, which is compatible with the two-dimensional spatial variational autoencoder. The expanded time-space variational autoencoder is a variational autoencoder with the spatial compression capability of the spatial variational autoencoder.

[0055] It should be noted that by copying the weight information of the spatial variational autoencoder to the initial time-space variational autoencoder, an expanded time-space variational autoencoder can be obtained, so that the expanded time-space variational autoencoder has the same spatial compression capability as the spatial variational autoencoder from the beginning. After a small amount of subsequent training, a time-space variational autoencoder with excellent compression capability can be obtained, which greatly reduces the waste of computing resources.

[0056] S230: training the dilated time-space variational autoencoder to obtain a trained time-space variational autoencoder.

[0057] S240: Obtain the video to be processed.

[0058] S250, inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed.

[0059] Optionally, the weight information of the spatial variational autoencoder includes two-dimensional convolution kernel weight information; the initial time-space variational autoencoder includes an initial three-dimensional convolution kernel, and the initial three-dimensional convolution kernel includes a target position and a remaining position; accordingly, the weight information of the spatial variational autoencoder is copied to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder, including: copying the two-dimensional convolution kernel weight information to the target position of the initial three-dimensional convolution kernel, and filling zeros to the remaining position of the initial three-dimensional convolution kernel to obtain an expanded three-dimensional convolution kernel, and the expanded time-space variational autoencoder includes an expanded three-dimensional convolution kernel.

[0060] Among them, the two-dimensional convolution kernel weight information refers to the weight of the two-dimensional convolution kernel in the spatial variational autoencoder. Exemplarily, the tensor dimension of the two-dimensional convolution kernel can be 3×3, corresponding to H×W respectively. The initial three-dimensional convolution kernel refers to the convolution kernel of the three-dimensional convolution in the initial time-space variational autoencoder. Exemplarily, the tensor dimension of the initial three-dimensional convolution kernel can be 3×3×3, corresponding to T×H×W respectively.

[0061] The initial time-space variational autoencoder may include the same number of 3D convolutions as the 2D convolutions in the spatial variational autoencoder so as to be replicated one-to-one. The target position refers to the position for storing the weights of the 2D convolution kernel, and the remaining position refers to the position in the initial 3D convolution kernel other than the target position. The dilated 3D convolution kernel refers to a unit having the capability of the 2D convolution kernel in the spatial variational autoencoder.

[0062] In an embodiment of the present invention, Figure 3 is a schematic diagram of a two-dimensional convolution kernel expansion according to an embodiment of the present invention. Specifically, the target position can be the center position, the head position or the tail position of the initial three-dimensional convolution kernel, corresponding to Figure 3 In addition, the average value of the 2D convolution kernel can be copied to each position of the initial 3D convolution kernel to obtain the dilated 3D convolution kernel, corresponding to Figure 3 (D) in.

[0063] Optionally, the target position is the center position of the initial three-dimensional convolution kernel.

[0064] Among them, the mathematical expression of the center position weight replication scheme is as follows:

[0065] K3[0]=0;

[0066] K3[1]=K2;

[0067] K3[2]=0;

[0068] Among them, K2 represents the two-dimensional convolution kernel, K3 represents the initial three-dimensional convolution kernel, K3[0] represents the head position of the initial three-dimensional convolution kernel, K3[1] represents the center position of the initial three-dimensional convolution kernel, and K3[2] represents the tail position of the initial three-dimensional convolution kernel.

[0069] It should be noted that, compared with the head position weight replication scheme, the tail position weight replication scheme and the average value weight replication scheme, the compression capability of the time-space variational autoencoder obtained by the center position weight replication scheme is the best.

[0070] Optionally, the weight information of the spatial variational autoencoder includes a query vector of a two-dimensional self-attention mechanism, a key vector of the two-dimensional self-attention mechanism, and a numerical vector of the two-dimensional self-attention mechanism; the initial time-space variational autoencoder includes an initial three-dimensional self-attention mechanism unit, and the initial three-dimensional self-attention mechanism unit includes a query vector position, a key vector position, and a numerical vector position; accordingly, the weight information of the spatial variational autoencoder is copied to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder, including: copying the query vector of the two-dimensional self-attention mechanism to the query vector position of the initial three-dimensional self-attention mechanism unit, copying the key vector of the two-dimensional self-attention mechanism to the key vector position of the initial three-dimensional self-attention mechanism unit, and copying the numerical vector of the two-dimensional self-attention mechanism to the numerical vector position of the initial three-dimensional self-attention mechanism unit to obtain an expanded three-dimensional self-attention mechanism unit, and the expanded time-space variational autoencoder includes the expanded three-dimensional self-attention mechanism unit.

[0071] Among them, the query vector (Query) is the weight Q in the self-attention mechanism. The key vector (Key) is the weight K in the self-attention mechanism. The value vector (Value) is the weight V in the self-attention mechanism. The expanded three-dimensional self-attention mechanism unit refers to the self-attention mechanism unit that completes the weight replication.

[0072] For example, the mathematical expression of the weight replication of the self-attention mechanism unit is as follows:

[0073] 3D(Q)=2D(Q);

[0074] 3D(K)=2D(K);

[0075] 3D(V)=2D(V);

[0076] Among them, 2D(Q) represents the query vector of the two-dimensional self-attention mechanism, 3D(Q) represents the query vector position of the initial three-dimensional self-attention mechanism unit, and 3D(Q)=2D(Q) represents copying the query vector of the two-dimensional self-attention mechanism to the query vector position of the initial three-dimensional self-attention mechanism unit; 2D(K) represents the key vector of the two-dimensional self-attention mechanism, 3D(K) represents the key vector position of the initial three-dimensional self-attention mechanism unit, and 3D(K)=2D(K) represents copying the key vector of the two-dimensional self-attention mechanism to the key vector position of the initial three-dimensional self-attention mechanism unit; 2D(V) represents the numerical vector of the two-dimensional self-attention mechanism, 3D(V) represents the numerical vector position of the initial three-dimensional self-attention mechanism unit, and 3D(V)=2D(V) represents copying the numerical vector of the two-dimensional self-attention mechanism to the numerical vector position of the initial three-dimensional self-attention mechanism unit.

[0077] Optionally, the self-attention sequence of the initial three-dimensional self-attention mechanism unit contains spatial dimension information and temporal dimension information.

[0078] Exemplarily, the self-attention sequence of the two-dimensional self-attention mechanism is h×w, where h represents the video length, w represents the video width, and h×w represents the spatial dimension information. The self-attention sequence of the initial three-dimensional self-attention mechanism unit is t×h×w, where t represents the time dimension information.

[0079] The technical solution of the embodiment of the present invention can obtain an expanded time-space variational autoencoder by copying the weight information of the spatial variational autoencoder to the initial time-space variational autoencoder, so that the expanded time-space variational autoencoder has the same spatial compression capability as the spatial variational autoencoder from the beginning. After a small amount of subsequent training, a time-space variational autoencoder with excellent compression capability can be obtained, which greatly reduces the waste of computing resources.

[0080] Embodiment 3

[0081] Figure 4 A flowchart of a video processing method provided in Embodiment 3 of the present invention, the method of this embodiment can be combined with each optional scheme in the video processing method provided in the above embodiment. The video processing method provided in this embodiment is further optimized. Optionally, after inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, it also includes: training a diffusion model to be trained based on the video encoding result to obtain a trained diffusion model.

[0082] like Figure 4 As shown, the method includes:

[0083] S310: Obtain the video to be processed.

[0084] S320: Input the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed.

[0085] S330 , training the diffusion model to be trained based on the video encoding result to obtain a trained diffusion model.

[0086] Specifically, noise can be added to the video encoding result to obtain a noisy video encoding result, and then the diffusion model outputs a predicted noise based on the noisy video encoding result, calculates the training loss based on the predicted noise and the actual noise, and updates the diffusion model parameters until the training stop condition is met, thereby obtaining a trained diffusion model.

[0087] Regarding the training process of the diffusion model, the embodiments of the present invention can significantly reduce the resources required for the training of the diffusion model and reduce the difficulty of training the diffusion model by using a time-space variational autoencoder with efficient dimensionality reduction.

[0088] Embodiment 4

[0089] Figure 5 FIG. 4 is a schematic diagram of a video processing device provided in Embodiment 4 of the present invention. Figure 5 As shown, the device comprises:

[0090] The to-be-processed video acquisition module 410 is used to acquire the to-be-processed video;

[0091] The time-space variational autoencoder encoding module 420 is used to input the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed;

[0092] The time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit;

[0093] The time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension;

[0094] The time-space self-attention mechanism unit is used to capture the time-space dependency relationship of the video to be processed from the time dimension and the space dimension;

[0095] The time-space upsampling unit is used to perform numerical interpolation on the video to be processed from the time dimension and the space dimension;

[0096] The time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution.

[0097] The technical solution of the embodiment of the present invention obtains the video to be processed, and then inputs the video to be processed into the pre-trained time-space variational autoencoder to obtain the video encoding result, wherein the time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit; the time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension; the time-space self-attention mechanism unit is used to capture the spatiotemporal dependency relationship of the video to be processed from the time dimension and the space dimension; the time-space upsampling unit is used to numerically interpolate the video to be processed from the time dimension and the space dimension; the time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution. The above technical solution realizes the video dimensionality reduction of the time dimension and the space dimension through the time-space variational autoencoder, which effectively improves the video dimensionality reduction effect.

[0098] In some optional implementations, the video processing device further includes:

[0099] A weight information acquisition module is used to obtain the weight information of the spatial variational autoencoder;

[0100] A weight information copying module, used to copy the weight information of the spatial variational autoencoder to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder;

[0101] The dilated time-space variational autoencoder training module is used to train the dilated time-space variational autoencoder to obtain a trained time-space variational autoencoder.

[0102] In some optional embodiments, the weight information of the spatial variational autoencoder includes two-dimensional convolution kernel weight information; the initial time-space variational autoencoder includes an initial three-dimensional convolution kernel, and the initial three-dimensional convolution kernel includes a target position and a residual position;

[0103] Correspondingly, the weight information replication module includes:

[0104] A two-dimensional convolution kernel expansion unit is used to copy the two-dimensional convolution kernel weight information to the target position of the initial three-dimensional convolution kernel, and fill zeros to the remaining position of the initial three-dimensional convolution kernel to obtain an expanded three-dimensional convolution kernel. The expanded time-space variational autoencoder includes the expanded three-dimensional convolution kernel.

[0105] In some optional embodiments, the target position is the center position of the initial three-dimensional convolution kernel

[0106] In some optional embodiments, the weight information of the spatial variational autoencoder includes a query vector of a two-dimensional self-attention mechanism, a key vector of the two-dimensional self-attention mechanism, and a numerical vector of the two-dimensional self-attention mechanism; the initial time-space variational autoencoder includes an initial three-dimensional self-attention mechanism unit, and the initial three-dimensional self-attention mechanism unit includes a query vector position, a key vector position, and a numerical vector position;

[0107] Correspondingly, the weight information replication module includes:

[0108] A two-dimensional self-attention mechanism expansion unit is used to copy the query vector of the two-dimensional self-attention mechanism to the query vector position of the initial three-dimensional self-attention mechanism unit, copy the key vector of the two-dimensional self-attention mechanism to the key vector position of the initial three-dimensional self-attention mechanism unit, and copy the numerical vector of the two-dimensional self-attention mechanism to the numerical vector position of the initial three-dimensional self-attention mechanism unit to obtain an expanded three-dimensional self-attention mechanism unit. The expanded time-space variational autoencoder includes the expanded three-dimensional self-attention mechanism unit.

[0109] In some optional embodiments, the self-attention sequence of the initial three-dimensional self-attention mechanism unit contains spatial dimension information and temporal dimension information.

[0110] In some optional implementations, the video processing device further includes:

[0111] The diffusion model training unit is used to train the diffusion model to be trained based on the video encoding result to obtain a trained diffusion model.

[0112] The video processing device provided in the embodiment of the present invention can execute the video processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] Embodiment 5

[0114] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0115] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0116] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a video processing method, which includes:

[0118] Get the video to be processed;

[0119] Inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed;

[0120] The time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit;

[0121] The time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension;

[0122] The time-space self-attention mechanism unit is used to capture the time-space dependency relationship of the video to be processed from the time dimension and the space dimension;

[0123] The time-space upsampling unit is used to perform numerical interpolation on the video to be processed from the time dimension and the space dimension;

[0124] The time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution.

[0125] In some embodiments, the video processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video processing method in any other appropriate manner (e.g., by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video processing method, characterized in that: include: Get the video to be processed; Inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed; The time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit; The time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension; The time-space self-attention mechanism unit is used to capture the time-space dependency relationship of the video to be processed from the time dimension and the space dimension; The time-space upsampling unit is used to perform numerical interpolation on the video to be processed from the time dimension and the space dimension; The time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution.

2. The method according to claim 1, characterized in that: Before inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, the method further includes: Get the weight information of the spatial variational autoencoder; Copying the weight information of the spatial variational autoencoder to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder; The dilated time-space variational autoencoder is trained to obtain a trained time-space variational autoencoder.

3. The method according to claim 2, characterized in that The weight information of the spatial variational autoencoder includes two-dimensional convolution kernel weight information; the initial time-space variational autoencoder includes an initial three-dimensional convolution kernel, and the initial three-dimensional convolution kernel includes a target position and a residual position; Accordingly, the weight information of the spatial variational autoencoder is copied to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder, including: The two-dimensional convolution kernel weight information is copied to the target position of the initial three-dimensional convolution kernel, and zeros are filled to the remaining position of the initial three-dimensional convolution kernel to obtain an expanded three-dimensional convolution kernel, and the expanded time-space variational autoencoder includes the expanded three-dimensional convolution kernel.

4. The method according to claim 3, characterized in that The target position is the center position of the initial three-dimensional convolution kernel.

5. The method according to claim 2, characterized in that: The weight information of the spatial variational autoencoder includes a query vector (Query) of a two-dimensional self-attention mechanism, a key vector (Key) of the two-dimensional self-attention mechanism, and a numerical vector (Value) of the two-dimensional self-attention mechanism; the initial time-space variational autoencoder includes an initial three-dimensional self-attention mechanism unit, and the initial three-dimensional self-attention mechanism unit includes a query vector position, a key vector position, and a numerical vector position; Accordingly, the weight information of the spatial variational autoencoder is copied to the initial time-space variational autoencoder to obtain an expanded time-space variational autoencoder, including: The query vector of the two-dimensional self-attention mechanism is copied to the query vector position of the initial three-dimensional self-attention mechanism unit, the key vector of the two-dimensional self-attention mechanism is copied to the key vector position of the initial three-dimensional self-attention mechanism unit, and the numerical vector of the two-dimensional self-attention mechanism is copied to the numerical vector position of the initial three-dimensional self-attention mechanism unit to obtain an expanded three-dimensional self-attention mechanism unit. The expanded time-space variational autoencoder includes the expanded three-dimensional self-attention mechanism unit.

6. The method according to claim 5, characterized in that The self-attention sequence of the initial three-dimensional self-attention mechanism unit contains spatial dimension information and temporal dimension information.

7. The method according to any one of claims 1 to 6, characterized in that: After inputting the video to be processed into the pre-trained time-space variational autoencoder to obtain the video encoding result, the method further includes: The diffusion model to be trained is trained based on the video encoding result to obtain a trained diffusion model.

8. A video processing device, characterized in that: include: A module for obtaining videos to be processed, used for obtaining videos to be processed; A time-space variational autoencoder encoding module, used for inputting the video to be processed into a pre-trained time-space variational autoencoder to obtain a video encoding result, wherein the dimension of the video encoding result is smaller than the dimension of the video to be processed; The time-space variational autoencoder includes a time-space three-dimensional convolution unit, a time-space self-attention mechanism unit, a time-space upsampling unit and a time-space downsampling unit; The time-space three-dimensional convolution unit is used to convolve the video to be processed from the time dimension and the space dimension; The time-space self-attention mechanism unit is used to capture the time-space dependency relationship of the video to be processed from the time dimension and the space dimension; The time-space upsampling unit is used to perform numerical interpolation on the video to be processed from the time dimension and the space dimension; The time-space downsampling unit is used to downsample the video to be processed from the time dimension and the space dimension through three-dimensional convolution.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video processing method according to any one of claims 1 to 7 when executed.

Images