Video processing method and device, and electronic device and computer storage medium

Abstract

A video processing method and device, and an electronic device and a computer storage medium. The video processing method comprises: training a diffusion model; encoding a video to be outpainted and inputting the encoded video into the diffusion model; and the diffusion model performing noise addition, noise prediction and denoising on said video, and decoding said video to obtain an outpainted video. The step of training a diffusion model comprises: extracting odd-numbered feature maps from odd-numbered frames of a video sample to which noise has been added; extracting even-numbered feature maps from even-numbered frames of the video sample to which noise has been added; acquiring odd-indexed transformer blocks and even-indexed transformer blocks in a diffusion model; the odd-numbered transformer blocks performing attention computation on the odd-numbered feature maps to obtain queries, keys and values of the odd-numbered transformer blocks; and the even-numbered transformer blocks performing attention computation on the even-numbered feature maps to obtain queries, keys and values of the even-numbered transformer blocks.

Description

Video processing method and device, electronic device, and computer storage medium TECHNICAL FIELD

[0001] Embodiments of the present disclosure belong to the technical field of video processing, and particularly relate to a video processing method and device, an electronic device, and a computer storage medium. BACKGROUND

[0002] At present, diffusion models are increasingly applied in the field of image and video editing, but in the field of video editing, especially in the application of video outpainting editing, diffusion models still need to be improved. SUMMARY

[0003] Embodiments of the present disclosure provide a video processing method, which comprises training a diffusion model.

[0004] Feature encoding is performed on the video to be outpainted to obtain a latent feature sequence of the video to be outpainted.

[0005] The latent feature sequence of the video to be outpainted is input into the trained diffusion model.

[0006] The diffusion model adds noise to the latent feature sequence of the video to be outpainted to make it a random noise matrix.

[0007] Noise prediction and denoising are performed on the random noise matrix to obtain a latent feature sequence of an outpainted video.

[0008] The latent feature sequence of the outpainted video is decoded to obtain the outpainted video.

[0009] The training of the diffusion model comprises:

[0010] Odd feature maps are extracted from odd frames of the latent feature sequence of the noisy video sample, and even feature maps are extracted from even frames of the latent feature sequence of the noisy video sample.

[0011] The transformer blocks ranked as odd numbers and the transformer blocks ranked as even numbers in the diffusion model are obtained.

[0012] The transformer block ranked as an odd number performs attention calculation on the odd feature map to obtain the query, key, and value of the transformer block ranked as an odd number.

[0013] The transformer block ranked as an even number performs attention calculation on the even feature map to obtain the query, key, and value of the transformer block ranked as an even number.

[0014] In some embodiments, after adding noise to the latent feature sequence of the video to be expanded to obtain a random noise matrix, and before performing noise prediction and denoising on the random noise matrix to obtain the latent feature sequence of the expanded video, the method further comprises:

[0015] inputting additional control condition information into the random noise matrix;

[0016] The additional control condition information includes edge, pose and depth information of the image.

[0017] In some embodiments, the diffusion model adds noise to the latent feature sequence of the video to be expanded to obtain a random noise matrix, comprising:

[0018] extracting latent vector noise from the latent feature sequence of the video to be expanded to obtain a random noise matrix of the video to be expanded;

[0019] According to the content prompt of the expansion of the video to be expanded, the Gaussian noise of the expanded video is randomly sampled, and the region corresponding to the video to be expanded in the Gaussian noise of the expanded video is replaced with the random noise matrix of the video to be expanded.

[0020] In some embodiments, further comprising: weighting and smoothing the expanded regions of the previous and subsequent frames in the replaced Gaussian noise of the expanded video, and the formula of the weighting and smoothing is:

[0021] wherein M represents a mask matrix, the mask matrix value corresponding to the video to be expanded is 1, and the mask value of the expanded region is 0; x t represents the t-th frame of the video to be expanded, x t-1 represents the t-1-th frame of the video to be expanded.

[0022] In some embodiments, the formula for the diffusion model to add noise to the latent feature sequence of the video to be expanded to obtain a random noise matrix is:

[0023] wherein t is a timestamp, t∈{1,…,T}, q(x t |x t-1 ) is a conditional probability; α t is a hyperparameter, α t =1-β t , β gradually increases from 0.0001 to 0.02 over time; I is the noise of the video to be expanded before or after expansion, and I is an identity matrix;

[0024] x trepresents the latent feature sequence of all frames of the video to be expanded in the noise adding step corresponding to time t; x t-1 represents the latent feature sequence of all frames of the video to be expanded in the noise adding step corresponding to time t-1; the noise adding step corresponding to time t and the noise adding step corresponding to time t-1 are two adjacent steps; and x t only related to x t-1 .

[0025] In some embodiments, the noise prediction and denoising of the random noise matrix to obtain the latent feature sequence of the expanded video comprises:

[0026] performing convolution calculation on the random noise matrix to obtain a first result;

[0027] calculating attention weight on the first result and performing weighted summation to obtain a second result;

[0028] calculating attention according to the second result and the content prompt for expanding the video to be expanded to obtain a third result;

[0029] performing full connection calculation on the third result to obtain the latent feature sequence of the expanded video.

[0030] In some embodiments, before denoising the random noise matrix, further comprising:

[0031] predicting the Gaussian noise in the random noise matrix to obtain a predicted noise;

[0032] comparing the predicted noise with the added Gaussian noise in the latent feature sequence of the video to be expanded to calculate the difference between them to obtain a noise loss;

[0033] calculating the gradient according to the noise loss and updating the parameters of the diffusion model according to the gradient;

[0034] The parameters of the diffusion model include the query, key and value of the transformer block.

[0035] In some embodiments, before the diffusion model adds noise to the latent feature sequence of the video to be expanded to become a random noise matrix, further comprising mapping the latent feature sequence of the video to be expanded to a latent space;

[0036] The size of the latent feature sequence of the video to be expanded in the latent space is 1 / 2 n , n = 1, 2, 3…, n is an integer.

[0037] In some embodiments, the extended video has a video region to be extended and an extended region, the extended region is located at an arbitrary position outside the video region to be extended.

[0038] The embodiments of the present disclosure further provide a video processing device, comprising an encoder configured to encode a video to be extended, obtain a latent feature sequence of the video to be extended;

[0039] a diffusion model block configured to add noise to the latent feature sequence of the video to be extended to obtain a random noise matrix, perform noise prediction and denoising on the random noise matrix to obtain a latent feature sequence of an extended video, wherein an odd transformer block in the diffusion model block has determined query, key and value, and an even transformer block has determined query, key and value;

[0040] a decoder configured to decode the latent feature sequence of the extended video to obtain the extended video.

[0041] In some embodiments, the diffusion model block further comprises a ControlNet model block configured to input additional control condition information into the random noise matrix;

[0042] The additional control condition information comprises edge, pose and depth information of an image.

[0043] In some embodiments, the diffusion model block comprises a UNet model block;

[0044] The UNet model block comprises a plurality of network layers;

[0045] Each network layer comprises two transformer blocks;

[0046] The transformer block comprises:

[0047] a convolution block configured to perform convolution calculation on the random noise matrix to obtain a first result;

[0048] an inter-frame attention calculation block configured to calculate attention weights on the first result and perform weighted summation to obtain a second result;

[0049] a cross-attention calculation block configured to calculate attention according to the second result and content prompts for extending the video to be extended to obtain a third result;

[0050] a fully connected calculation block configured to perform fully connected calculation on the third result to obtain the latent feature sequence of the extended video.

[0051] The embodiment of the present disclosure further provides an electronic device, comprising a memory and a processor, at least one program is stored in the memory, wherein the at least one program is loaded and executed by the processor to realize the video processing method.

[0052] The embodiment of the present disclosure further provides a non-transitory computer storage medium, which stores a computer program, wherein the computer program realizes the video processing method when executed by a processor.

[0053] The video processing method provided by the embodiment of the present disclosure can reduce the calculation amount of the diffusion model in the training process, reduce the consumption of a large amount of computing resources, and thus improve the training efficiency and accuracy of the diffusion model used for video edge expansion, and further improve the efficiency and accuracy of video edge expansion processing.

[0054] The electronic device provided by the embodiment of the present disclosure can reduce the calculation amount of video edge expansion processing, thereby reducing the consumption of a large amount of computing resources, and further improving the efficiency and accuracy of video edge expansion processing.

[0055] The non-transitory computer storage medium provided by the embodiment of the present disclosure can reduce the calculation amount of video edge expansion processing, thereby reducing the consumption of a large amount of computing resources, and further improving the efficiency and accuracy of video edge expansion processing. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings are included to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, which are used to explain the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing in detail the specific example embodiments with reference to the accompanying drawings, in which:

[0057] FIG. 1a is a flowchart of a video processing method in the embodiment of the present disclosure.

[0058] FIG. 1b is a specific flowchart of step S1 in the video processing method in the embodiment of the present disclosure.

[0059] FIG. 1c is a whole network structure diagram of diffusion model training in the embodiment of the present disclosure.

[0060] FIG. 1d is a schematic diagram of a calculation principle of a transformer block in the diffusion model in the embodiment of the present disclosure.

[0061] FIG. 1e is another schematic diagram of the calculation principle of the transformer block in the diffusion model according to an embodiment of the present disclosure.

[0062] FIG. 1f is a specific flowchart of step S4 in the video processing method according to an embodiment of the present disclosure.

[0063] FIG. 2a is a schematic diagram of a neural network block before adding a ControlNet model block according to an embodiment of the present disclosure.

[0064] FIG. 2b is a schematic diagram of a neural network block after adding a ControlNet model block according to an embodiment of the present disclosure.

[0065] FIG. 3 is a schematic diagram of the network structure of the ControlNet model block added to the Stablediffusion model according to an embodiment of the present disclosure.

[0066] FIG. 4 is an example schematic diagram of video expansion according to an embodiment of the present disclosure.

[0067] FIG. 5a is a schematic diagram of the process of adding noise and denoising an image by the diffusion model according to an embodiment of the present disclosure.

[0068] FIG. 5b is a schematic diagram of the forward noise adding process of the diffusion model according to an embodiment of the present disclosure.

[0069] FIG. 5c is a schematic diagram of the reverse denoising process of the diffusion model according to an embodiment of the present disclosure.

[0070] FIG. 5d is a schematic diagram of the principle of the reverse diffusion process of the diffusion model according to an embodiment of the present disclosure.

[0071] FIG. 6 is a schematic diagram of the background smoothing technique in Text2Video-zero.

[0072] FIG. 7 is a schematic diagram of the cross-attention calculation principle according to an embodiment of the present disclosure.

[0073] FIG. 8a is a schematic diagram of the Epoch iteration in the diffusion model training process according to an embodiment of the present disclosure.

[0074] FIG. 8b is a schematic diagram of the process of each training step in the diffusion model training process according to an embodiment of the present disclosure.

[0075] FIG. 9 is a schematic diagram of the principle block diagram of the video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] In order for those skilled in the art to better understand the technical solutions of the embodiments of the present disclosure, a video processing method and device, an electronic device and a computer storage medium provided by the embodiments of the present disclosure are further described in detail below with reference to the drawings and specific embodiments.

[0077] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the disclosure to those skilled in the art.

[0078] Embodiments of the present disclosure are not limited to the embodiments shown in the drawings, but include modifications of configurations formed based on manufacturing processes. Therefore, the regions exemplified in the drawings have a schematic property, and the shape of the regions shown in the drawings exemplifies a specific shape of the regions, but is not intended to be restrictive.

[0079] In public places such as shopping malls, exhibition halls, and the like involving video playing, the size of the original video may not be suitable for the size of the actual display device, and direct scaling may cause the deformation of the video content, so it is necessary to edit the original video content to a certain extent, such as outpainting the original video content.

[0080] At present, when using a diffusion model to edit the video outpainting, since the diffusion model processes video data, the processing and training of the video are relatively large in calculation amount compared with picture calculation, which consumes a large amount of computing resources, resulting in that slightly increasing the resolution of the video cannot be processed and trained on a GPU (Graphics Processing Unit, graphics processor, display core, microprocessor dedicated to image and graphics related operation) with a normal video memory size (such as a 24G A100 graphics card).

[0081] In order to solve the problem of large calculation amount when using a diffusion model to edit the video outpainting, embodiments of the present disclosure provide a video processing method, as shown in FIGS. 1a, 1b, 1c, 1d, 1e and 1f, which comprises the following steps.

[0082] The step comprises the following steps: step S11: extracting odd feature maps for odd frames of the noise-added video sample latent feature sequence; and extracting even feature maps for even frames of the noise-added video sample latent feature sequence.

[0083] Step S12: obtaining a transformer block ranked as odd and a transformer block ranked as even in the diffusion model.

[0084] Step S13: The odd transformer block performs attention calculation on the odd feature map to obtain the query, key and value of the odd transformer block; and the even transformer block performs attention calculation on the even feature map to obtain the query, key and value of the even transformer block.

[0085] Step S2: Feature encoding is performed on the video to be expanded, to obtain a latent feature sequence of the video to be expanded.

[0086] Step S3: The latent feature sequence of the video to be expanded is input into the trained diffusion model; the diffusion model adds noise to the latent feature sequence of the video to be expanded, to make it a random noise matrix.

[0087] Step S4: Noise prediction and denoising are performed on the random noise matrix, to obtain a latent feature sequence of the expanded video.

[0088] Step S5: The latent feature sequence of the expanded video is decoded, to obtain the expanded video.

[0089] Through feature encoding, the latent feature of each frame in the video sample can be obtained; the latent features of the frames of the video sample constitute a latent feature sequence of the video sample. Through feature encoding, the latent feature of each frame in the video to be expanded can also be obtained; the latent features of the frames of the video to be expanded constitute a latent feature sequence of the video to be expanded. Through feature encoding, the computational complexity of subsequent calculations can be reduced, and the required computing resources can be reduced.

[0090] In this embodiment, the training of the diffusion model uses a cross-frame attention mechanism to perform attention calculation on the latent feature sequence of the video sample. The cross-frame attention mechanism is a self-attention mechanism, which is derived from the study of human vision. By simulating the ability of human selective attention, the model can prioritize processing key parts when dealing with a large amount of information. The cross-frame attention mechanism is an attention mechanism applied in the field of deep learning and artificial intelligence, which allows the model to focus on key information of the current frame or time step while ignoring unimportant information when processing video or sequence data. The cross-frame attention mechanism can improve the efficiency and accuracy of information processing. When processing video or sequence data, the model not only focuses on the information of the current frame, but also considers the relationship between the previous and subsequent frames to improve the recognition and response speed of key information.

[0091] In some embodiments, as shown in FIGS. 1c and 1d, the diffusion model adopts a UNet model, which adds noise to the latent feature sequence of the video sample, predicts the noise, and removes the noise. The UNet model is a convolutional neural network model for semantic segmentation, which is based on a fully convolutional network and is modified and extended to produce more accurate segmentation results using fewer training images. The main execution process of the UNet model is to first extract features by convolution downsampling, and then upsample to obtain an image corresponding to each pixel point. The UNet model uses multiple compressed feature layers of different levels for upsampling, thereby outputting more refined image segmentation results. The fusion of multiple levels of features helps to improve the segmentation accuracy of the model.

[0092] As shown in FIG. 1d, the UNet model includes multiple network layers, each of which includes two transformer blocks, each of which includes a convolution block (Convolution), a cross-frame attention calculation block (Cross-Frame Attention), and a cross-attention calculation block (Cross Attention). In some embodiments, the diffusion model uses video samples with 16 frames for training, and each network layer of the UNet model includes two transformer blocks, which can completely divide the odd and even frames of the video sample, facilitating data processing.

[0093] Each network layer of the UNet model includes an odd-numbered transformer block and an even-numbered transformer block. The transformer block is a neural network model for natural language processing, which mainly converts input sequences into vectors and encodes and decodes them using multiple transformer structures. The transformer block consists of two core parts: an encoder and a decoder. The encoder encodes the input sequence into a vector, and the decoder restores the vector to the original sequence. The transformer block assigns a weight to each position in the input sequence, and then outputs these weighted position vectors.

[0094] In this embodiment, the odd-numbered transformer block calculates the query, key, and value in the transformer block from the odd frames of the video sample latent feature sequence and all other odd frames of the video sample latent feature sequence. The even-numbered transformer block calculates the query, key, and value in the transformer block from the even frames of the video sample latent feature sequence and all other even frames of the video sample latent feature sequence.

[0095] In some embodiments, the diffusion model is trained, for example: assuming that the feature map dimension feature map corresponding to the latent feature of 4 frame video samples is [4, 64, 32, 32], through the slicing operation in python programming, the odd feature map is extracted by program operation odd_feature_map = feature_map[1::2]; the even feature map is extracted by program operation even_feature_map = feature_map[::2]. Among them, feature_map[1::2] means selecting feature maps with an interval of 2 and an index starting from 1; feature_map[::2] means selecting feature maps with an interval of 2 and an index starting from 0. For the transformer block, the transformer blocks sorted as odd and even can be taken out through the for loop. The odd transformer block only calculates the odd feature map, and the even transformer block only calculates the even feature map. Among them, the feature map dimension [4, 64, 32, 32] corresponding to the latent feature of 4 frame video samples, 4 represents the number of frames; 64 represents the number of channels; 32, 32 represents the resolution size, that is, the resolution of 32x32.

[0096] As shown in FIG. 1e, Query (Q), Key (K) and Value (V) are the key components of the inter-frame attention mechanism. Query (Q), Key (K) and Value (V) are three matrices used to calculate attention weights in the self-attention mechanism, which are usually obtained by linear transformation of the input sequence. Query (Q matrix) is used to generate query vectors, each of which represents a patch's query in the attention mechanism, i.e., to find information related to the current patch. Key (K matrix) is used to generate key vectors, each of which represents a patch's key in the attention mechanism, i.e., to represent the relationship between the current patch and other patches. Value (V matrix) is used to generate value vectors, each of which represents a patch's value in the attention mechanism, i.e., to represent the feature information of the current patch. Query (Q), Key (K) and Value (V) in the inter-frame attention mechanism all come from the same set of elements. The working principle of the inter-frame attention mechanism is as follows: ① input representation: given an input sequence (such as a sequence of latent features of a video sample), obtain the query, key and value matrices through linear transformation; ② calculate attention scores: for the query vector (Q_i) of each position (i), calculate the dot product with all key vectors (K), and then apply the Softmax function to obtain the weights; ③ weighted sum: apply the weights to the value vectors (V) to obtain the output vectors (O_i).

[0097] Referring to FIG. 1e, in the self-attention mechanism, the input sequence is first transformed by three different linear transformations to obtain the query matrix Q, the key matrix K and the value matrix V, which are used to calculate the attention weights, and then the input sequence is weighted summed to obtain the final representation. Among them, the dot product of the query matrix Q and the key matrix K is the attention weight matrix A. QKV of the video image is used to calculate attention, which can capture non-local information in the video image, i.e., the association between different positions, so as to better model the dependence between pixels.

[0098] In FIG. 1e, the X input matrix is the latent feature sequence of the video to be expanded after adding noise; the Y output matrix is the output matrix obtained by sequentially calculating and processing the X input matrix through each odd and even transformer block in the diffusion model. The dimensions of the X input matrix and the Y output matrix are the same, i.e., the input dimension and the output dimension are the same.

[0099] In this embodiment, after the diffusion model training is completed, the queries, keys and values of the odd transformer blocks and the queries, keys and values of the even transformer blocks are determined, and in the subsequent edge expansion process of the video to be expanded, when the video to be expanded is input into the diffusion model, the diffusion model performs noise prediction and denoising processing on the noisy video to be expanded through the odd or even transformer blocks with determined queries, keys and values, and finally obtains the expanded video.

[0100] In this embodiment, in the training process of the diffusion model, the odd transformer blocks perform attention calculation on the odd feature maps to obtain the queries, keys and values of the odd transformer blocks, and the even transformer blocks perform attention calculation on the even feature maps to obtain the queries, keys and values of the even transformer blocks, which can reduce the calculation amount of the diffusion model in the training process, reduce the consumption of a large amount of computing resources, and thus improve the training efficiency and accuracy of the diffusion model for video edge expansion, and further improve the efficiency and accuracy of video edge expansion processing.

[0101] In some embodiments, as shown in FIGS. 1c and 1d, in order to reduce the complexity of the diffusion model as much as possible, this embodiment designs a sparse attention calculation method by referring to the ProPainter (video repair, which combines the advantages of image and feature repair, and efficient conversion technology) idea. Since there is a high degree of redundancy and repeated texture in adjacent frames of the video sample latent feature sequence, it is not necessary to mark all frames as queries, keys and values, and only alternate time frames can be selected as queries, keys and values. That is, in each odd-numbered Transformer block, only odd frames are activated to participate in the attention calculation of the queries, keys and values with odd frames; in each even-numbered Transformer block, only even frames are activated to participate in the attention calculation of the queries, keys and values with even frames. As shown in FIGS. 1c and 1d, the UNet model has 0, 1, 2…n transformer blocks, and when calculating the attention of each frame in the video sample latent feature sequence, the calculation of the queries, keys and values of the odd-numbered transformer blocks only comes from the odd frames of the video sample latent feature sequence, and the calculation of the queries, keys and values of the even-numbered transformer blocks only comes from the even frames of the video sample latent feature sequence. Through this inter-frame attention mechanism, the calculation amount of attention calculation is reduced.

[0102] In some embodiments, after adding noise to the latent feature sequence of the video to be extended to obtain a random noise matrix, and before performing noise prediction and denoising on the random noise matrix to obtain the latent feature sequence of the extended video, the method further comprises: inputting additional control condition information into the random noise matrix; the additional control condition information includes edge, pose and depth information of the image.

[0103] In some embodiments, if additional conditions are needed to control the generated content, one of the most direct ways is to fine-tune the diffusion model, but this requires more training data and more computing resources, because if the amount of data is small, it is easy to cause overfitting of the diffusion model and even destroy the expression ability of the original pre-trained model. In order to avoid this problem, as shown in FIG. 1c, a controllable module can be added to the diffusion model to achieve the purpose of adding additional information while maintaining the powerful generation ability of the pre-trained model.

[0104] In some embodiments, the controllable module added to the diffusion model can be a ControlNet model block. FIG. 2a is a neural network block before adding the ControlNet model block, where x is the input of the neural network block and y is the output of the neural network block. FIG. 2b is a neural network block after adding the ControlNet model block. The role of the ControlNet model block is to achieve precise condition control. By combining a stable diffusion model, it can perform more precise control in detail. The ControlNet model block can enhance the pre-trained image diffusion model through specific task conditions, including using a "trainable copy" and a "locked copy" of the pre-trained model, and a set of input conditions (c) to control the output (y c ) of the neural network block. In FIG. 2b, x is the input of the neural network block and the input of the trainable copy in the ControlNet model block; c is the input condition of the ControlNet model block; the "locked copy" of the pre-trained model used by the ControlNet model block refers to that the ControlNet model block uses the input x of the neural network block; y c ) refers to the output of the neural network block under the control of the ControlNet model block. The zero convolution at the input end of the trainable copy in the ControlNet model block refers to that the input condition c is first subjected to zero convolution operation and then input into the trainable copy; the zero convolution at the output end of the trainable copy in the ControlNet model block refers to that the output of the trainable copy is subjected to zero convolution operation and then output.

[0105] In some embodiments, as shown in FIG. 3, the network structure of the ControlNet model block is introduced on the basis of the pre-trained large model, which is responsible for learning to map additional conditional information into the pre-trained large model with fixed parameters. For example, a ControlNet model block (b) is added to the Stable diffusion (SD) model (a) used in the noise adding process of the UNet model. The left part (a) of FIG. 3 is the pre-trained Stable diffusion model, and its parameters (Prompt, Time, Input) are fixed. The right part (b) of FIG. 3 is the ControlNet model block. The ControlNet model block and the Stable diffusion model have consistent Encoder Block and Middle Block. The initialization parameters (Prompt, Time, Input) of the ControlNet model block are directly copied from the Stable diffusion model and updated in the subsequent training process. The trained ControlNet model block can map additional conditional information into the fixed Stable diffusion model. Since the parameters of the Stable diffusion model do not change during training, it can maintain its strong generation ability and avoid overfitting problems caused by small data sets.

[0106] In some embodiments, the Stable diffusion model is a machine learning model based on latent diffusion, which can convert text descriptions into high-quality images. The Stable diffusion model is a latent diffusion model that generates image samples of interest such as human faces by gradually removing random Gaussian noise. The advantage of the Stable diffusion model is its high-quality image generation and flexibility, which can be applied to various scenarios. The model can generate high-quality images with good fidelity and detail, better meeting actual needs.

[0107] In this embodiment, the introduction of the ControlNet model block can better control the generation results of the Stable diffusion model. The ControlNet model block in this embodiment is consistent with the standard ControlNet model structure, and the internal detailed structure is not introduced in detail here.

[0108] In some embodiments, as shown in FIG. 1c, the additional control condition information of the ControlNet model block includes the edge canny, pose and depth information depth of the image.

[0109] In some embodiments, the diffusion model adds noise to the latent feature sequence of the video to be extended, so that it becomes a random noise matrix, including: extracting a latent vector noise from the latent feature sequence of the video to be extended, obtaining a random noise matrix of the video to be extended; according to the content prompt for extending the video to be extended, randomly sampling the Gaussian noise of the extended video, and replacing the region corresponding to the video to be extended in the Gaussian noise of the extended video with the random noise matrix of the video to be extended.

[0110] In some embodiments, as shown in FIG. 1c, for a video sample containing t frames, represented as First, it is sent into the pre-trained encoder ε to obtain the corresponding latent feature sequence For a picture, the feature obtained after the encoder is x0=ε(I); then the latent feature sequence is subjected to the noise adding operation (i.e. Diffusion process) in the diffusion model (i.e. UNet model). In order to realize the extension of the video sample, in the inference stage (i.e. video sample extension stage), given a video sample, assuming its length and width size is h*w (here the length and width size refers to the resolution), in order to extend the video sample to a length and width size of H*W (here the length and width size refers to the resolution), first, the h*w video sample is subjected to DDIM inversion (diffusion model acceleration sampling algorithm inversion) noise adding, which is equivalent to extracting the latent vector noise from the original video sample, obtaining a latent feature sequence of h / 8*w / 8; then according to the content prompt (text prompts) for extending the original video sample, randomly sampling the H / 8*W / 8 Gaussian noise, and replacing the h / 8*w / 8 region in the H / 8*W / 8 Gaussian noise with the latent feature sequence of the original video sample. Such noise adding can ensure that the region corresponding to the original video sample remains unchanged after video extension.

[0111] In this embodiment, as shown in FIG. 4, for example: the provided original video sample is a video of a pet dog walking, and the content prompt word (text prompts) for extending the original video sample is "a cute dog, garden, flowers". Since the prompt word and the original video sample have small differences, the extended video and the original video can maintain consistency in content to a certain extent.

[0112] In this embodiment, as shown in FIG. 1c, the UNet model trained is used to predict the noise of the replaced Gaussian noise and perform denoising, and finally obtain the denoised extended video features, and then obtain the final extended video through the decoding operation corresponding to the encoding.

[0113] In some embodiments, the diffusion model adds noise to the latent feature sequence of the video to be extended, so that it becomes a random noise matrix, and the formula is:

[0114] where t is the timestamp, t e {1,...,T}, q(x t |x t-1 ) is the conditional probability; a t is a hyperparameter, a t = 1-β t , β gradually increases from 0.0001 to 0.02 over time; I is the noise of the video to be extended before or after the extension, and I is the unit matrix.

[0115] x t represents the latent feature sequence of all frames of the video to be extended in the corresponding noise adding step at time t; x t-1 represents the latent feature sequence of all frames of the video to be extended in the corresponding noise adding step at time t-1; the corresponding noise adding step at time t and the corresponding noise adding step at time t-1 are two adjacent steps; and x t only relates to x t-1 .

[0116] In this embodiment, the target of the training of the diffusion model for video extension (i.e. the image noise output after the UNet model calculation) is That is, it is hoped that the UNet model can predict the noise in the input image.

[0117] In some embodiments, as shown in FIG. 5a, a flowchart of adding noise and denoising to an image by the diffusion model is shown; in the forward diffusion process, the diffusion model continuously adds Gaussian noise to an image until it becomes a random noise matrix. In the reverse diffusion process, the diffusion model performs a denoising process to gradually denoise a random noise matrix until an image is generated.

[0118] In some embodiments, as shown in FIG. 5b, the forward noise adding process of the diffusion model is a process of continuously adding noise to the image. For example, Gaussian noise is added multiple times in a cat picture until the picture becomes a random noise matrix. It can be seen that for the initial data, the diffusion step number K is set, and a certain amount of noise is added at each step. If K is large enough, the initial data can be converted into a random noise matrix.

[0119] In some embodiments, as shown in FIG. 5c, the reverse denoising process of the diffusion model is exactly the opposite of the forward diffusion process, which is a process of continuously denoising the image. For example, a random Gaussian noise matrix is put through the reverse diffusion process of the diffusion model, the noise is predicted and gradually denoised, and finally a clear picture of a small villa is generated.

[0120] In some embodiments, as shown in FIG. 5d, it is a complete illustration of the principle of the reverse diffusion process of the diffusion model. The specific process is as follows: 1. Sample a Gaussian noise image, and input the noise image (e.g. an image with 1000 steps of noise added) into the diffusion model; 2. Iteratively denoise the image, i.e. the diffusion model predicts the noise image added at step 1000 for the current image, then subtracts the predicted noise from the input noise image x1000 to get a slightly clearer image x999 without the 1000th step noise, then inputs x999 into the diffusion model to predict the noise again, then subtracts the noise to get x998, and so on, finally gets a completely denoised clear image; 3. Output the denoised image.

[0121] Wherein, the noise distribution predicted and removed at each step needs to be learned by the diffusion model in training. The forward noise adding process and the reverse denoising process of the diffusion model are both for the training of the diffusion model, and the training target is to regress the noise predicted by the diffusion model each time and the noise actually added each time, so that the diffusion model can accurately predict the real noise actually added each time.

[0122] In some embodiments, the diffusion model training method further comprises: weighting and smoothing the edge expansion regions of the two frames in the Gaussian noise of the replaced edge expansion video, and the weighting and smoothing formula is:

[0123] Wherein, M represents the mask matrix, the mask matrix value corresponding to the video to be expanded is 1, and the mask value of the edge expansion region is 0; x t represents the t-th frame of the video to be expanded, x t-1 represents the t-1-th frame of the video to be expanded.

[0124] In this embodiment, since the frames of the video to be expanded have continuity, the change of the background of the video to be expanded should also be continuous. In order to ensure the smooth transition of the background of the video to be expanded, the background smoothing technique of Text2Video-zero in FIG. 6 is referred to in the video expansion process in this embodiment, but it is modified. In text2video-zero, the initial frame is used to generate all subsequent frames, that is, the initial frame is the core frame, so each frame of the background smoothing needs to be weighted with the image obtained by warping the initial frame. The difference between this embodiment and the background smoothing technique of Text2Video-zero in FIG. 6 is that: for the replaced part (i.e. the part of h / 8*w / 8, which can be regarded as the foreground), it is reserved after being multiplied by the mask with a value of 1 (i.e. ); and for the expanded part, the t-1 frame and the t frame are weighted, the weighted background (i.e. ) and the previously reserved foreground are fused to obtain a complete frame image, which reduces the jumping of the background and ensures the consistency and smoothness of the expansion area of the frames of the video to be expanded as much as possible, thereby improving or avoiding the shaking of the expanded content, and further ensuring the display effect of the video.

[0125] In the background smoothing technique of Text2Video-zero in FIG. 6, Salient Object Detector represents salient object detection, which aims to identify and segment the most important or most significant objects in the image.

[0126] In some embodiments, as shown in FIGS. 1d and 1f, step S4: noise prediction and denoising are performed on the random noise matrix to obtain the latent feature sequence of the expanded video, including: step S41: performing convolution calculation on the random noise matrix to obtain a first result.

[0127] Step S42: calculating attention weight on the first result and performing weighted summation to obtain a second result.

[0128] This step is to perform cross-frame attention calculation on the first result to obtain a second result.

[0129] Step S43: calculating attention according to the second result and the content prompt for expanding the video to be expanded to obtain a third result.

[0130] This step is to perform cross attention calculation on the video to be expanded.

[0131] In this step, as shown in FIG. 7, the content prompt (such as a text prompt, i.e., Context Embedding) for expanding the video to be expanded is calculated with the second result (such as the semantic information of the image, which is the output result of the cross-frame attention calculation, i.e., Latent Feature) to perform attention calculation, thereby enhancing the control of the input text prompt on the generated image. Among them, the input text prompt is encoded by the CLIP Text Encoder model to obtain a content embedding vector (Context Embedding), and then the content embedding vector and the feature (Latent Feature) output after the cross-frame attention calculation of the previous network layer are calculated for attention, thereby realizing the guidance of the text prompt to the diffusion model expansion content.

[0132] Step S44: performing full connection (FFN) calculation on the third result to obtain the latent feature sequence of the expanded video.

[0133] In this embodiment, the network structure and parameters of the cross-attention module in the encoding, decoding and UNet model and the Stable diffusion model are consistent and fixed.

[0134] In some embodiments, before denoising the random noise matrix, further comprising: predicting the Gaussian noise in the random noise matrix to obtain a predicted noise; comparing the predicted noise with the added Gaussian noise in the latent feature sequence of the video to be expanded, calculating the difference between the two, and obtaining a noise loss; calculating the gradient according to the noise loss, and updating the parameters of the diffusion model according to the gradient; the parameters of the diffusion model include the query, key and value of the transformer block.

[0135] As shown in FIG. 8a, it is a schematic diagram of Epoch iteration in the diffusion model training process. Epoch refers to the period in which the entire data set of the neural network model is used once in the training process. For example, if a neural network model is trained on 100 data points, and the batch size is 10, it will take 10 Epochs to complete the training of the entire data set. In each Epoch, the neural network will iterate all the data in the data set multiple times, and update the weights and biases of the model in each iteration to minimize the loss function. In FIG. 8a, t represents the noise intensity, and the larger the value of t, the greater the intensity.

[0136] As shown in FIG. 8b, for each training step in the diffusion model training process, each training step of the diffusion model includes: 1, randomly select a time step and encode it; 2, add noise to the image; 3, train the UNet.

[0137] As can be seen from FIGS. 8a and 8b, the noise loss calculation is to input the noise-added image into the diffusion model, the diffusion model predicts the corresponding noise image, and then compares the noise image predicted by the diffusion model with the real added noise and calculates the difference, which is the noise loss. The calculation of the noise loss and the updating of the parameters of the diffusion model according to the noise loss can make the diffusion model accurately predict the current added noise.

[0138] In some embodiments, the potential feature sequence of the video to be expanded is subjected to MSE loss function calculation and AdamW optimizer calculation.

[0139] The MSE (Mean Squared Error) loss function is a commonly used loss function in the field of machine learning, which is used to measure the error between the output result (final predicted value) and the expected true value. The MSE loss function can minimize the loss of the model by reducing the difference between the predicted value and the true value during training, thereby improving the performance of the model. The AdamW optimizer can separate the weight decay and gradient update for processing, which helps to solve the problem of incompatibility between weight decay and adaptive learning rate algorithm.

[0140] In some embodiments, before the diffusion model adds noise to the potential feature sequence of the video to be expanded to make it a random noise matrix, the potential feature sequence of the video to be expanded is also mapped to a latent space; the size of the potential feature sequence of the video to be expanded in the latent space is 1 / 2 of the size of the expanded video. n n = 1, 2, 3, …, n is an integer. For example, if the size of the potential feature in the latent space is 32*32, and the size of the expanded video output by the diffusion model is 256*256.

[0141] In some embodiments, the expanded video has a video sample region and an expanded region, and the expanded region is located at any position outside the video sample region. The video sample region of the expanded video is determined according to different needs, such as the expanded region generated is above the video sample region, then the video sample region is placed below, and the content above the video sample region is completed by the diffusion model. If the expanded region generated is in other regions, the same applies.

[0142] In some embodiments, the dataset of the video sample is WebVid-10M. WebVid-10M is a dataset indispensable for video generation of diffusion models. The learning rate of the diffusion model is 5*10 -3 . The diffusion model training method uses 8 V100 graphics cards as computing resources.

[0143] The video processing method provided by the embodiments of the present disclosure can reduce the computational complexity of the diffusion model in the training process, reduce the consumption of computing resources, and thus improve the training efficiency and accuracy of the diffusion model for video expansion, and further improve the efficiency and accuracy of video expansion processing.

[0144] Based on the above-mentioned video processing method, the present embodiment further provides a video processing device, as shown in FIG. 9, which comprises an encoder 1 configured to encode the features of the video to be expanded, and obtain the latent feature sequence of the video to be expanded; a diffusion model block 2 configured to add noise to the latent feature sequence of the video to be expanded, so as to make it a random noise matrix; and to predict and denoise the random noise matrix, and obtain the latent feature sequence of the expanded video; the odd transformer blocks in the diffusion model block 2 have determined queries, keys and values, and the even transformer blocks have determined queries, keys and values; and a decoder 3 configured to decode the latent feature sequence of the expanded video, and obtain the expanded video.

[0145] The video processing device can improve the efficiency and accuracy of video expansion processing by using the diffusion model block 2 with the odd transformer blocks and the even transformer blocks having determined queries, keys and values.

[0146] In some embodiments, the video processing device further comprises a ControlNet model block configured to input additional control condition information into the random noise matrix; the additional control condition information includes edge, pose and depth information of the image.

[0147] In some embodiments, the diffusion model block 2 comprises a UNet model block; the UNet model block comprises a plurality of network layers; each network layer comprises two transformer blocks 20; the transformer block 20 comprises: a convolution block 201 configured to perform convolution calculation on the random noise matrix to obtain a first result; an inter-frame attention calculation block 202 configured to calculate attention weights on the first result and perform weighted summation to obtain a second result; a cross-attention calculation block 203 configured to calculate attention according to the second result and the content prompt for expanding the to-be-expanded video to obtain a third result; and a fully connected calculation block 204 configured to perform fully connected calculation on the third result to obtain a latent feature sequence of the expanded video.

[0148] The UNet model block adopts a DDPM model (i.e., Denoising Diffusion Probabilistic Models), and the main feature of the DDPM model is to generate new data samples by simulating the diffusion and denoising process of data. The DDPM model is a latent variable model based on variational inference, which is suitable for processing high-dimensional data such as images. The core idea of the DDPM model is to gradually evolve a simple distribution (usually a Gaussian distribution) into the distribution of the target data by iteratively applying a diffusion process. This process can be divided into two stages: a forward diffusion process and a reverse diffusion process. In the forward diffusion process, the model starts from the original data and gradually adds noise with a normal distribution, so that the data gradually loses its original features and finally forms an approximate prior distribution. This process is similar to the "blurring" or "diffusion" of data. The reverse diffusion process is the inverse of the forward diffusion process, which gradually removes the noise added before, so that the data gradually recovers to its original state, and is therefore also called "denoising" or "restoration". By continuously iterating this process, the model can generate new samples similar to the original data.

[0149] In this embodiment, the forward diffusion process (i.e., the noise adding process) of the DDPM model is implemented by a Stable diffusion model.

[0150] In some embodiments, the video processing device comprises 8 V100 graphics cards. In this embodiment, the video processing device is implemented by using 8 V100 graphics cards.

[0151] The video processing device provided in this embodiment can reduce the computational complexity of video expansion processing by using a diffusion model block, thereby reducing the consumption of a large amount of computing resources, and further improving the efficiency and accuracy of video expansion processing.

[0152] The embodiment of the present disclosure further provides an electronic device, comprising a memory and a processor, at least one program is stored in the memory, wherein the at least one program is loaded and executed by the processor to realize the video processing method.

[0153] The electronic device provided in the embodiment can reduce the calculation amount of the video edge expansion processing, thereby reducing the consumption of a large amount of computing resources, and further improving the efficiency and accuracy of the video edge expansion processing.

[0154] The embodiment of the present disclosure further provides a non-transitory computer storage medium, which stores a computer program, wherein the computer program realizes the video processing method when executed by a processor.

[0155] The non-transitory computer storage medium provided in the embodiment can reduce the calculation amount of the video edge expansion processing, thereby reducing the consumption of a large amount of computing resources, and further improving the efficiency and accuracy of the video edge expansion processing.

[0156] It can be understood that the above embodiments are only exemplary embodiments adopted for illustrating the principles of the present disclosure, but the present disclosure is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and essence of the present disclosure, and these modifications and improvements are also considered as the protection scope of the present disclosure.

Claims

1. A video processing method, wherein, include: Train the diffusion model; The potential feature sequence of the video to be expanded is obtained by performing feature encoding on the video to be expanded. The latent feature sequence of the video to be expanded is input into the trained diffusion model; The diffusion model adds noise to the latent feature sequence of the video to be expanded, making it a random noise matrix; The random noise matrix is ​​subjected to noise prediction and denoising to obtain the potential feature sequence of the expanded-side video; Decode the latent feature sequence of the extended-side video to obtain the extended-side video; Training the diffusion model includes: Extract odd-numbered feature maps from odd-numbered frames of the latent feature sequence of the noisy video samples; Even-numbered feature maps are extracted from even-numbered frames of the latent feature sequence of the noisy video samples; Obtain the transformer blocks with odd order and the transformer blocks with even order in the diffusion model; The odd number of transformer blocks perform attention calculations on the odd number of feature maps to obtain the query, key, and value of the odd number of transformer blocks; The even-numbered transformer blocks perform attention calculations on the even-numbered feature maps to obtain the queries, keys, and values ​​of the even-numbered transformer blocks.

2. The video processing method according to claim 1, wherein, After adding noise to the latent feature sequence of the video to be expanded, making it a random noise matrix, and before performing noise prediction and denoising on the random noise matrix to obtain the latent feature sequence of the expanded video, the method further includes: Input additional control condition information into the random noise matrix; The additional control condition information includes the image's edge, pose, and depth information.

3. The video processing method according to claim 1, wherein, The diffusion model adds noise to the latent feature sequence of the video to be expanded, making it a random noise matrix, including: Extract latent vector noise from the latent feature sequence of the video to be expanded to obtain the random noise matrix of the video to be expanded; Based on the content prompts for expanding the edge of the video to be expanded, Gaussian noise of the video to be expanded is randomly sampled, and the region in the Gaussian noise of the video to be expanded corresponding to the video to be expanded is replaced with the random noise matrix of the video to be expanded.

4. The video processing method according to claim 3, wherein, Also includes: The Gaussian noise in the replaced expanded video is then subjected to weighted smoothing of the expanded regions of the two consecutive frames. The formula for weighted smoothing is as follows: Where M represents the mask matrix, the mask matrix value corresponding to the video to be expanded has a value of 1, and the mask value of the expanded region has a value of 0; x t Let x represent the t-th frame of the video to be expanded. t-1 This represents the (t-1)th frame of the video to be expanded.

5. The video processing method according to claim 1, wherein, The diffusion model adds noise to the latent feature sequence of the video to be expanded, making it a random noise matrix, using the following formula: Where t is a timestamp, t∈{1,...,T}, q(x t |x t-1 ) represents the conditional probability; α t For hyperparameters, α t =1-β t β gradually increases from 0.0001 to 0.02 over time; I is the noise of the video to be expanded before or after expansion, and I is the identity matrix; x t x represents the latent feature sequence of all frames of the video to be expanded in the noise-adding step corresponding to time t; t-1 This represents the latent feature sequence of all frames of the video to be expanded in the noise addition step corresponding to time t-1; the noise addition step corresponding to time t and the noise addition step corresponding to time t-1 are two adjacent steps; and x t Only with x t-1 related.

6. The video processing method according to claim 3, wherein, The step of performing noise prediction and denoising on the random noise matrix to obtain the latent feature sequence of the expanded-side video includes: Perform convolution calculation on the random noise matrix to obtain the first result; Calculate the attention weights for the first result and perform a weighted summation to obtain the second result; Based on the second result and the content prompts for expanding the edge of the video to be expanded, the attention is calculated to obtain the third result; The third result is subjected to a fully connected computation to obtain the latent feature sequence of the expanded-side video.

7. The video processing method according to claim 1, wherein, Before denoising the random noise matrix, the process further includes: Predict the Gaussian noise in the random noise matrix to obtain the predicted noise; The predicted noise is compared with the Gaussian noise added to the latent feature sequence of the video to be expanded, and the difference between the two is calculated to obtain the noise loss. The gradient is calculated based on the noise loss, and the parameters of the diffusion model are updated based on the gradient. The parameters of the diffusion model include the query, key, and value of the transformer block.

8. The video processing method according to claim 1, wherein, Before adding noise to the latent feature sequence of the video to be expanded in the diffusion model to make it a random noise matrix, the method further includes mapping the latent feature sequence of the video to be expanded to a latent space. The size of the latent feature sequence of the video to be expanded within the latent space is 1 / 2 the size of the video to be expanded. n n = 1, 2, 3, ..., where n is an integer.

9. The video processing method according to claim 1, wherein, The expanded video has a video area to be expanded and an expanded area, wherein the expanded area is located at any position outside the video area to be expanded.

10. A video processing apparatus, comprising an encoder configured to perform feature encoding on a video to be expanded to obtain a potential feature sequence of the video; A diffusion model block is configured to add noise to the latent feature sequence of the video to be expanded, making it a random noise matrix; perform noise prediction and denoising on the random noise matrix to obtain the latent feature sequence of the video to be expanded; odd-numbered transformer blocks in the diffusion model block have definite queries, keys and values, and even-numbered transformer blocks have definite queries, keys and values. A decoder is configured to decode the latent feature sequence of the extended-side video to obtain the extended-side video.

11. The video processing apparatus according to claim 10, wherein, It also includes a ControlNet model block, configured to input additional control condition information into the random noise matrix; The additional control condition information includes the image's edge, pose, and depth information.

12. The video processing apparatus according to claim 10, wherein, The diffusion model block includes the UNet model block; The UNet model block includes multiple network layers; Each of the network layers includes two transformer blocks; The transformer block includes: A convolutional block is configured to perform a convolution calculation on the random noise matrix to obtain a first result; The inter-frame attention calculation block is configured to calculate attention weights on the first result and perform a weighted summation to obtain the second result; A cross-attention calculation block is configured to calculate attention based on the second result and the content cues for expanding the edge of the video to be expanded, and obtain a third result; A fully connected computation block is configured to perform fully connected computation on the third result to obtain the latent feature sequence of the expanded-side video.

13. An electronic device comprising a memory and a processor, wherein the memory stores at least one program, wherein... The at least one program segment is loaded and executed by the processor to implement the video processing method according to any one of claims 1-9.

14. A non-transient computer storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the video processing method as described in any one of claims 1-9.

Images