A video processing method and device based on a diffusion model and a medium

Abstract

The application relates to the technical field of video processing, and provides a video processing method based on a diffusion model, equipment and a medium. The method comprises the following steps: acquiring physical prior features of a target video of a time step of a diffusion model in reverse, and calculating head distribution vectors of a multi-head attention model based on the physical prior features; performing feature extraction on each frame of image to obtain local features, and performing aggregation to obtain global features; encoding the local features based on the head distribution vectors to obtain local encoding vectors corresponding to the heads, weighting each local encoding vector to obtain final local encoding vectors, and integrating the final local encoding vectors to obtain local encodings; encoding the global features, weighting and integrating the global features to obtain global encodings; decoding the global encodings and each local encoding to obtain denoised images, and splicing the denoised images to obtain a denoised video. The method can improve the quality of the video output by the diffusion model.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video processing method, device and medium based on a diffusion model. Background Technology

[0002] Diffusion models are a type of generative model that has attracted much attention in recent years. They can generate images or videos based on text and generate high-quality images or videos by iteratively injecting noise into the data distribution and gradually removing the noise.

[0003] However, the performance of diffusion models largely depends on the sampling method used. Traditional sampling methods, such as Markov Chain Monte Carlo (MCMC) and Langevin dynamics, suffer from slow convergence speed and poor sample quality. To overcome these limitations, researchers have explored various improvement schemes, such as Denoising Diffusion Probabilistic Models (DDPM) and Denoising Diffusion Implicit Models (DDIM). Although these methods improve the performance of diffusion models to some extent, they still have shortcomings in handling high-dimensional data and capturing long-range dependencies.

[0004] Traditional MCMC and Langevin dynamics sampling methods converge slowly and require numerous iterations to generate high-quality samples. This is particularly evident when processing high-dimensional data such as high-resolution images.

[0005] While methods such as DDPM and DDIM alleviate the aforementioned problems to some extent, they still struggle to accurately capture long-range dependencies in the data. This can lead to a lack of global consistency and coherence in the generated samples.

[0006] Existing methods fall short in capturing the underlying structure and semantic information of data. Although some methods incorporate prior knowledge or conditional information, they still struggle to accurately model the inherent patterns and hierarchical structure of the data. This results in room for improvement in the semantic consistency and detail fidelity of generated samples.

[0007] The training and sampling processes of diffusion models both involve the introduction and removal of noise. However, existing methods have limited robustness to noise, especially when faced with strong or non-Gaussian noise, where denoising results are often unsatisfactory. Residual noise can lead to artifacts, distortion, and other quality issues in the generated samples, resulting in low-quality videos output by the diffusion model and severely impacting its practical value. Summary of the Invention

[0008] This application provides a video processing method, device, and medium based on a diffusion model, which can solve the problem of low video quality output by the diffusion model.

[0009] In a first aspect, embodiments of this application provide a video processing method based on a diffusion model, the video processing method comprising:

[0010] For each time step of the reverse diffusion model, perform the following steps:

[0011] Obtain the physical prior features of the target video at each time step, and calculate the head assignment vector of the multi-head attention model based on the physical prior features; the physical prior features are used to describe the physical information of the target video, and the head assignment vector is used to describe the importance of each head in the multi-head attention model.

[0012] Feature extraction is performed on each frame of the target video to obtain the local features of each frame, and all local features are aggregated to obtain the global features of the target video.

[0013] For each local feature, a head-assigned vector is used to encode the local feature using a multi-head attention model, resulting in a local encoding vector for each head in the multi-head attention model. Each local encoding vector is then weighted according to all local encoding vectors and all heads to obtain the final local encoding vector for each local encoding vector. All final local encoding vectors are then integrated to obtain the local encoding of the local feature.

[0014] Based on the head assignment vector, the global features are encoded using a multi-head attention model to obtain the global encoding vector corresponding to each head in the multi-head attention model. Then, each global encoding vector is weighted according to all global encoding vectors and all heads to obtain the final global encoding vector of each global encoding vector. All final global encoding vectors are integrated to obtain the global encoding of the global features.

[0015] Decode the global encoder and each local encoder to obtain the denoised image corresponding to each frame, and then stitch all the denoised images together to obtain the denoised video output by the diffusion model at each time step.

[0016] Optionally, the head assignment vector for the multi-head attention model is calculated based on physical prior features, including:

[0017] Through the formula:

[0018] α=σ(W2·ReLU(W1·[e t ;f phy ]+b1)+b2)

[0019] Calculate the head assignment vector α;

[0020] Where W1 and W2 both represent weight matrices, b1 and b2 both represent bias terms, and e t f represents the step embedding of time steps. phy σ() and ReLU() represent physical prior features, and both represent activation functions.

[0021] Optionally, all local features can be aggregated to obtain the global features of the target video, including:

[0022] Through the formula:

[0023] f global =F temporal (f locl,1 ,f locl,2 ,...,f locl,T )

[0024] Obtain the global features f of the target video global ;

[0025] Among them, F temporal () represents an aggregate function, f locl,1 f represents the local features of the first frame image. locl,2 f represents the local features of the second frame image. locl,N This represents the local features of the Nth frame image.

[0026] Optionally, each local coding vector is weighted based on all local coding vectors and all headers to obtain the final local coding vector for each local coding vector, including:

[0027] Each local coding vector is weighted according to all local coding vectors to obtain the intermediate local coding vector of each local coding vector;

[0028] For each head, calculate the information transfer weight between the head and each other head. Based on all information transfer weights, weight the intermediate local encoding vectors corresponding to the head to obtain the final local encoding vector for each local encoding vector.

[0029] Optionally, each local coding vector is weighted according to all local coding vectors to obtain an intermediate local coding vector for each local coding vector, including:

[0030] Through the formula:

[0031] H′ i =γ i ⊙H i

[0032] Obtain the intermediate local coding vector H' of the i-th local coding vector i ;

[0033] Among them, H i Let γ represent the i-th local encoding vector, i = 1, 2, ..., I, where I represents the number of heads in the multi-head attention model. i The dynamic importance weight corresponding to the i-th local encoding vector is:

[0034] γ i =A i ⊙β i

[0035] A i =softmax(W a ·H i )

[0036] Among them, A i β represents the self-attention weight corresponding to the i-th local encoding vector. i W represents the initial importance weight of the i-th local encoding vector. a This represents the weight matrix.

[0037] Optionally, calculate the information transfer weights between the head and each other head, including:

[0038] Through the formula:

[0039] X ijk =g×1U1(i,:)×2U2(j,:)×3U3(k,:)

[0040] Calculate the message passing weight X between the i-th header and the j-th header under the k-th routing type. ijk ;

[0041] Where g represents the kernel tensor, U1(i,:) represents the factor matrix of the i-th head, U2(j,:) represents the factor matrix of the j-th head, U3(k,:) represents the factor matrix of the k-th route type, i≠j, i,j∈{1,2,...,I}, I represents the number of heads in the multi-head attention model, k=1,2,...,K, K represents the number of route types.

[0042] Optionally, the intermediate local coding vectors corresponding to the header are weighted based on the weights of all information transmission to obtain the final local coding vector for each local coding vector, including:

[0043] Through the formula:

[0044]

[0045] Obtain the final local encoding vector of the i-th local encoding vector.

[0046] Among them, H'i This represents the intermediate local coding vector of the i-th local coding vector.

[0047] Optionally, the global encoder and each local encoder are decoded to obtain the denoised image corresponding to each frame, including:

[0048] Through the formula:

[0049]

[0050] Get the denoised image of the nth frame.

[0051] in, This represents the local encoding of the nth frame image. This represents global encoding, where n = 1, 2, ..., N, and N represents the number of frames in the target video, F decode () indicates the decoder.

[0052] Secondly, embodiments of this application provide a video processing apparatus based on a diffusion model, comprising:

[0053] The generation module is used to obtain the physical prior features of the target video at each time step, and calculate the head assignment vector of the multi-head attention model based on the physical prior features; the physical prior features are used to describe the physical information of the target video, and the head assignment vector is used to describe the importance of each head in the multi-head attention model.

[0054] The feature extraction module is used to extract features from each frame of the target video, obtain the local features of each frame, and aggregate all local features to obtain the global features of the target video.

[0055] The first encoding module is used to encode each local feature based on the head-assigned vector and the multi-head attention model to obtain the local encoding vector corresponding to each head in the multi-head attention model. Then, the local encoding vector is weighted according to all local encoding vectors and all heads to obtain the final local encoding vector of each local encoding vector. Finally, all the final local encoding vectors are integrated to obtain the local encoding of the local feature.

[0056] The second encoding module is used to encode global features based on the head allocation vector and the multi-head attention model to obtain the global encoding vector corresponding to each head in the multi-head attention model. Then, each global encoding vector is weighted according to all global encoding vectors and all heads to obtain the final global encoding vector of each global encoding vector. All final global encoding vectors are integrated to obtain the global encoding of global features.

[0057] The decoding module is used to decode the global encoder and each local encoder to obtain the denoised image corresponding to each frame, and then stitch all the denoised images together to obtain the denoised video output by the diffusion model at the time step.

[0058] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned video processing method based on a diffusion model.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned video processing method based on a diffusion model.

[0060] The above-mentioned solution in this application has the following beneficial effects:

[0061] In the embodiments of this application, the head allocation vector of the multi-head attention model is calculated based on physical prior features, taking into account the physical information of the target video, thereby improving the accuracy and practicality of the head allocation vector. Based on the head allocation vector, the multi-head attention model is used to encode local features to obtain the local encoding vector corresponding to each head in the multi-head attention model. Each local encoding vector is weighted according to all local encoding vectors and all heads to obtain the final local encoding vector of each local encoding vector. All the final local encoding vectors are integrated to obtain the local encoding of local features. This takes into account each head in the multi-head attention model and updates the output of each head by weighting, thereby improving the accuracy of the local encoding vector and thus improving the precision of the local encoding. The global encoding and each local encoding are decoded to obtain the denoised image corresponding to each frame. All denoised images are then stitched together to obtain the denoised video of the target video. This ensures that the global and local information of the target video are considered during decoding, thereby improving the quality of the video output by the diffusion model.

[0062] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1A flowchart illustrating a video processing method based on a diffusion model provided in an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the structure of a video processing device based on a diffusion model provided in an embodiment of this application;

[0066] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0067] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0068] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0069] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0070] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0071] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0072] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0073] To address the issue of low video quality output by existing diffusion models, this application provides a video processing method based on a diffusion model. This method calculates the head allocation vector of a multi-head attention model based on physical prior features, considering the physical information of the target video and improving the accuracy and practicality of the head allocation vector. Based on the head allocation vector, the multi-head attention model is used to encode local features, obtaining a local encoding vector corresponding to each head in the multi-head attention model. Each local encoding vector is then weighted according to all local encoding vectors and all heads to obtain a final local encoding vector. All final local encoding vectors are integrated to obtain the local encoding of local features. By considering each head in the multi-head attention model and updating the output of each head through weighting, the accuracy of the local encoding vector is improved, thereby increasing the precision of the local encoding. The global encoding and each local encoding are decoded to obtain a denoised image corresponding to each frame. All denoised images are then stitched together to obtain the denoised video of the target video. This method considers both global and local information of the target video during decoding, improving the quality of the video output by the diffusion model.

[0074] The following is an illustrative description of the video processing method based on the diffusion model provided in this application.

[0075] In some embodiments of this application, when the diffusion model processes the video, in each forward time step, the diffusion model adds noise to the video or image input at that time step, and then inputs the noise-added image or video to the next time step to continue adding noise, and so on for multiple time steps; in the reverse process, the video output at the last time step is denoised, and then input to the penultimate time step to continue denoising, and so on, until the first time step outputs a denoised video or image, so as to improve the quality of the video or image. In the forward process of the diffusion model, the time step count is from the first time step to the last time step, and in the reverse process, the time step count is from the last time step to the first time step.

[0076] like Figure 1 As shown, the video processing method based on the diffusion model provided in this application includes the following steps:

[0077] For each time step of the reverse diffusion model, perform the following steps:

[0078] Step 11: Obtain the physical prior features of the target video at the time step, and calculate the head assignment vector of the multi-head attention model based on the physical prior features.

[0079] The aforementioned physical prior features are used to describe the physical information of the target video (such as resolution, color, etc.), and the head assignment vector is used to describe the importance of each head in the multi-head attention model.

[0080] Before this step begins, a forward noise-adding process needs to be performed on the initial video using a diffusion model. The diffusion model adds noise to the initial video multiple times in a forward time step, and uses the initial video after adding noise at the last time step as the input for the last time step in the reverse noise-de-noising process.

[0081] It should be noted that the target video of a time step is the video output from the previous time step in reverse order. For example, if the time step is the 3rd time step, then the target video of the 3rd time step is the video output from the 4th time step.

[0082] In some embodiments of this application, feature extraction models such as convolutional neural networks can be used to obtain the physical prior features of the target video. The specific steps for calculating the head assignment vector of the multi-head attention model based on the physical prior features are as follows:

[0083] Through the formula:

[0084] α=σ(W2·ReLU(W1·[e t ;f phy ]+b1)+b2)

[0085] Calculate the head assignment vector α.

[0086] Where W1 and W2 both represent weight matrices, b1 and b2 both represent bias terms, and e t f represents the step embedding of time steps. phy σ() and ReLU() represent physical prior features, and both represent activation functions.

[0087] It should be noted that the head assignment vector mentioned above includes multiple elements, each corresponding to a different head. These elements describe the importance of the corresponding head, and the step embedding represents the positional information of the current time step within the entire denoising process. Specifically, the step embedding is a fixed-dimensional vector where elements at different positions correspond to different time steps, and the values ​​of these elements represent the relative position of the current time step within the entire denoising sequence. By combining physical prior features with the step embedding, the model can simultaneously learn the physical properties of the target video at the current denoising stage and the progress information of that stage within the entire denoising process, thereby better guiding the model to perform adaptive denoising. The introduction of step embedding allows the head assignment vector to be dynamically adjusted according to the different denoising progresses, improving the ability of the multi-head attention mechanism to capture local and global dependencies in the video. The multi-head attention model mentioned above is a multi-head attention mechanism, which can be specifically represented as follows:

[0088] MultiHead(Q,K,V)=Concat(head1,...,head I )

[0089]

[0090] Where MultiHead(Q,K,V) represents the output of the multi-head attention mechanism, Concat() represents the concatenation operation, head1 represents the output of the first head in the multi-head attention mechanism, and head... I This represents the output of the i-th head in the multi-head attention mechanism. i Let represent the output of the i-th head in the multi-head attention mechanism, where i = 1, 2, ..., I; `Attention()` represents the attention mechanism; `Q` represents the query vector; `K` represents the key vector; and `V` represents the value vector. This represents the query weight corresponding to the i-th header. This represents the key weight corresponding to the i-th head. Let d represent the weight of the i-th head, α represent the weight assigned to the head, and d k Indicates dimension.

[0091] For example, the above steps can be performed using computer software such as MATLAB to obtain the head assignment vector.

[0092] It is worth mentioning that the head assignment vector obtained based on physical prior features takes into account the physical information of the target video, which improves the rationality and practicality of the head assignment vector.

[0093] Step 12: Extract features from each frame of the target video to obtain the local features of each frame, and aggregate all local features to obtain the global features of the target video.

[0094] In some embodiments of this application, feature extraction models such as convolutional neural networks can be used to obtain local features of each frame of image. The specific steps for aggregating all local features to obtain the global features of the target video are as follows:

[0095] Through the formula:

[0096] f global =F temporal (f locl,1 ,f locl,2 ,...,f locl,T )

[0097] Obtain the global features f of the target video global .

[0098] Among them, F temporal () represents an aggregate function, f locl,1 f represents the local features of the first frame image. locl,2 f represents the local features of the second frame image. locl,N This represents the local features of the Nth frame image.

[0099] For example, the above feature extraction model and formula can be run using computer software such as MATLAB to obtain local and global features.

[0100] Step 13: For each local feature, based on the head allocation vector, encode the local feature using a multi-head attention model to obtain the local encoding vector corresponding to each head in the multi-head attention model. Then, weight each local encoding vector according to all local encoding vectors and all heads to obtain the final local encoding vector of each local encoding vector. Integrate all the final local encoding vectors to obtain the local encoding of the local feature.

[0101] For each local feature, the above-mentioned method, based on the head-assigned vector, encodes the local feature using a multi-head attention model, obtaining a local encoding vector corresponding to each head in the multi-head attention model. Then, each local encoding vector is weighted according to all local encoding vectors and all heads to obtain the final local encoding vector for each local encoding vector. The specific steps for integrating all final local encoding vectors to obtain the local encoding of the local feature are as follows:

[0102] The first step is to encode local features based on the head assignment vector using a multi-head attention model, thereby obtaining the local encoding vector corresponding to each head in the multi-head attention model.

[0103] Specifically, the local features are divided into multiple feature segments. Based on the head allocation vector, feature segments are assigned to each head in the multi-head attention model. For example, if the head allocation vector indicates that the first head has the highest importance, then the first head is assigned the most feature segments. Then, the assigned feature segments are input into each head of the multi-head attention model to obtain the local encoding vector corresponding to each head.

[0104] The second step is to weight each local coding vector based on all local coding vectors and all headers to obtain the final local coding vector for each local coding vector.

[0105] First, each local coding vector is weighted according to all local coding vectors to obtain the intermediate local coding vector for each local coding vector.

[0106] Specifically, through the formula:

[0107] H′ i =γ i ⊙H i

[0108] Obtain the intermediate local coding vector H' of the i-th local coding vector i .

[0109] Among them, Hi 表 Let γ represent the i-th local encoding vector, i = 1, 2, ..., I, where I represents the number of heads in the multi-head attention model. i The dynamic importance weight corresponding to the i-th local encoding vector is:

[0110] γ i =A i ⊙β i

[0111] A i =softmax(W a ·H i )

[0112] Among them, A i β represents the self-attention weight corresponding to the i-th local encoding vector. i W represents the initial importance weight of the i-th local encoding vector. a This represents the weight matrix.

[0113] Then, for each head, the information transfer weight between the head and each other head is calculated.

[0114] Specifically, through the formula:

[0115] X ijk =g×1U1(i,:)×2U2(j,:)×3U3(k,:)

[0116] Calculate the message passing weight X between the i-th header and the j-th header under the k-th routing type. ijk .

[0117] Where g represents the kernel tensor, U1(i,:) represents the factor matrix of the i-th head, U2(j,:) represents the factor matrix of the j-th head, U3(k,:) represents the factor matrix of the k-th route type, i≠j, i,j∈{1,2,...,I}, I represents the number of heads in the multi-head attention model, k=1,2,...,K, K represents the number of route types.

[0118] Finally, the intermediate local coding vectors corresponding to the header are weighted based on the weights of all information transmission to obtain the final local coding vector for each local coding vector.

[0119] Specifically, through the formula:

[0120]

[0121] Obtain the final local encoding vector of the i-th local encoding vector.

[0122] Among them, H' i This represents the intermediate local coding vector of the i-th local coding vector.

[0123] It should be noted that the above routing types refer to the information flow and aggregation methods between heads, such as: point-to-point routing (each head only interacts and aggregates information with another specific head, with information being passed one-to-one between heads, such as head 1 interacting only with head 2, head 3 interacting with head 4, etc. This routing method can reduce information redundancy between heads), fully connected routing (each head interacts and aggregates information with all other heads, i.e., heads are fully interconnected and densely transmit information. This routing method helps to obtain rich global context information), and random routing (heads randomly select to interact and aggregate information, i.e., each head randomly selects one or more other heads to transmit information. This routing method can strike a balance between point-to-point routing and fully connected routing, achieving a balance between obtaining global information and preventing redundancy). Different routing types have different characteristics and applicable scenarios. This application can flexibly select or combine different routing types to maximize the performance of the multi-head attention mechanism and improve the quality of video denoising.

[0124] The third step is to integrate all the final local encoding vectors to obtain the local encoding of the local features.

[0125] For example, the concatenation operation in the multi-head attention mechanism described above can be used to integrate all the final local encoding vectors to obtain the local encoding.

[0126] It is worth mentioning that using the head assignment vector to assign to each head can improve the accuracy of the local encoding vector obtained by the head. At the same time, when weighting the local encoding vector, the importance of the head itself and the information transmission between heads are taken into account, which improves the accuracy of the local encoding vector and thus improves the precision of local encoding.

[0127] Step 14: Based on the head allocation vector, encode the global features using a multi-head attention model to obtain the global encoding vector corresponding to each head in the multi-head attention model. Then, weight each global encoding vector according to all global encoding vectors and all heads to obtain the final global encoding vector of each global encoding vector. Integrate all the final global encoding vectors to obtain the global encoding of the global features.

[0128] In some embodiments of this application, the process of obtaining global encoding described above is consistent with step 13, that is, global features are allocated using the head allocation vector to obtain the global encoding vector of each head, and each global encoding vector is weighted to obtain the final global encoding vector, and all the final global encoding vectors are integrated to obtain the global encoding.

[0129] For example, the above steps can be performed using computer software such as MATLAB to obtain the global code.

[0130] It is worth mentioning that by obtaining the global encoding vector corresponding to each head through a multi-head attention mechanism and weighting each global encoding vector, the accuracy of the global encoding vector can be improved, accurately describing the global information of the target video.

[0131] Step 15: Decode the global encoder and each local encoder to obtain the denoised image corresponding to each frame, and stitch all the denoised images together to obtain the denoised video output by the diffusion model at the time step.

[0132] Specifically, through the formula:

[0133]

[0134] Get the denoised image of the nth frame.

[0135] in, This represents the local encoding of the nth frame image. This represents global encoding, where n = 1, 2, ..., N, and N represents the number of frames in the target video, F decode () indicates the decoder.

[0136] For example, all the denoised images are stitched together in the order they appear in the target video to obtain the denoised video of the target video. The decoder mentioned above can be an audio video coding standard (AVS), etc.

[0137] It should be noted that when the time step is the first time step, the entire video denoising process of the diffusion model is completed, and the denoised video is the final denoised video. When the time step is any other than the first time step, the denoised video is used as the target video for the next time step in reverse. In the next time step in reverse, the target video is denoised to obtain the denoised video. The diffusion model processing method of this application can be applied to practical tasks that use the diffusion model. For example, if it is necessary to generate video from text using the diffusion model, the video processing method of this application can be used to process the generated video and improve its quality.

[0138] For example, there are 5 time steps in total. For the second time step, which is not the first time step, the denoised video corresponding to the second time step is used as the target video of the first time step for denoising, and the denoised video corresponding to the first time step is obtained. The denoised video at this time is the final denoised video.

[0139] It is worth mentioning that by considering each head in the multi-head attention model and updating the output of each head by weighting, the accuracy of the local encoding vector is improved, thereby improving the precision of local encoding. During decoding, the global and local information of the target video are considered, which can improve the quality of the video output by the diffusion model.

[0140] The following is an exemplary description of the video processing apparatus based on the diffusion model provided in this application.

[0141] like Figure 2 As shown, this application provides a video processing apparatus based on a diffusion model, the video processing apparatus 200 based on a diffusion model includes:

[0142] The generation module 201 is used to obtain the physical prior features of the target video at the time step, and calculate the head assignment vector of the multi-head attention model based on the physical prior features; the physical prior features are used to describe the physical information of the target video, and the head assignment vector is used to describe the importance of each head in the multi-head attention model.

[0143] The feature extraction module 202 is used to extract features from each frame of the target video, obtain the local features of each frame, and aggregate all the local features to obtain the global features of the target video.

[0144] The first encoding module 203 is used to encode each local feature based on the head allocation vector and the multi-head attention model to obtain the local encoding vector corresponding to each head in the multi-head attention model. Then, it weights each local encoding vector according to all local encoding vectors and all heads to obtain the final local encoding vector of each local encoding vector. Finally, it integrates all the final local encoding vectors to obtain the local encoding of the local feature.

[0145] The second encoding module 204 is used to encode global features based on the head allocation vector and a multi-head attention model to obtain the global encoding vector corresponding to each head in the multi-head attention model. It also weights each global encoding vector according to all global encoding vectors and all heads to obtain the final global encoding vector of each global encoding vector. Finally, it integrates all the final global encoding vectors to obtain the global encoding of the global features.

[0146] The decoding module 205 is used to decode the global encoding and each local encoding to obtain the denoised image corresponding to each frame, and to stitch all the denoised images together to obtain the denoised video output by the diffusion model at the time step.

[0147] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0149] like Figure 3 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0150] Specifically, when the processor D100 executes the computer program D102, it calculates the head allocation vector of the multi-head attention model based on the physical prior features, taking into account the physical information of the target video, thereby improving the accuracy and practicality of the head allocation vector. Based on the head allocation vector, the multi-head attention model is used to encode the local features to obtain the local encoding vector corresponding to each head in the multi-head attention model. Each local encoding vector is weighted according to all local encoding vectors and all heads to obtain the final local encoding vector of each local encoding vector. All final local encoding vectors are integrated to obtain the local encoding of the local features. This considers each head in the multi-head attention model, and the output of each head is updated by weighting, which improves the accuracy of the local encoding vector and thus improves the precision of the local encoding. The global encoding and each local encoding are decoded to obtain the denoised image corresponding to each image. All denoised images are then stitched together to obtain the denoised video of the target video. This ensures that the decoding considers both the global and local information of the target video, improving the quality of the video output by the diffusion model.

[0151] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0152] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0153] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0154] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a video processing method apparatus / terminal device based on a diffusion model, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0156] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0157] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video processing method based on a diffusion model, characterized in that, include: For each time step of the reverse diffusion model, perform the following steps: Obtain the physical prior features of the target video at the time step, and calculate the head assignment vector of the multi-head attention model based on the physical prior features; the physical prior features are used to describe the physical information of the target video, and the head assignment vector is used to describe the importance of each head in the multi-head attention model. Feature extraction is performed on each frame of the target video to obtain the local features of each frame, and all local features are aggregated to obtain the global features of the target video. For each of the local features, based on the head allocation vector, the local features are encoded using the multi-head attention model to obtain the local encoding vector corresponding to each head in the multi-head attention model. Each local encoding vector is weighted according to all local encoding vectors and all heads to obtain the final local encoding vector of each local encoding vector. All final local encoding vectors are integrated to obtain the local encoding of the local feature. Based on the head assignment vector, the global features are encoded using the multi-head attention model to obtain the global encoding vector corresponding to each head in the multi-head attention model. Each global encoding vector is weighted according to all global encoding vectors and all heads to obtain the final global encoding vector of each global encoding vector. All final global encoding vectors are integrated to obtain the global encoding of the global features. The global code and each of the local codes are decoded to obtain the denoised image corresponding to each frame of the image, and all the denoised images are stitched together to obtain the denoised video output by the diffusion model at the time step.

2. The video processing method according to claim 1, characterized in that, The calculation of the head assignment vector for the multi-head attention model based on the physical prior features includes: Through the formula: Calculate the head assignment vector ; in, and Both represent weight matrices. and Both represent bias terms. This indicates the step embedding of the time step. This represents the physical prior features. and Both represent activation functions.

3. The video processing method according to claim 1, characterized in that, The aggregation of all local features to obtain the global features of the target video includes: Through the formula: Obtain the global features of the target video ; in, Represents aggregate functions, This represents the local features of the first frame image. This represents the local features of the second frame image. Indicates the first Local features of a frame image.

4. The video processing method according to claim 1, characterized in that, The step of weighting each local coding vector based on all local coding vectors and all headers to obtain the final local coding vector for each local coding vector includes: Each local coding vector is weighted according to all local coding vectors to obtain an intermediate local coding vector for each local coding vector; For each of the heads, the information transfer weight between the head and each other head is calculated. The intermediate local encoding vectors corresponding to the heads are weighted based on all the information transfer weights to obtain the final local encoding vector of each local encoding vector.

5. The video processing method according to claim 4, characterized in that, The step of weighting each local coding vector based on all local coding vectors to obtain an intermediate local coding vector for each local coding vector includes: Through the formula: Get the The intermediate local code vector of a local code vector ; in, Indicates the first Local encoding vectors, , This indicates the number of heads in the multi-head attention model. Indicates the first The dynamic importance weights corresponding to each local encoding vector: in, Indicates the first The self-attention weights corresponding to each local encoding vector Indicates the first The initial importance weights of each local encoding vector. This represents the weight matrix.

6. The video processing method according to claim 4, characterized in that, The calculation of the information transfer weight between the head and each other head includes: Through the formula: Calculate the first The head and the first Between the heads in the first Information transmission weights under each routing type ; in, Represents the kernel tensor. Indicates the first Factor matrix of each head, Indicates the first Factor matrix of each head, Indicates the first A factor matrix of routing types, , , This indicates the number of heads in the multi-head attention model. , Indicates the number of route types.

7. The video processing method according to claim 6, characterized in that, The step of weighting the intermediate local coding vectors corresponding to the header based on all information transmission weights to obtain the final local coding vector for each local coding vector includes: Through the formula: Get the The final local encoded vector of each local encoded vector ; in, Indicates the first The intermediate local encoding vector of a local encoding vector.

8. The video processing method according to claim 1, characterized in that, Decoding the global code and each of the local codes to obtain the denoised image corresponding to each frame of the image includes: Through the formula: Get the Denoising of frame images ; in, Indicates the first Local coding of frame images This refers to the global encoding. , This indicates the frame number of the target video. This indicates the decoder.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video processing method based on the diffusion model as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video processing method based on the diffusion model as described in any one of claims 1 to 8.

Images