Local video style migration method and system without training

Abstract

The invention discloses a training-free local video style migration method and system, and the method comprises the steps: carrying out the denoising inversion of a video frame of an original video through a content branch, and obtaining the potential content features Zt (t = 1-T) of different timestamps; in the process of denoising inversion of the original video, a first video frame of the original video is masked, and the mask is transmitted to all video frames of the original video based on point matching; the method comprises the following steps of: performing de-noising inversion on a reference image by a style branch to obtain style potential characteristics # imgabs0 # of different timestamps, performing de-noising on an initial migration potential characteristic # imgabs1 # based on the initial migration potential characteristic in an editing branch, performing style migration on the migration potential characteristics based on an AdaIN style migration technology in a de-noising process to obtain a style migration result, and outputting the style migration result. And decoding to pixel representation frame by frame through a decoder to serve as a video after local style migration. According to the method, the requirement of a user on local area migration can be met, and meanwhile flicker and artifacts caused in the migration process are reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a training-free local video style transfer method and system. Background Art

[0002] Video stylization is gaining more and more attention as a subfield of video editing. It involves adding artistic style to video content. With the continuous progress in the generative field, many methods have emerged that use diffusion models to transfer the style of videos. In these methods, users can simply use text or images to complete the style rendering of the video. For example, StyleCrafter fine-tunes the Adapter and combines the reference pictures input by the user to complete personalized video generation; Diffutoon completes the cartoon coloring task of the video through text information; BIVDiff uses the downstream image editing model to perform frame-by-frame style transfer and uses the T2V model for temporal consistency smoothing; AnyV2V uses the downstream image editing model to transfer the style of the first frame of the video, and propagates the migration result of the first frame to all subsequent frames through the I2V model.

[0003] Existing video style transfer methods are mainly oriented towards large-scale style images, and fine-tuning the model to learn the prior information of a specific style, which usually requires massive data and heavy computational overhead. In addition, some users have style transfers for areas of interest in the video, and existing methods are based on global style transfer and cannot meet specific user needs. And directly extending some advanced image transfer methods to the video field will lead to temporal inconsistencies such as flickering and artifacts. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that local style transfer cannot be achieved in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a local video style transfer method without training, comprising the following steps:

[0006] The content branch denoises and reverses the video frames of the original video to obtain the content potential features Z at different timestamps. t , t=1→T; in the process of denoising and inverting the original video, a mask is applied to the first frame of the original video, and the mask is transferred to all frames of the original video based on point matching to obtain all masks;

[0007] The style branch denoises and inverts the reference image to obtain the style potential features at different timestamps.

[0008] In the edit branch, let the initial migration potential feature Denoising is performed based on the initial migrated latent features. During the denoising process, the style of the migrated latent features is migrated based on the AdaIN style migration technology to obtain the style migration result. The style migration result is decoded frame by frame into pixel representation through a decoder as the video after local style migration.

[0009] In the denoising process, the style migration of the migration potential features is performed based on the AdaIN style migration technology, including potential feature offset and attention offset; the potential feature offset is based on the migration potential features and style potential features of the previous timestamp to update the migration potential features of the current timestamp, and then based on the migration potential features and style potential features of the current timestamp to update the migration potential features of the next timestamp; repeat the above steps until the migration potential features of the initial timestamp are obtained as the style migration result; the attention offset updates the attention mechanism in the migration potential feature generation process based on the attention mechanism in the style potential feature generation process.

[0010] Preferably, the step of transferring the mask to all video frames of the original video based on point matching comprises the following steps:

[0011] In the process of denoising and reversing the original video, the feature maps of all frames of the original video at a specific timestamp t0 are saved as mask propagation feature maps;

[0012] In the first frame of the mask propagation feature map, a mask is applied according to the local requirements for style transfer;

[0013] Based on point matching, the mask of the current frame is transferred to the next frame until all frames in the mask propagation feature map are masked to obtain all masks M; the mask of the current frame is transferred to the next frame based on point matching, which is expressed as:

[0014]

[0015] Among them, F A represents the mask propagation feature map of the current frame, p A Indicates F A Point in the middle; F B represents the mask propagation feature map of the next frame, p B Indicates F B In and p A corresponding points; Dist represents cosine similarity, Indicates that when Dist reaches its minimum value, p B The value of .

[0016] Preferably, in the process of transferring the mask to all video frames of the original video based on point matching, the foreground area and the background area of ​​the current frame are randomly downsampled at a sampling rate r, and the sampling rate r is further adjusted according to the ratio of the foreground area to the background area, thereby reducing computational overhead.

[0017] Preferably, the potential feature shifting comprises the following steps:

[0018] Update the migration potential features based on the entire mask M It is expressed as:

[0019]

[0020] Introduce a time-varying coefficient η t Updated migration potential features Further update is performed, expressed as:

[0021]

[0022] Among them, [τ0,τ1] represents the introduced coefficient η t time interval, represents the value of η at time τ1, represents the value of η at τ0 time, and AdaIN represents the AdaIN operation.

[0023] Preferably, the potential attention shift comprises the following steps:

[0024] The distribution of keys and values ​​of the self-attention process in the edit branch during denoising is offset to the distribution of keys and values ​​of the self-attention process in the style branch by introducing a time-varying coefficient β t To complete, it is expressed as:

[0025]

[0026] Among them, [τ2,τ3] represents the introduced coefficient β t time interval, and denote the key and value of the self-attention process in the editing branch, respectively; and Represent the key and value of the self-attention process in the style branch respectively; and Represent the offset key and value respectively;

[0027] A balancing hyperparameter γ is introduced to balance the Q in the content branch t Do a simple linear weighting to alleviate the content loss caused by attention shift, expressed as:

[0028]

[0029] Preferably, before the migration potential features and style potential features based on the current timestamp are updated to generate the migration potential features of the next timestamp, a smooth update is performed using a sliding window smoothing technique based on optical flow, including the following steps:

[0030] The noise corresponding to the current timestamp is predicted through the Unet network, and the potential features are calculated based on it It is expressed as:

[0031]

[0032] Among them, α t represents the noise parameter in the diffusion model, ∈ t represents the noise predicted by the Unet network at step t;

[0033] Through the decoder The potential characteristics Decode frame by frame into pixel representation

[0034] In pixels Each frame in performs an optical flow-based deformation operation, expressed as:

[0035]

[0036] in, express The value after the transformation operation, and Represents potential characteristics The i-th frame and the j-th frame after decoding into the pixel space; Warp represents the deformation operation based on optical flow, and the optical flow model is used to estimate and The optical flow map between Perform deformation operations;

[0037] The smoothed updated pixel space representation is represented by the encoder ε Re-encoding to latent space representation The estimated noise is rescaled according to the following formula:

[0038]

[0039] Preferably, the updating of the migration potential features and style potential features based on the current timestamp to generate the migration potential features of the next timestamp is specifically to reconstruct the migration potential features of the next timestamp based on the smoothed migration potential features and the estimated noise, which is expressed as:

[0040]

[0041] Among them, α t-1 Represents the noise addition parameter in the diffusion model.

[0042] Preferably, the style transfer result is decoded frame by frame into pixel representation by a decoder, specifically: using the migration potential feature of the initial timestamp as the style transfer result, using a decoder The style transfer result is decoded into pixel representation frame by frame as the video after local style transfer.

[0043] The present invention also provides a local video style transfer system that does not require training, including:

[0044] The local selection module, the content branch denoises and reverses the video frames of the original video to obtain the content potential features Z at different timestamps t , t=1→T; in the process of denoising and inverting the original video, a mask is applied to the first frame of the original video, and the mask is transferred to all frames of the original video based on point matching to obtain all masks;

[0045] Style extraction module, the style branch performs denoising and inversion on the reference image to obtain the style potential features at different timestamps

[0046] Style transfer module, in the editing branch, the initial transfer potential features Denoising is performed based on the initial migrated latent features. During the denoising process, the style of the migrated latent features is migrated based on the AdaIN style migration technology to obtain the style migration result. The style migration result is decoded frame by frame into pixel representation through a decoder as the video after local style migration.

[0047] In the denoising process, the style migration of the migration potential features is performed based on the AdaIN style migration technology, including potential feature offset and attention offset; the potential feature offset is based on the migration potential features and style potential features of the previous timestamp to update the migration potential features of the current timestamp, and then based on the migration potential features and style potential features of the current timestamp to update the migration potential features of the next timestamp; repeat the above steps until the migration potential features of the initial timestamp are obtained as the style migration result; the attention offset updates the attention mechanism in the migration potential feature generation process based on the attention mechanism in the style potential feature generation process.

[0048] The present invention has the following beneficial effects:

[0049] (1) The present invention uses a mask propagation technique based on point matching, and only uses the feature map of the DDIM denoising inversion process to propagate the first frame mask input by the user to all frames of the video, thereby achieving more accurate local control and avoiding the tediousness of introducing an additional segmentation model;

[0050] (2) The present invention effectively transfers the style information of the reference image to the video through the style transfer technology based on AdaIN, and prevents the loss of video content information to a large extent;

[0051] (3) The present invention further alleviates the flickering and artifacts caused by the migration process through the sliding window smoothing technology based on optical flow.

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A method step diagram of an embodiment of the present invention;

[0054] Figure 2 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0055] Figure 3 A qualitative comparison between the embodiments of the present invention and other prior arts;

[0056] Figure 4 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0057] See also Figure 1 and Figure 2 The method step diagram and flow chart of an embodiment of the present invention are shown, which include the following steps:

[0058] S101, the content branch performs denoising and inversion on the video frames of the original video to obtain content potential features Z at different timestamps t (t=1→T); In the process of denoising and reversing the original video, a mask is applied to the first frame of the original video, and the mask is transferred to all frames of the original video based on point matching to obtain all masks;

[0059] S102, the style branch performs denoising and inversion on the reference image to obtain style potential features at different timestamps

[0060] S103, in the editing branch, the initial migration latent features are set equal to the initial content latent features, denoising is performed based on the initial migration latent features, and during the denoising process, the migration latent features are style-migrated based on the AdaIN style migration technology to obtain a style migration result; the style migration result is decoded frame by frame into pixel representation through a decoder as a video after local style migration.

[0061] The latent features of the content at different timestamps are represented as Z t (t=1→T), the style potential features of different timestamps are expressed as The initial transfer latent feature is equal to the initial content latent feature expressed as

[0062] Specifically, given an original video sequence consisting of N frames and a reference style image I s , the goal is to convert the original video sequence into a new sequence During this conversion process, the style of the reference image will be transferred to the original video while keeping the style of the main objects in the video unchanged. and the reference style image I s Applying DDIM denoising inversion, their respective latent feature representations Z can be obtained t and Then, let the initial latent feature representation of the editing process Depend on Figure 2 It can be seen that this method mainly includes three technical points. The first part is the mask propagation technology based on point matching: when the original video is denoised and inverted by DDIM, the feature map of a specific layer will be saved at a specific timestamp. According to the correspondence of the feature map, the mask of the first frame input by the user can be propagated to all subsequent frames; the second part is the style transfer technology based on AdaIN: a three-branch architecture is used to implement information interaction between content branch, editing branch and style branch in parallel, so as to transfer the style information of the reference image to the original video; the third part is the sliding window smoothing technology based on optical flow: the obtained optical flow information is used to perform sliding window smoothing operations on the video frames in the pixel space to alleviate the flicker and artifacts that may occur during the migration process. The following introduces these three technical points in turn.

[0063] 1. Mask propagation technology based on point matching

[0064] This technology uses the feature map of the UpBlock2 layer in the Unet structure corresponding to a certain timestamp t0 in the original video denoising and inversion process for point matching, and combines the idea of ​​KNN to propagate the mask information of the first frame provided by the user to all frames of the video. The timestamp t0 is determined according to the empirical value, generally in the middle of the denoising and inversion process, because its feature map contains the richest semantic information. The specific algorithm flow chart can be seen in the attached Figure 2 .

[0065] Based on the feature map F obtained during the denoising inversion process A With F B , we can calculate the cosine similarity between the two and find F A Medium A The best matching point under the index is in F B The corresponding index p in B , and finally determine the corresponding point relationship between frame A and frame B:

[0066]

[0067] Using such a matching mechanism, this embodiment propagates the mask based on the idea of ​​KNN. For each point in the i-th frame, the k most similar points in the i-1-th frame are considered respectively. If most of these k points are foreground, then the corresponding points in the i-th frame are also likely to belong to the foreground; otherwise, they belong to the background. In such an autoregressive manner, it is easy to propagate the first frame mask input by the user to all subsequent frames. However, such a propagation method is not efficient for lengthy videos. Noting that there is a large redundancy between frames, this embodiment designs a random downsampling strategy to make the entire propagation process more efficient. The algorithm is shown in Algorithm 1:

[0068] Algorithm 1

[0069]

[0070] As shown in lines 11 to 14 of Algorithm 1, the foreground and background regions of the i-1 frame are randomly downsampled at a sampling rate r, and the sampling rate r is further adjusted according to the relative proportion of the foreground and background. Although such a downsampling operation may lead to inaccuracy in point matching, the mask information in the feature map of the i-1 frame can still be effectively propagated to the corresponding region in the i-th frame. Therefore, this adjustment strategy improves the propagation efficiency while ensuring accuracy.

[0071] Finally, through all the masks M obtained in the above process, the potential feature representation Z in the content branch is t The foreground information and latent feature representation in the editing branch The background information is updated frame by frame:

[0072]

[0073] (II) Style transfer technology based on AdaIN

[0074] AdaIN technology mainly uses normalization operations to change the mean and variance of a distribution. Specifically, it shifts the mean and variance of distribution A to the mean and variance of distribution B through the following formula. Among them, μ(·) and σ(·) represent the corresponding mean and standard deviation respectively.

[0075]

[0076] This embodiment mainly uses AdaIN operations in two dimensions: the potential features in the denoising process The distribution shift (latent-shift) of the self-attention process, as well as the distribution shift (attention-shift) of the key and value. First, in a certain timestamp interval [τ0,τ1], by introducing a coefficient η that varies with time t , thereby gradually transforming the potential features in the edit branch The distribution of is shifted to the latent features in the style branch Distribution:

[0077]

[0078] Secondly, in a certain timestamp interval [τ2,τ3], the distribution of keys and values ​​of the self-attention process in the edit branch is shifted to the distribution of keys and values ​​of the self-attention process in the style branch. However, directly applying this operation will lead to the loss and blurring of local information. To solve this problem, a time-varying coefficient β is also introduced. t To complete the gradual transition of information:

[0079]

[0080] After updating the key and value in formula (5), the subsequent query and key matching operation will lead to further loss of content information. Similar to the previous method, this embodiment introduces a balancing hyperparameter γ to adjust the Q in the content branch t Do a simple linear weighting to alleviate the content loss caused by key and value updates:

[0081]

[0082] (III) Sliding window smoothing technology based on optical flow

[0083] This embodiment uses the estimated optical flow information and the sliding window smoothing technology to Updated to maintain temporal consistency to resolve flickering and artifacts during migration.

[0084] First, the corresponding optical flow map is estimated using the optical flow model. Using this optical flow information, the image of a certain frame can be warped to another frame, thereby achieving the effect of information alignment. Next, in a certain timestamp interval [τ4, τ5], the noise corresponding to the current timestamp is predicted through the Unet network, and the potential features are calculated based on this.

[0085]

[0086] Among them, α t represents the noise parameter in the diffusion model, ∈ θ Represents the noise predicted by the Unet network.

[0087] Then through the decoder Decode these features frame by frame into pixel representation Next, through a sliding window of size 2m, the other frames in each window are warped to key frame i in turn, and the results of all warps in the window are averaged and pooled to complete the update of key frame i to achieve a local smoothing effect:

[0088]

[0089] The keyframe updated in the current window will participate in the calculation of the next window, as shown in the attached Figure 1 The pixel space representation is then transformed through the encoder ε. Re-decoding to latent space representation The estimated noise is then rescaled according to the following formula:

[0090]

[0091] Finally, DDIM is used to update the latent features:

[0092]

[0093] The above three technical points will perform denoising iterations in different timestamp intervals, and then finally pass through the decoder Decode these features frame by frame into pixel representation This is the final video after style transfer.

[0094] The embodiments of the present invention and other existing technologies are verified on two data sets, and the overall style transfer indicators (ArtFID, FID, LPIPS) and foreground style preservation indicators (SSIM, CLIP-I), as well as temporal consistency (CLIP-F) are calculated respectively.

[0095] The quantitative comparison of the experimental results is shown in Table 1. According to Table 1, it can be seen that this embodiment completely surpasses all other existing methods in terms of foreground style preservation; in terms of overall style transfer, it achieves a trade-off between content fidelity and style diversity (the comprehensive indicator AriFID is relatively high), surpassing most of the existing methods; and in terms of temporal consistency, it surpasses all existing methods.

[0096] Table 1

[0097]

[0098] The qualitative comparison of the experimental results is Figure 3 As shown, it can be seen that this embodiment surpasses the existing style transfer method in terms of visual effect, and also surpasses the existing style transfer method in terms of local style preservation.

[0099] See also Figure 4 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0100] Local selection module 401, the content branch performs denoising and inversion on the video frames of the original video to obtain the content potential features Z of different timestamps t (t=1→T); In the process of denoising and reversing the original video, a mask is applied to the first frame of the original video, and the mask is transferred to all frames of the original video based on point matching to obtain all masks;

[0101] Style extraction module 402, the style branch performs denoising and inversion on the reference image to obtain style potential features at different timestamps

[0102] Style transfer module 403, in the editing branch, the initial transfer potential features Denoising is performed based on the initial migrated latent features. During the denoising process, the style of the migrated latent features is transferred based on the AdaIN style transfer technology to obtain the style transfer result. The style transfer result is decoded frame by frame into pixel representation through a decoder as the video after local style transfer.

[0103] The present invention proposes a local video style transfer method that does not require training, which meets the user's specific needs for migrating local areas while reducing computational overhead. Through the mask propagation technology based on point matching, the first frame mask input by the user can be propagated to all frames of the video by only using the feature map of the DDIM denoising inversion process, thereby achieving more accurate local control and avoiding the introduction of additional models. Through the style transfer technology based on AdaIN, the style information of the reference image is effectively transferred to the video, and the loss of video content information is prevented to a large extent. Through the sliding window smoothing technology based on optical flow, the flickering and artifacts caused by the migration process are further alleviated. The experimental comparison results show that our method surpasses most of the current technologies quantitatively and qualitatively, especially for the preservation of video foreground style.

[0104] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A local video style transfer method without training, characterized in that: The following steps are involved: The content branch denoises and reverses the video frames of the original video to obtain the content potential features Z at different timestamps. t , t=1→T; in the process of denoising and inverting the original video, a mask is applied to the first frame of the original video, and the mask is transferred to all frames of the original video based on point matching to obtain all masks; The style branch denoises and inverts the reference image to obtain the style potential features at different timestamps. t=1→T; In the edit branch, let the initial migration potential feature Denoising is performed based on the initial migrated latent features. During the denoising process, the style of the migrated latent features is transferred based on the AdaIN style transfer technology to obtain the style transfer result; The style transfer result is decoded frame by frame into pixel representation through a decoder as the video after local style transfer; In the denoising process, the style migration technology based on AdaIN is used to migrate the style of the potential features, including potential feature shift and attention shift; The latent feature shift generates a migration latent feature of the current timestamp based on the migration latent feature and the style latent feature of the previous timestamp, and then generates a migration latent feature of the next timestamp based on the migration latent feature and the style latent feature of the current timestamp; Repeat the above steps until the migration potential features of the initial timestamp are obtained as the style migration result; the attention offset updates the attention mechanism in the migration potential feature generation process based on the attention mechanism in the style potential feature generation process.

2. The method for local video style transfer without training according to claim 1, characterized in that: The point matching-based transfer of the mask to all video frames of the original video includes the following steps: In the process of denoising and reversing the original video, the feature maps of all frames of the original video at a specific timestamp t0 are saved as mask propagation feature maps; In the first frame of the mask propagation feature map, a mask is applied according to the local requirements for style transfer; Based on point matching, the mask of the current frame is transferred to the next frame until all frames in the mask propagation feature map are masked to obtain all masks M; the mask of the current frame is transferred to the next frame based on point matching, which is expressed as: Among them, F A represents the mask propagation feature map of the current frame, p A Indicates F A Point in the middle; F B represents the mask propagation feature map of the next frame, p B Indicates F B In and p A corresponding points; Dist represents cosine similarity, Indicates that when Dist reaches its minimum value, p B The value of .

3. The method for local video style transfer without training according to claim 2, characterized in that: In the process of transferring the mask to all video frames of the original video based on point matching, the foreground area and the background area of ​​the current frame are randomly downsampled at a sampling rate r, and the sampling rate r is further adjusted according to the ratio of the foreground area to the background area, thereby reducing the computational overhead.

4. The method for local video style transfer without training according to claim 1, characterized in that: The potential feature shift includes the following steps: Update the migration potential features based on the entire mask M It is expressed as: Introduce a time-varying coefficient η t Updated migration potential features Further update is performed, expressed as: Among them, [τ0,τ1] represents the introduced coefficient η t time interval, represents the value of η at time τ1, represents the value of η at τ0 time, and AdaIN represents the AdaIN operation.

5. The method for local video style transfer without training according to claim 1, characterized in that: The potential attention shift includes the following steps: The distribution of keys and values ​​of the self-attention process in the edit branch during denoising is offset to the distribution of keys and values ​​of the self-attention process in the style branch by introducing a time-varying coefficient β t To complete, it is expressed as: Among them, [τ2,τ3] represents the introduced coefficient β t time interval, and denote the key and value of the self-attention process in the editing branch, respectively; and Represent the key and value of the self-attention process in the style branch respectively; and Represent the offset key and value respectively; A balancing hyperparameter γ is introduced to balance the Q in the content branch t Do a simple linear weighting to alleviate the content loss caused by attention shift, expressed as:

6. The method for local video style transfer without training according to claim 1, characterized in that: Before the migration potential features and style potential features based on the current timestamp are updated to generate the migration potential features of the next timestamp, a smooth update is performed using a sliding window smoothing technique based on optical flow, including the following steps: The noise corresponding to the current timestamp is predicted through the Unet network, and the potential features are calculated based on it It is expressed as: Among them, α t represents the noise parameter in the diffusion model, ∈ t represents the noise predicted by the Unet network at step t; Through the decoder The potential characteristics Decode frame by frame into pixel representation In pixels Each frame in performs an optical flow-based deformation operation, expressed as: in, express The value after the transformation operation, and Represents potential characteristics The i-th frame and the j-th frame after decoding into the pixel space; Warp represents the deformation operation based on optical flow, and the optical flow model is used to estimate and The optical flow map between Perform deformation operations; The smoothed updated pixel space representation is represented by the encoder ε Re-encoding to latent space representation The estimated noise is rescaled according to the following formula:

7. The method for local video style transfer without training according to claim 6, characterized in that: The migration potential feature of the next timestamp is updated based on the migration potential feature of the current timestamp and the style potential feature, specifically, the migration potential feature of the next timestamp is reconstructed based on the smoothed migration potential feature and the estimated noise, which is expressed as: Among them, α t-1 Represents the noise addition parameter in the diffusion model.

8. The method for local video style transfer without training according to claim 1, characterized in that: The style transfer result is decoded frame by frame into pixel representation by a decoder, specifically: the migration potential feature of the initial timestamp is used as the style transfer result, and the decoder is used The style transfer result is decoded into pixel representation frame by frame as the video after local style transfer.

9. A local video style transfer system without training, characterized in that: include: The local selection module, the content branch denoises and reverses the video frames of the original video to obtain the content potential features Z at different timestamps t , t=1→T; in the process of denoising and inverting the original video, a mask is applied to the first frame of the original video, and the mask is transferred to all frames of the original video based on point matching to obtain all masks; Style extraction module, the style branch performs denoising and inversion on the reference image to obtain the style potential features at different timestamps t=1→T; Style transfer module, in the editing branch, the initial transfer potential features Denoising is performed based on the initial migrated latent features. During the denoising process, the style of the migrated latent features is transferred based on the AdaIN style transfer technology to obtain the style transfer result; The style transfer result is decoded frame by frame into pixel representation through a decoder as the video after local style transfer; In the denoising process, the style migration technology based on AdaIN is used to migrate the style of the potential features, including potential feature shift and attention shift; The latent feature shift generates a migration latent feature of the current timestamp based on the migration latent feature and the style latent feature of the previous timestamp, and then generates a migration latent feature of the next timestamp based on the migration latent feature and the style latent feature of the current timestamp; Repeat the above steps until the migration potential features of the initial timestamp are obtained as the style migration result; the attention offset updates the attention mechanism in the migration potential feature generation process based on the attention mechanism in the style potential feature generation process.

Images