A training-free local video style transfer method and system

Abstract

The application discloses a kind of local video style migration method and system without training, method includes: content branch is to the video frame of original video denoising reverse, obtain different time stamp content latent feature Z t (t=1→T);In the process of denoising reverse to original video, the first frame video frame of original video is marked with mask, mask is passed to all video frames of original video based on point matching;Style branch is to reference image denoising reverse, obtain different time stamp style latent feature editing branch, let initial migration latent feature be based on initial migration latent feature denoising, in the process of denoising, the style migration technology based on AdaIN is carried out to the style migration of migration latent feature, obtains style migration result, is decoded to pixel representation frame by frame by decoder, as the video after local style migration.The application can meet the demand of user to local area migration, reduce the flicker and artifact caused in migration process simultaneously.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a local video style transfer method and system without training. BACKGROUND

[0002] Video stylization, as a subfield of video editing, is attracting more and more attention. It involves adding artistic styles to video content. With the continuous progress of the generative field, there are many methods that use diffusion models to perform style transfer on videos. In these methods, users can simply complete the style rendering of the video through text or images. For example, StyleCrafter combines the reference picture input by the user to complete personalized video generation by fine-tuning the Adapter; Diffutoon completes the cartoon coloring task of the video through text information; BIVDiff uses the downstream image editing model to perform frame-by-frame style transfer, and smoothes the temporal consistency through the T2V model; AnyV2V uses the downstream image editing model to perform style transfer on the first frame of the video, and propagates the transfer result of the first frame to all subsequent frames through the I2V model.

[0003] Existing video style transfer methods are mainly aimed at large-scale style pictures, and the model is fine-tuned to learn the prior information of a specific style, which usually requires a large amount of data and heavy computational overhead. In addition, some users want to perform style conversion on the region of interest in the video, but the existing methods are based on global style transfer and cannot meet the specific user needs. Moreover, directly extending some advanced image transfer methods to the video field will cause temporal inconsistency phenomena such as flicker and artifacts. SUMMARY

[0004] The present application aims to solve the problem that the prior art cannot realize local style transfer.

[0005] The technical scheme adopted by the present application to solve its technical problems is to provide a local video style transfer method without training, comprising the following steps:

[0006] The content branch denoises and reverses the video frames of the original video to obtain content latent features Z t , t = 1 → T; During the denoising and reversing process of the original video, the first frame of the original video is marked with a mask, and the mask is transmitted to all video frames of the original video based on point matching to obtain all masks;

[0007] The style branch denoises and reverses the reference image to obtain style latent features

[0008] In the editing branch, the initial transfer latent features The denoising process is based on the AdaIN-based style transfer technology to perform style transfer on the migration latent features, and a style transfer result is obtained; and the style transfer result is decoded frame by frame to pixel representation by the decoder, as a video after local style transfer.

[0009] The denoising process is based on the AdaIN-based style transfer technology to perform style transfer on the migration latent features, and a style transfer result is obtained; and the style transfer result is decoded frame by frame to pixel representation by the decoder, as a video after local style transfer.

[0010] Preferably, the mask is transmitted to all video frames of the original video based on point matching, including the following steps:

[0011] In the process of denoising and reversing the original video, the feature map of all frames of the original video at a specific timestamp t0 is saved as a mask propagation feature map.

[0012] In the first frame of the mask propagation feature map, a mask is marked according to the local demand for style transfer.

[0013] The mask of the current frame is transmitted to the next frame based on point matching until all frames in the mask propagation feature map are marked with a mask, and all masks M are obtained; the mask of the current frame is transmitted to the next frame based on point matching, which is represented as:

[0014]

[0015] wherein, F A represents the current frame mask propagation feature map, p A represents a point in F A ; F B represents the next frame mask propagation feature map, p B represents a point in F B corresponding to p A ; Dist represents the cosine similarity, represents the value of p B when Dist reaches the minimum value.

[0016] Preferably, in the process of transmitting the mask to all video frames of the original video based on point matching, the foreground and background regions of the current frame are randomly downsampled at a sampling rate r, and the sampling rate r is further adjusted according to the ratio of the foreground and background regions, thereby reducing computational overhead.

[0017] Preferably, the latent feature offset includes the following steps:

[0018] Update the latent features of the migration based on the full mask M. Represented as:

[0019]

[0020] Introduce a time-varying coefficient η t Updated migration potential features Further updates are indicated as follows:

[0021]

[0022] Where [τ0,τ1] represents the introduction of the coefficient η t The time interval This represents the value of η at time τ1. This represents the value of η at time τ0, and AdaIN represents the AdaIN operation.

[0023] Preferably, the potential attention shift includes the following steps:

[0024] The distribution of keys and values ​​in the self-attention process during the denoising process in the editing branch is shifted to the distribution of keys and values ​​in the self-attention process in the style branch. This shift is achieved by introducing a time-varying coefficient β. t To complete, as shown below:

[0025]

[0026] Where [τ2,τ3] represents the introduction of coefficient β t The time interval and These represent the key and value of the self-attention process in the edit branch, respectively; and These represent the key and value of the self-attention process in the style branch, respectively; and These represent the key and value after the offset, respectively;

[0027] Introduce a balancing hyperparameter γ to balance the editing branches Q in the content branch t To mitigate the content loss caused by attention shift, a simple linear weighting is performed, expressed as:

[0028]

[0029] Preferably, before the migration latent feature and the style latent feature based on the current timestamp are updated to generate the migration latent feature of the next timestamp, a smoothing update is performed through a light flow-based sliding window smoothing technique, including the following steps:

[0030] The noise corresponding to the current timestamp is predicted through the Unet network, and the latent feature is calculated accordingly is expressed as:

[0031]

[0032] wherein α t represents a noise addition parameter in the diffusion model, ∈ t represents the noise predicted by the Unet network at the t-th step;

[0033] The latent feature is decoded frame by frame into a pixel representation

[0034] Each frame in the pixel representation is sequentially subjected to a light flow-based deformation operation, which is expressed as:

[0035]

[0036] wherein represents the value after the deformation operation, and represent the i-th frame and the j-th frame of the latent feature decoded into the pixel space; Warp represents the light flow-based deformation operation, which estimates the light flow map between and based on the light flow model, and deforms based on the light flow map;

[0037] The pixel space representation after the smoothing update is re-encoded into the latent space representation by the encoder εThe estimated noise is re-adjusted according to the following formula:

[0038]

[0039] Preferably, the migration latent feature and the style latent feature based on the current timestamp are updated to generate the migration latent feature of the next timestamp, specifically, the migration latent feature of the next timestamp is reconstructed based on the smoothed migration latent feature and the estimated noise, which is expressed as:

[0040]

[0041] wherein, alpha t-1 represents a noise adding parameter in the diffusion model.

[0042] Preferably, the style transfer result is decoded frame by frame to a pixel representation by a decoder, specifically, taking the transfer latent feature of the initial timestamp as the style transfer result, and using the decoder to decode the style transfer result to a pixel representation. The style transfer result is decoded frame by frame to a pixel representation, as a video after local style transfer.

[0043] The application also provides a local video style transfer system without training, comprising:

[0044] The local selection module, the content branch denoises and reverses the video frames of the original video to obtain content latent features Z t , t = 1 -> T; during the denoising and reversing of the original video, the first video frame of the original video is marked with a mask, and the mask is transmitted to all video frames of the original video based on point matching to obtain all masks;

[0045] The style extraction module, the style branch denoises and reverses the reference image to obtain style latent features

[0046] The style transfer module, in the editing branch, takes the initial transfer latent feature based on the initial transfer latent feature, and the denoising process is performed on the transfer latent feature based on the AdaIN style transfer technology to obtain a style transfer result; the style transfer result is decoded frame by frame to a pixel representation by a decoder, as a video after local style transfer.

[0047] The denoising process is performed on the transfer latent feature based on the AdaIN style transfer technology, including latent feature offset and attention offset; the latent feature offset updates to generate the transfer latent feature of the current timestamp based on the transfer latent feature and the style latent feature of the previous timestamp, and then updates to generate the transfer latent feature of the next timestamp based on the transfer latent feature and the style latent feature of the current timestamp; the above steps are repeated until the transfer latent feature of the initial timestamp is obtained as the style transfer result; the attention offset updates the attention mechanism in the generation process of the transfer latent feature based on the attention mechanism in the generation process of the style latent feature.

[0048] The application has the following beneficial effects:

[0049] (1) The application can propagate the first frame mask input by the user to all frames of the video by using the feature map of the DDIM denoising reverse process only, thereby achieving more accurate local control and avoiding the complexity of introducing an additional segmentation model;

[0050] (2) The application effectively transfers the style information of the reference picture to the video by the AdaIN-based style transfer technology, and largely prevents the loss of video content information;

[0051] (3) The application further alleviates the flicker and artifacts caused by the migration process by the optical flow-based sliding window smoothing technology.

[0052] The application will be further described in detail in combination with the accompanying drawings and examples, but the application is not limited to the examples. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The method step diagram of the embodiment of the application;

[0054] Figure 2 The flowchart of the embodiment of the application;

[0055] Figure 3 The qualitative comparison of the embodiment of the application and other prior art;

[0056] Figure 4 The system structure diagram of the embodiment of the application. DETAILED DESCRIPTION

[0057] Referring to FIGS. Figure 1 and Figure 2 The method step diagram and the flowchart of the embodiment of the application, including the following steps:

[0058] S101, the content branch denoises and reverses the video frames of the original video to obtain content latent features Z of different timestamps t (t=1→T); during the denoising and reversing of the original video, the first frame of the original video is marked with a mask, the mask is transmitted to all video frames of the original video based on point matching, and all masks are obtained;

[0059] S102, the style branch denoises and reverses the reference image to obtain style latent features

[0060] S103, in the editing branch, let the initial migration latent feature equal the initial content latent feature, denoising based on the initial migration latent feature, in the denoising process, the style migration technology based on AdaIN performs style migration on the migration latent feature, and obtains a style migration result; the style migration result is decoded frame by frame to pixel representation by the decoder, as the video after local style migration.

[0061] The content latent feature of different timestamps is represented as Z t (t = 1 → T), the style latent feature of different timestamps is represented as The initial migration latent feature equal to the initial content latent feature is represented as

[0062] Specifically, given an original video sequence composed of N frames and a reference style image I s , the goal is to convert the original video sequence into a new sequence In this conversion process, the style of the reference image will be transferred to the original video while keeping the style of the main objects in the video unchanged. By applying DDIM denoising inversion to the original video sequence and the reference style image I s , their respective latent feature representations Z t and Then, let the initial latent feature representation of the editing process be Figure 2 It can be seen that this method mainly includes three technical points, the first part is the mask propagation technology based on point matching: when doing DDIM denoising inversion on the original video, the feature map of a specific layer at a specific timestamp is saved, and according to the corresponding relationship of the feature map, the first frame mask input by the user can be propagated to all subsequent frames; the second part is the style migration technology based on AdaIN: a three-branch architecture is used to realize the information interaction of the content branch, the editing branch and the style branch in parallel, so as to transfer the style information of the reference image to the original video; the third part is the sliding window smoothing technology based on optical flow: the optical flow information obtained is used to perform sliding window smoothing operation on the video frames in the pixel space, so as to alleviate the flicker and artifacts that may occur in the migration process. The following introduces these three technical points in turn.

[0063] (1) Mask propagation technology based on point matching

[0064] The technology utilizes point matching of the feature map of the UpBlock2 layer in the Unet structure corresponding to a timestamp t0 in the original video denoising reverse process, and combines the idea of KNN, so as to propagate the first frame mask information provided by the user to all frames of the video. The determination of the timestamp t0 is in accordance with the experience value, and generally in the middle of the denoising reverse, because the semantic information contained in the feature map is the most abundant. The specific algorithm flowchart can be seen in the attached Figure 2 .

[0065] Based on the feature map F A obtained in the denoising reverse process B , the cosine similarity of the two can be calculated, so as to find the most matched point in F A under the index p A , and the index p B corresponding to the point in F B , and finally determine the corresponding point relationship between frame A and frame B:

[0066]

[0067] Using such a matching mechanism, the embodiment propagates the mask based on the idea of KNN. For each point in the i-th frame, consider its most similar k points in the i-1-th frame. If most of the k points are foreground, then the corresponding point in the i-th frame is also likely to belong to the foreground; otherwise, it belongs to the background. In this way, the first frame mask input by the user can be easily propagated to all subsequent frames in a self-regressive manner. However, such a propagation method is not efficient for long videos. Noting that there is a large redundancy between frames, the embodiment designs a random down-sampling strategy to make the entire propagation process more efficient, as shown in Algorithm 1:

[0068] Algorithm 1

[0069]

[0070] As shown in lines 11 to 14 of Algorithm 1, the foreground and background regions of the i-1-th frame are randomly down-sampled at a sampling rate r, and the sampling rate r is further adjusted according to the relative proportion of the foreground and the background. Although such a down-sampling operation may cause inaccuracy in point matching, the mask information in the i-1-th frame feature map can still be effectively propagated to the corresponding region in the i-th frame. Therefore, this adjustment strategy improves the propagation efficiency while ensuring accuracy.

[0071] Finally, through all the masks M obtained in the above process, the foreground information of the latent feature representation Z t in the content branch is fused and updated with the background information of the latent feature representation in the editing branch frame by frame:

[0072]

[0073] (II) Style Transfer Techniques Based on AdaIN

[0074] The AdaIN technique primarily uses normalization to alter the mean and variance of a distribution. Specifically, it shifts the mean and variance of distribution A to those of distribution B using the following formula: where μ(·) and σ(·) represent the corresponding mean and standard deviation, respectively.

[0075]

[0076] This embodiment primarily utilizes AdaIN operations in two dimensions: latent features during the denoising process. The distribution shift (latent-shift) and the distribution shift of key and value during self-attention (attention-shift) are discussed. First, within a certain timestamp interval [τ0, τ1], a time-varying coefficient η is introduced. t This gradually reveals the potential features within the editing branches. The distribution shifts to latent features in style branches Distribution:

[0077]

[0078] Secondly, within a certain timestamp interval [τ2, τ3], the key-value distribution of the self-attention process in the editing branch is shifted to the key-value distribution of the self-attention process in the style branch. However, directly applying this operation can lead to the loss and blurring of local information. To address this issue, a time-varying coefficient β is also introduced. t To complete the gradual transition of information:

[0079]

[0080] After updating the key and value using formula (5), subsequent query-key matching operations will lead to further loss of content information. Similar to the previous method, this embodiment introduces a balancing hyperparameter γ to balance the content information in the edit branch. Q in the content branch t Perform a simple linear weighting to mitigate the content loss caused by key and value updates:

[0081]

[0082] (III) Sliding window smoothing technique based on optical flow

[0083] This embodiment uses the estimated optical flow information to update the potential features with a sliding window smoothing technique to maintain temporal consistency, thus solving the flickering and artifact problem during the migration process.

[0084] First, the corresponding optical flow map is estimated using the optical flow model. Using this optical flow information, the image of a certain frame can be warped to another frame, thereby achieving information alignment. Next, in a certain timestamp interval [τ4, τ5], the noise corresponding to the current timestamp is predicted by the Unet network, and the potential features

[0085]

[0086] where α t represents the noise addition parameter in the diffusion model, ∈ θ represents the noise predicted by the Unet network.

[0087] These features are then decoded frame by frame to pixel representation by the decoder Next, through a sliding window of size 2m, each window is sequentially warped to the key frame i, and the average pooling of all warped results in the window is performed, thereby completing the update of the key frame i to achieve local smoothing:

[0088]

[0089] The key frame updated in the current window will participate in the calculation of the next window, as shown in the right side of the schematic diagram in the accompanying Figure 1 Next, the pixel space representation is re-decoded to the latent space representation Then the estimated noise is adjusted according to the following formula:

[0090]

[0091] Finally, the potential features are updated using DDIM:

[0092]

[0093] The above three technical points will be iterated for denoising in different timestamp intervals, and then finally decoded by the decoder to pixel representation is the final style-transferred video.

[0094] ​The embodiment of the present application and other prior arts are verified on two data sets, and the overall style transfer indicators (ArtFID, FID, LPIPS) and the foreground style preservation indicators (SSIM, CLIP-I), and the temporal consistency (CLIP-F) are calculated.

[0095] The quantitative comparison of the experimental results is shown in Table 1, and according to Table 1, it can be seen that the embodiment completely exceeds all other existing methods in the foreground style preservation; achieves the trade-off between content fidelity and style diversity (higher comprehensive indicator AriFID) in the overall style transfer, and exceeds most of the existing methods; and exceeds all existing methods in the temporal consistency.

[0096] Table 1

[0097]

[0098] The qualitative comparison of the experimental results is shown in Figure 3 It can be seen that the embodiment exceeds the existing style transfer methods in visual effect, and also exceeds the existing style transfer methods in local style preservation.

[0099] Referring to Figure 4 Fig. 1 is a system structure diagram of the embodiment of the present application, which includes:

[0100] The local selection module 401 denoises and inverts the video frames of the original video to obtain content latent features Z t (t = 1 → T); during the denoising and inversion of the original video, the first video frame of the original video is marked with a mask, and the mask is transmitted to all video frames of the original video based on point matching to obtain all masks;

[0101] The style extraction module 402 denoises and inverts the reference image to obtain style latent features

[0102] The style transfer module 403, in the editing branch, sets the initial transfer latent feature based on the initial transfer latent feature, denoises the transfer latent feature, and performs style transfer on the transfer latent feature based on the AdaIN style transfer technology during the denoising process to obtain a style transfer result; and decodes the style transfer result to a pixel representation frame by frame through the decoder as a video after local style transfer.

[0103] The application proposes a local video style transfer method without training, which meets the specific needs of users for local area transfer while reducing the computational overhead. Through the mask propagation technology based on point matching, only the feature map of the DDIM denoising reverse process is used to propagate the first frame mask input by the user to all frames of the video, thereby achieving more accurate local control and avoiding the introduction of additional models. Through the style transfer technology based on AdaIN, the style information of the reference picture is effectively transferred to the video, and the loss of video content information is largely prevented. Through the sliding window smoothing technology based on optical flow, flicker and artifacts caused by the transfer process are further alleviated. The experimental comparison results show that our method surpasses most of the current technologies in quantitative and qualitative aspects, especially for the preservation of video foreground style.

[0104] The above is only a preferred embodiment of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A training-free local video style transfer method, characterized in that, comprising the following steps: The content branch denoises and reverses the video frames of the original video to obtain content latent features Z with different timestamps t In the process of denoising and reversing the original video, a mask is applied to the first video frame of the original video, and the mask is transmitted to all video frames of the original video based on point matching to obtain all masks; The style branch denoises the reference image to obtain the style latent feature of different timestamps t = 1 → T; In the editing branch, the initial migration latent feature is obtained by editing the initial image and the initial latent feature Based on the initial migration latent feature, denoising is performed, and in the denoising process, the style migration technology based on AdaIN is used to perform style migration on the migration latent feature to obtain a style migration result. decoding the style transfer result frame by frame to pixel representation through the decoder as the video after local style transfer; the style transfer technology based on AdaIN in the denoising process performs style transfer on the transfer latent feature, including latent feature offset and attention offset; the latent feature offset updates the transfer latent feature at the current timestamp based on the transfer latent feature at the previous timestamp and the style latent feature, and then updates the transfer latent feature at the next timestamp based on the transfer latent feature at the current timestamp and the style latent feature; the above steps are repeated until the transfer latent feature at the initial timestamp is obtained as the style transfer result; the attention offset updates the attention mechanism in the transfer latent feature generation process based on the attention mechanism in the style latent feature generation process; the latent feature offset comprises the following steps: updating the migration latent feature based on the entire mask M is represented as: Introducing a time-varying coefficient η t On the updated transferred latent features Further update, denoted as: wherein [τ0, τ1] denotes the time interval in which the coefficient η is introduced t denotes the value of η at time τ1, denotes the value of η at time τ0, AdaIN denotes an AdaIN operation;​ the attention offset comprises the following steps: shift the distribution of keys and values of the self-attention process in the editing branch during the denoising process to the distribution of keys and values of the self-attention process in the style branch, the shift being accomplished by introducing a time-varying coefficient β t as: wherein [τ2, τ3] represents the time interval of introducing the coefficient β t , and represent the key and value of the self-attention process in the editing branch, respectively; and represent the key and value of the self-attention process in the style branch, respectively; and represent the key and value after the offset, respectively. A balance hyper-parameter γ is introduced to scale the editing branch Q in the content branch t A simple linear weighting is done to alleviate the content loss due to attention drift, denoted as:

2. The training-free local video style transfer method of claim 1, wherein, the process of transferring the mask to all video frames of the original video based on point matching comprises the following steps: in the process of denoising inversion of the original video, the feature map of all frames of the original video at a specific timestamp t0 is saved as a mask propagation feature map; in the first frame of the mask propagation feature map, a mask is marked according to the local demand for style transfer; the mask of the current frame is transferred to the next frame based on point matching until all frames in the mask propagation feature map are marked with a mask, and all masks M are obtained; the mask of the current frame is transferred to the next frame based on point matching, which is represented as: where F A represents the current frame mask propagation feature map, p A represents a point in F A ; F B represents the next frame mask propagation feature map, p B represents a point in F B corresponding to p A ; Dist represents the cosine similarity, represents the value of p B when Dist reaches the minimum value.

3. The training-free local video style transfer method of claim 2, wherein, in the process of transferring the mask to all video frames of the original video based on point matching, the foreground area and the background area of the current frame are randomly down-sampled at a sampling rate r, and the sampling rate r is further adjusted according to the proportion of the foreground area and the background area, so as to reduce the calculation overhead.

4. The training-free local video style transfer method of claim 1, wherein, Before the transfer latent feature at the next timestamp is generated based on the transfer latent feature at the current timestamp and the style latent feature update, a smoothing update is performed through a sliding window smoothing technology based on optical flow, comprising the following steps: The noise corresponding to the current timestamp is predicted through the Unet network, and the potential features are calculated accordingly is represented as: wherein, α t represents a noise adding parameter in the diffusion model, ∈ t represents the noise predicted by the Unet network at the tth step; By a decoder Decoding latent features Frame-wise decoding to pixel representation Represented in pixels Each frame in the process undergoes an optical flow-based deformation operation, represented as: wherein, denotes the value after the warping operation, and denotes the latent feature the i-th and j-th frame after decoding to the pixel space; Warp denotes a warping operation based on the optical flow, which estimates the optical flow map between and based on the optical flow map warps . Smoothed updated pixel space representation Re-encode to latent space representation Re-adjust the estimated noise according to the following formula:

5. The training-free local video style transfer method of claim 4, wherein, the transfer latent feature at the next timestamp is generated based on the smoothed transfer latent feature and the estimated noise, which is represented as: where a t-1 denotes the noise parameter in the diffusion model.

6. The training-free local video style transfer method of claim 1, wherein, The style migration result is decoded frame by frame to a pixel representation by the decoder, specifically: taking the migration latent feature with the initial timestamp as the style migration result, and using the decoder The style migration result is decoded frame by frame to a pixel representation, as a video after local style migration.

7. A training-free local video style transfer system, comprising: comprising: The local selection module reverses the denoising of the video frames of the original video to obtain the content potential features Z with different timestamps t In the process of reversing the denoising of the original video, a mask is marked on the first video frame of the original video, the mask is transmitted to all video frames of the original video based on point matching, and all masks are obtained The style extraction module denoises and reverses the reference image to obtain style latent features of different timestamps t = 1 → T; The style transfer module edits the initial transfer latent feature in the branch, and the editing branch is used for editing the initial transfer latent feature Based on the initial transfer latent feature, the denoising process is performed based on the AdaIN style transfer technology to transfer the transfer latent feature, and the style transfer result is obtained; decoding the style transfer result frame by frame to pixel representation through the decoder as the video after local style transfer; the style transfer technology based on AdaIN in the denoising process performs style transfer on the transfer latent feature, including latent feature offset and attention offset; the latent feature offset updates the transfer latent feature at the current timestamp based on the transfer latent feature at the previous timestamp and the style latent feature, and then updates the transfer latent feature at the next timestamp based on the transfer latent feature at the current timestamp and the style latent feature; the above steps are repeated until the transfer latent feature at the initial timestamp is obtained as the style transfer result; the attention offset updates the attention mechanism in the transfer latent feature generation process based on the attention mechanism in the style latent feature generation process; the latent feature offset comprises the following steps: updating the migration latent feature based on the entire mask M is represented as: Introducing a time-varying coefficient η t On the updated transferred latent features Further update, denoted as: wherein [τ0, τ1] denotes a time interval in which the coefficient η is introduced t denotes the value of η at time τ1, denotes the value of η at time τ0, AdaIN denotes an AdaIN operation;​ the attention offset comprises the following steps: shift the distribution of keys and values of the self-attention process in the editing branch during the denoising process to the distribution of keys and values of the self-attention process in the style branch, the shift being done by introducing a time-varying coefficient β t is done, denoted as: wherein [τ2, τ3] represents the time interval of introducing the coefficient β t , and respectively represent the key and value of the self-attention process in the editing branch; and respectively represent the key and value of the self-attention process in the style branch; and respectively represent the offset key and value; A balance hyper-parameter γ is introduced to scale the editing branch Q in the content branch t A simple linear weighting is done to alleviate the content loss due to attention drift, denoted as:

Images