Video style conversion method and device based on stable diffusion network, and program product

Abstract

The invention provides a video style conversion method and device based on a stable diffusion network, and a program product. The method comprises the following steps: receiving a video of a first video style; converting the video of the first video style into a video of a second video style by using an enhanced stable diffusion network according to the text indication of style conversion; the enhanced stable diffusion network is obtained through the following steps: S1, obtaining a lightweight Unet network through a distillation method iterative training mode; s2, the obtained lightweight UNet network is enhanced, an enhanced lightweight UNet network structure is obtained, and enhancement comprises the steps that a noise information reduction module is introduced; the noise information reduction module comprises a feature combination learning module; introducing a semantic segmentation graph module; a feature information enhancement module is introduced and comprises a pooling layer, a feature information connection module and a semantic segmentation fusion module. By means of the technical scheme, video style conversion can be completed, meanwhile, the equipment performance requirement needed by system operation is lowered, and the video conversion quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a video style conversion method, device and program product based on a stable diffusion network. Background Art

[0002] Stable Diffusion is a latent diffusion model LDM for text to image txt2img, which was implemented and open-sourced by researchers from CompVis, Stability AI and LAION. Figure 1 The biggest advantage of SD is that it performs loop iterations in the latent space with relatively small feature map scale, which improves the computational efficiency of UNet.

[0003] Among them, S is the number of samples / iterations, which is generally 50-100 steps.

[0004] based on Figure 1 , describe the whole process of txt2image processing step by step:

[0005] (1) Input the prompt word "a photo of cartoon" and convert it into 77×768 TokenEmbeddings after being processed by the Text Encoder component. The Embeddings are then input into the Image Information Creator component.

[0006] (2) The random image information tensor is a 64×64 image representation randomly generated by a latent seed (Gaussian noise ~ N(0,1)). It represents a completely noisy image and serves as another initial input of the Image Information Creator component.

[0007] (3) Through the processing of the Image Information Creator component, a 64×64 processed image tensor containing image information is generated. This output contains the information of the image with the semantic information of the previous input prompt word. This process is called diffusion.

[0008] (4) The processed image tensor information generated in the previous step is processed by the ImageDecoder component to generate the final 512×512 image output related to the input Prompt word.

[0009] The denoising process of the scheduler is as follows: Figure 2 As shown below. The figure below is just a schematic diagram for easy understanding. The actual iterative process is implemented in the Latent space (that is, implicit features or potential features. Latent features can use a small amount of data to represent a certain type of information). Before the Latent feature map is generated, the model first generates a completely random Latent feature map in the Latent space, and then the noise predictor UNet starts working to remove the predicted noise from the Latent feature map. It is not actually a simple subtraction to remove noise, but a certain sampling calculation rule in the mathematical sense. With the continuous repetition of this step, we finally get a clear Latent feature map, which is actually a Latent feature map that incorporates text information.

[0010] The above describes the process of generating images from text. In the process of generating images from images, we first need to specify a reference image, and then add noise to this reference image after passing it through the image encoder to obtain the initial image information tensor. In fact, it replaces the random image information tensor mentioned above as another input of the Image Information Creator, such as Figure 3 shown.

[0011] Among them, the network structure of UNet is as follows Figure 4For the sake of convenience, the following description uses b×c×h×w to describe the size of the feature map, where b represents the batch size, i.e., the number of images processed at a time; c represents the number of channels, which is 3 for images; h represents the height of the feature map; and w represents the width of the feature map. The network structure diagram is drawn with an input image of size 2x3x512x512, where b=2, c=3, h=512, and w=512, to show the size of the output feature map of each layer. After being downsampled 8 times by Image Encoder, the initial image information tensor (Inial image informationtensor), after superimposing noise through Scheduler, this is used as the initial value of Latent feature, that is, implicit feature or potential feature. The subsequent Latent feature is iterated repeatedly through UNet and Scheduler. Each iteration obtains prediction noise (Prediction) through UNet network, and the Latent feature removes the prediction noise through Scheduler to obtain a new Latent feature; at this time, the noise of the Latent feature is removed or the noise level of the Latent feature is reduced; the new Latent feature is superimposed with noise again through Scheduler to obtain the input Latent feature map of the next iteration, until the prompt, that is, the text prompt information, is completely integrated into the Latent feature, and a feature map containing text information is obtained, that is Figure 3 The processed image information tensor shown.

[0012] Explanation of the prior art:

[0013] 1. Continuously superimposing noise is also to continuously inject text information into the Latent feature map. 2. The above-mentioned Latent features remove noise through Scheduler. In fact, Scheduler is a kind of mathematical calculation formula or process. Different Schedulers have different calculation processes. How Scheduler superimposes noise, and then the calculation process is reversed to remove noise. 3. Take an input image of size 2x3x768x512 (b=2, c=3, h=512, w=512), and after Image Encoder, downsample 8 times to obtain an initial image information tensor (Inial imageinformation tensor) of size 2x4x96x64.

[0014] The detailed UNet network structure of the existing technology is as follows Figure 5As shown, note that T1 to T16 in the figure represent each Transformer2DModel structure respectively. At the same time, the implicit feature maps (featuremaps) of the down-sampling outputs such as OutDown64x64A, OutDown64x64B, OutDown32x32A, OutDown32x32B, and the implicit feature maps of the up-sampling outputs such as OutUp64x64A, OutUp64x64B, OutUp32x32A, OutUp32x32B are marked in the figure for the convenience of subsequent explanation.

[0015] When the above-mentioned prior art image-to-image process is applied to the original video to convert it into smooth videos of various styles, such as comic-style videos, the following problems are bound to occur:

[0016] If every frame of the video is processed, it will inevitably lead to a sharp increase in video memory and serious time consumption. According to the current common NVIDIA V100 processing, each frame is processed in 1.5-2 seconds. A 10s video has 20 frames per second, which takes 6-7 minutes, pushing up the deployment cost of the server; due to the process of adding random noise, the video generation result of the continuous frame rate will be seriously discontinuous, resulting in incoherent final video results and serious video flickering problems; given that img2img takes up a lot of memory and is very time-consuming when processing videos, if it is deployed on a personal PC or mobile terminal, not a server, the time consumption will increase sharply, which may lead to serious memory shortage; Summary of the invention

[0017] To solve at least one of the above problems, embodiments of the present invention provide a video style transfer method, device and program product based on a stable diffusion network, so as to reduce the performance requirements of the device while completing the video style transfer, improve the conversion speed, and ensure the conversion quality of the image.

[0018] In order to achieve the above-mentioned purpose, on the one hand, a video style conversion method based on a stable diffusion network is provided, wherein the stable diffusion network includes a text encoder and an image information creator component, the image information creator component includes a Unet network and a scheduler, and the stable diffusion network used is an enhanced stable diffusion network, and the method includes the following steps: receiving a video of a first video style; according to text indication information of video style conversion, using the enhanced stable diffusion network to convert the video of the first video style into a video of a second video style; wherein the enhanced stable diffusion network is obtained by the following steps: S1: obtaining a lightweight Unet network by iterative training using a distillation method, wherein the lightweight Unet network has a reduced number of network channels and a reduced number of cycles; S2: enhancing the obtained lightweight UNet network to obtain an enhanced lightweight UNet network structure, wherein the enhancement includes: introducing at least one noise information restoration module, wherein the noise information restoration module is used to receive the feature map output by the lightweight UNet network, and to restore the feature map output by the lightweight UNet network; The received feature map is subjected to noise information restoration; the noise information restoration module includes a feature combination learning module, and the feature combination learning module includes: a coefficient branch, an integral branch and a differential branch; the integral branch is implemented by global average pooling; the differential branch is implemented by convolution; a semantic segmentation map module is introduced, and the semantic segmentation map module is used to obtain the semantic segmentation map of the first video style video based on the semantic segmentation method; at least one feature information enhancement module is introduced, including: a pooling layer, a feature information connection module, and a semantic segmentation fusion module; the pooling layer processes the semantic segmentation map output by the semantic segmentation map module; the feature information connection module receives the output of the selected conversion module in the UNet lightweight network and the output of the noise information restoration module to connect the information of different input features and output the corresponding feature map; the semantic segmentation fusion module is used to fuse the semantic segmentation information output by the pooling layer with the information output by the feature information connection module; the feature information connection module and the semantic segmentation fusion module respectively include the feature combination learning module.

[0019] Preferably, in the video style conversion method, using the enhanced stable diffusion network to convert the video of the first video style into the video of the second video style comprises: extracting N frames of video key frames of the first video style from the video of the first video style, each of the video key frames V j Token embedding TE with the corresponding text encoder output j and the processed image information tensor PIT output by the image information creator component j, j = 1 ... N; using the enhanced stable diffusion network to convert the extracted video key frame of the first video style into the video key frame of the second video style; for the adjacent video key frame V in the first video style video j and V j+1 Each V in the K non-key frames between j，k , where k = 1...K, using the non-key frame V j，k With the video key frame V j The difference between j,k , the video key frame V j The corresponding TE j and the PIT j To obtain the non-key frame V j，k The corresponding processed image information tensor PIT j，k , the PIT j，k Input the image decoder included in the stable diffusion network to obtain the non-key frame V j，k The corresponding video frame of the second video style.

[0020] Preferably, in the video style transfer method, in the feature combination learning module, the integral branch includes, in sequence, a 1x1 convolution layer + activation layer, a global average pooling layer, 2 fully connected layers and a weight reorganization fitting layer; the differential branch includes, in sequence, a 3x3 convolution layer + activation layer, a dilation coefficient = 2 dilated convolution layer + activation layer, a dilation coefficient = 4 dilated convolution layer + activation layer, a dilation coefficient = 8 dilated convolution layer + activation layer, a connection layer, and a 3x3 convolution layer + activation layer; the coefficient branch includes, in sequence, a 1x1 convolution layer + activation layer, and a scaling layer.

[0021] Preferably, in the video style transfer method, the weight reorganization fitting layer is used to receive the feature map PF of c channels output by the coefficient branch, the feature map IF of c channels output by the integral branch, and the feature map DF of c channels output by the differential branch, and output the feature map WF of c channels, wherein the PF, IF and DF of each channel respectively introduce c parameters for the WF of one channel, and each channel's WF corresponds to 3c parameters.

[0022] Preferably, in the video style conversion method, 3-5 feature information enhancement modules are introduced, each feature information enhancement module shares parameters, and each feature information enhancement module is connected to a feature combination learning module.

[0023] Preferably, in the video style transfer method, the noise information restoration module introduces two feature combination learning modules in the deepest feature layer and one feature combination learning module in the second deepest layer.

[0024] Preferably, the video style transfer method, wherein the difference f is obtained by a difference analysis network structure j,k The difference analysis network structure includes: 2 encoders, which are used to extract the non-key frame V j，k With the video key frame V j feature map; multiple feature combination learning modules; multiple convolution blocks; multiple convolution downsampling blocks + activation layers; 1 feature map nearest neighbor upsampling module; multiple connection layers.

[0025] Preferably, the video style conversion method, wherein a predetermined conversion module is used to obtain the non-key frame V j，k The corresponding processed image information tensor PIT j，k , wherein the predetermined transformation module includes: 1 feature combination learning module; 2 feature information fusion modules; and 1 convolution module; wherein each feature information fusion block includes in sequence: a connection layer, the feature combination learning module, a convolution layer and a Transformer2DModel module.

[0026] On the other hand, an electronic device is provided, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of any of the video style conversion methods described above.

[0027] On the other hand, a computer program product is provided, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of any of the video style conversion methods described above are implemented.

[0028] The above technical solution has the following technical effects:

[0029] The technical solution of the embodiment of the present invention, by using an improved lightweight UNet network structure and introducing relevant enhancement schemes, accelerates the generation capability of the lightweight UNet network structure, reduces the number of iterations, reduces the performance requirements of the device while completing the video style conversion, improves the conversion speed, and ensures the image conversion quality; makes the localized deployment and processing of SD on the PC or mobile terminal easier, and reduces the performance requirements of the device; in a further embodiment, based on the recognition that the non-key frames between the key frames have the same or similar image information as the key frames, the non-key frames between the key frames are not processed by the img2img process, but the processed image information tensor of the key frames, namely the feature PIT, is used jThe difference between the non-key frame and the key frame is used to obtain the image information tensor of the non-key frame, namely the feature PIT j,k , which ensures the continuity of style conversion between consecutive frames in the generated image, and does not cause flickering between consecutive frames; further improves the image conversion speed, and the measured time spent in the preferred embodiment is 5% to 10% of the original time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the Stable Diffusion network Chinese raw image of the prior art;

[0031] Figure 2 This is a diagram of the working process of Scheduler step-by-step denoising in the Stable Diffusion network of the prior art;

[0032] Figure 3 It is a flowchart of the image generation process in the Stable Diffusion network of the prior art;

[0033] Figure 4 A brief network structure diagram of the original UNet in the Stable Diffusion network used in the prior art;

[0034] Figure 5 The detailed network structure diagram of the original UNet in the Stable Diffusion network used in the prior art;

[0035] Figure 6 A diagram of a UNet light network and its enhanced structure in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0036] Figure 7 A diagram of a UNet light network and its enhanced structure in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0037] Figure 8 A structural diagram of EnhanceBlock2D in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0038] Fig. 9 It is a structural diagram of ConcatBlock2D in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0039] Fig.10 2 is a structural diagram of MaskBlock2D in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0040] Fig.11A structural diagram of an N2I module in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0041] Fig.12 It is a structural diagram of PIDBlock2D in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0042] Fig.13 A structural diagram of an inter-frame difference network in a video style transfer method based on a stable diffusion network according to an embodiment of the present invention;

[0043] Fig.14 The main structure diagram of the WarpNet network in the video style transfer method based on the stable diffusion network according to an embodiment of the present invention, wherein the Encoder can use different structures;

[0044] Fig.15 A conventional pure convolutional Encoder structure in the WarpNet network main structure in the video style transfer method based on a stable diffusion network in an embodiment of the present invention;

[0045] Fig.16 Detailed structural diagram of the transfer module TransferBlock2D in the video style transfer method based on the stable diffusion network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0047] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0048] Embodiment 1:

[0049] In a video style transfer method based on a stable diffusion network according to an embodiment of the present invention, the stable diffusion network includes a text encoder and an image information creator component, the image information creator component includes a Unet network and a scheduler, wherein the stable diffusion network used is an enhanced stable diffusion network, and the method includes the following steps:

[0050] receiving a video of a first video style;

[0051] According to the textual indication information of the video style transfer, a video of a first video style is converted into a video of a second video style using an enhanced stable diffusion network;

[0052] The enhanced stable diffusion network is obtained by the following steps:

[0053] S1: A lightweight Unet network is obtained by iterative training using the distillation method. The lightweight Unet network has a reduced number of network channels and cycles.

[0054] S2: Enhance the obtained lightweight UNet network to obtain an enhanced lightweight UNet network structure, wherein the enhancement includes:

[0055] At least one noise information restoration module is introduced, and the noise information restoration module is used to receive the feature map output by the lightweight UNet network and perform noise information restoration on the received feature map; the noise information restoration module includes a feature combination learning module, and the feature combination learning module includes: a coefficient branch, an integral branch and a differential branch; the integral branch is implemented by global average pooling; the differential branch is implemented by convolution;

[0056] A semantic segmentation map module is introduced, and the semantic segmentation map module is used to obtain a semantic segmentation map of the first video style video based on a semantic segmentation method;

[0057] At least one feature information enhancement module is introduced, including: a pooling layer, a feature information connection module, and a semantic segmentation fusion module; the pooling layer processes the semantic segmentation map output by the semantic segmentation map module; the feature information connection module receives the output of the selected conversion module and the output of the noise information restoration module in the UNet lightweight network to connect the information of different input features and output the corresponding feature map; the semantic segmentation fusion module is used to fuse the semantic segmentation information output by the pooling layer with the information output by the feature information connection module; the feature information connection module and the semantic segmentation fusion module respectively include feature combination learning modules.

[0058] Embodiment 2:

[0059] For ease of understanding, here are some unified explanations: 1. The output of the intermediate graph of the network structure is generally called a feature map, where the feature map can be single-channel or multi-channel; Figure 1 Generally refers to the description in the implicit (latent) space (Latent Space); 2. The size of the feature map is described as b×c×h×w, where b represents the batch size, that is, the number of images processed at a time; c represents the number of channels, which is 3 for images; h represents the height of the feature map; and w represents the width of the feature map.

[0060] In order to solve the problems of time-consuming, high memory usage and video flickering in the existing img2img method for converting various styles of videos, the following solutions are proposed:

[0061] Introducing enhanced Stable Diffusion networks, including improvements to the used Unet network;

[0062] From the img2img process, the time consumption is mainly concentrated in the loop iteration of UNet and Scheduler. The time consumption of Scheduler is very low. The network structure of UNet is relatively large, and the time consumption of 50 loops accounts for 90%. Therefore, a lightweight UNet network structure and its training are designed, and the number of loops is reduced at the same time. Multiple enhancement results are generated at one time. In the diffusion process, it is essentially a process of continuously restoring image information from noise. To this end, the number of UNet network channels is reduced to 1 / 4 of the original number. Specifically, Figure 6 As shown, reducing the number of channels and adopting distillation to iterate training is a common speed-up solution. However, reducing the number of network channels means that the generation capacity of the network will be reduced. For this reason, this embodiment makes the following improvements on the network structure Unet:

[0063] (1) Introducing a feature information enhancement module EnhanceBlock2D; preferably, introducing 3-5 EnhanceBlock2Ds;

[0064] like Figure 6 As shown in the figure, four feature information enhancement modules are used, and three EnhanceBlock2Ds are shown in the figure. Each EnhanceBlock2D shares parameters, that is, the parameters of these EnhanceBlock2Ds are the same. Each EnhanceBlock2D is followed by a feature combination learning module PIDBlock2D to obtain Latent. PIDBlock2D has the ability to learn global and local information and can focus on the learning of useful information (refer to the analysis below for the specific reasons);

[0065] (2) Introducing the noise information restoration module (Noise2Image, abbreviated as N2I), all N2I modules share parameters;

[0066] (3) Introducing a semantic segmentation map. Semantic segmentation is a relatively mature technology and will not be described in detail. The semantic segmentation map can be obtained using existing technologies. The semantic segmentation map is a single-channel map.

[0067] (4) When actually performing forward reasoning, after obtaining Latent1, Latent2, Latent3, and Latent4, the four latent features are averaged; if there are only two groups of latents, only the two groups are averaged; the latent used for the next iteration (next latent) is averaged to further remove noise from the latent feature map and accelerate iteration.

[0068] Among them, the structure of EnhanceBlock2D is as follows Figure 8 As shown, the input includes a semantic segmentation feature map (featuremap, single channel), the 15th Transformer2DModel of the UNet lightweight structure (denoted as T15, see Figure 5 The result T15 Output outputted at the position corresponding to the structure; in other implementations, the output corresponding to T14 or T16 may also be used; preferably, T15 is used; and the N2I result N2IOutput includes ConcatBlock2D, MaskBlock2D and Conv2D in sequence.

[0069] SegPooling2D is a maximum pooling layer MaxPooling with kernel=8x8 and stride=8 (downsampled 8 times), 512x512 downsampled 8 times, and an output semantic feature map of size 64x64.

[0070] ConcatBlock2D, feature information connection module, such as Fig. 9 As shown in the figure, the information of different input features is connected to fuse the channel information, where T15 Output passes through a convolution module block Conv2D, N2IOutput passes through a convolution module Conv2D, the results of the two are connected through the connection layer Concat, and then pass through a feature combination learning module PIDBlock2D, and then connect to a convolution module Conv2D to output the result.

[0071] MaskBlock2D, semantic segmentation fusion module, such as Fig.10 As shown in the figure, the semantic segmentation information is fused with the feature map, so that the feature map has the semantic structure information of the original input map, which helps the latent feature to maintain the stability of the structure, avoids the uncontrollable factors in the denoising process, and thus avoids the flickering problem of the generated result. Among them, the input feature map, specifically, uses ConcatBlock2D output as the input feature map, multiplies the feature map and the semantic segmentation map, connects the result obtained by the convolution module, and connects the input feature map (ConcatBlock2D output) and passes through a feature combination learning module PIDBlock2D to output the result (MaskBlock2D output).

[0072] The structure of the noise information restoration module N2I is as follows Fig.11 As shown:

[0073] This structure is a common structure. The convolution block Conv2D represents conv3x3+PRelu (conv3x3 represents convolution with kernel=3x3 and stride=1, followed by activation layer PRelu), and the convolution downsampling block ConvStride2D represents convStride3x3+PRelu (convStride3x3 represents convolution with kernel=3x3 and stride=2, followed by activation layer PRelu, where stride=2 represents side convolution and side downsampling by 2 times, for example, 2x3x512x512 becomes 2x16x256x256 after ConvStride2D).

[0074] In convolution blocks or convolution downsampling blocks, the number of channels can change.

[0075] UpNearest2D means the nearest neighbor upsampling of the feature map, and the number of channels remains unchanged. Here it means upsampling by 2 times each time, for example, 2x16x256x256, the nearest neighbor upsampling is 2 times, becoming 2x16x512x512, keeping the number of channels unchanged.

[0076] The feature combination learning module PIDBlock2D is introduced to enhance the ability of restoring images from noise. Introducing 1-2 feature combination learning modules PIDBlock2D at the corresponding position of each deep feature in the N2I structure helps to better learn global and local features; preferably, 1-2 PIDBlock2Ds are sufficient. If the number is too large, learning feature information will cause redundant information and unnecessary time consumption. This embodiment introduces 2 feature combination learning modules PIDBlock2D in the deepest feature and 1 PIDBlock2D in the second deepest layer to improve the learning ability of the existing network structure in deep features; at the same time, the input starts to expand the 4-channel Latent featuremap to 80 channels through PIDBlock2D to expand its feature information.

[0077] like Fig.12As shown in the figure, PIDBlock2D is a feature combination learning module (proportional-integral-derivative) proposed in this paper to enhance learning ability, including coefficient (P) branch, integral (I) branch and differential (D) branch. The integral is realized by global average pooling (GlobalAvePooling), and the differential is realized by convolution. Conv3x3 represents convolution with kernel=3x3, stride=1, Dilate2Conv3x3 represents dilated convolution with kernel=3x3, stride=1, dilation=2, Dilate4Conv3x3 represents dilated convolution with kernel=3x3, stride=1, dilation=4, and kernel=3x3, stride=1. Dilate8Conv3x3 represents dilated convolution with dilation=8. Dilated convolution can connect local feature information in series and learn local features better. Convolution and dilated convolution are both convolution layers in neural networks, which are inherent conventional contents in the industry. Scale layer is a scaling layer in neural networks, which provides training and learning of proportional coefficients. PRelu is the activation layer in the neural network. Other activation layers can be used, but all activation layers need to be kept consistent. FC stands for fully connected layer. GlobalAvePooling stands for global average pooling layer. Fig.12 As shown in , if the input feature map of size 1×c0×h×w is given, the output size of each intermediate layer (the number of channels is c) is also given, and finally the output feature map of size 1×c1×h×w is output, where c0, c and c1 may not be equal.

[0078] PIDBlock2D introduces a weighted reorganization fitting layer (WeightCombineFit2D); PIDBlock2D is a whole that can reflect complete creativity. The features of the three branches are reorganized and fitted with weights to participate in training and learning. The coefficient (P) branch contains a 1x1 convolution layer + activation layer and a scaling layer Scale in sequence; the integral (I) branch contains a 1x1 convolution layer + activation layer, a global average pooling layer, and two fully connected layers in sequence; the differential (D) branch contains a 3x3 convolution layer + activation layer, a dilation coefficient dilation = 2 dilated convolution layer + activation layer, a dilated convolution layer + activation layer with dilation = 4, a dilated convolution layer + activation layer with dilation = 8, a connection layer concat, a 3x3 convolution layer + activation layer, and a dilated convolution layer that can fully learn local features. In addition to considering the pixels adjacent to itself, it also considers pixels with intervals of 2, 4, and 8. Finally, the weight recombination fitting layer WeightCombineFit2D is introduced to reorganize the feature information of the three branches. After fully learning various feature information, a 3x3 convolution layer + activation layer is connected.

[0079] PIDBlock2D can capture the important positions and semantic information in the input feature map, thereby achieving comprehensive attention and processing of the input feature map. Because the integral (I) branch in PIDBlock2D considers global features, the differential (D) branch considers local features, and the coefficient (P) branch considers the original input feature information, by training and learning the weight coefficients of each feature of the three branches, different parts of the input feature map (global features, local features, original features) can be assigned different weight coefficients, thereby adjusting the model's attention to different parts.

[0080] The specific weight parameters of the weight reorganization fitting layer WeightCombineFit2D are: the coefficient (P) branch outputs a feature map PF of size 1×c×h×w (including c feature maps of size h×w featuremaps), the integral (I) branch outputs a feature map IF of size 1×c (including c feature numbers), the differential (D) branch outputs a feature map DF of size 1×c×h×w (including c feature maps of size h×w featuremaps), and after WeightCombineFit2D outputs a feature map WF of size 1×c×h×w.

[0081] The channels of the feature map WF are denoted as {WF0, WF1…, WF c-1}, each channel WF j (j=0,1,…,c-1) is actually a single channel feature map of size h×w, for each channel WF j Construct the following learning parameters:

[0082] The channels of the feature map PF are denoted as {PF0, PF1…, PF c-1}, each channel PF k (k=0,1,…,c-1) is actually a single-channel feature map of size h×w, corresponding to the output WF j Introduce c parameters {pw j,0 ,pw j,1 …,pw j,c-1};

[0083] The channels of the feature map IF are denoted as {IF0, IF1…, IF c-1}, each channel IF k (k=0,1,…,c-1) is actually a number, corresponding to the output WF j Introduce c parameters {iw j,0 ,iw j,1 …,iw j,c-1};

[0084] The channels of the feature map DF are denoted as {DF0, DF1…, DF c-1}, each channel DF k (k=0,1,…,c-1) is actually a single-channel feature map of size h×w, corresponding to the output WF j Introduce c parameters {dw j,0 ,dw j,1 …,dw j,c-1};

[0085] The above features are summed up, that is

[0086] WF j =[pw j,0 ×PF j,0 +pw j,1 ×PF j,1 +…+pw j,c-1 ×PF j,c-1 ]

[0087] +[iw j,0 ×IF j,0 +iw j,1 ×IF j,1 +…+iw j,c-1 ×IF j,c-1 ]

[0088] +[dw j,0 ×DF j,0 +dw j,1 ×DF j,1 +…+dw j,c-1 ×DF j,c-1 ]

[0089] In short:

[0090]

[0091] Here, Σ represents summation.

[0092] Each output channel WF j Corresponding to 3c parameters, the output has c channels, and the required learning parameters are 3c in total 2 The specific weight parameters of the weight combination fitting layer WeightCombineFit2D include 3c 2 In training, let these 3C 2 Parameters are involved in learning to achieve the best combination effect.

[0093] The training of the network includes the following steps:

[0094] (1) First, use the SD (Stable Diffusion) original UNet network and its weights to train the UNet light network through the distillation method. At this time, the UNet light network has a certain generation ability, but the effect is not as good as the original UNet network. The generation ability needs to be further improved through subsequent enhancement blocks;

[0095] (2) Train SegNet through semantic segmentation data to achieve a certain level of accuracy. The training technology is relatively mature and will not be elaborated on here.

[0096] (3) Next, the UNet light network, the noise restoration module N2I and SegNet are trained simultaneously, where the learning rates of the UNet light network and SegNet are set to 25% to 35% of the learning rate of the noise restoration module N2I;

[0097] (4) After 4 iterations of SD original UNet and Scheduler, the 1st iteration, 2nd iteration, 3rd iteration, and 4th iteration can obtain 4 results that are diffused in sequence, which are used as the supervision target graphs of Latent1, Latent2, Latent3, and Latent4 respectively. After the 5th iteration, the original UNet and Scheduler can obtain the 5th diffusion result. The results of the 2nd, 3rd, 4th, and 5th diffusion are used as the supervision target graphs of the next training iterations of Latent1, Latent2, Latent3, and Latent4 respectively, and so on. The training is repeated in a cycle to achieve enhanced results.

[0098] Processing of non-keyframes

[0099] When converting videos, there is actually redundant information in the video. If each frame is processed according to the img2img process, the calculation time will inevitably increase, and there will still be some flickering between consecutive frames. In fact, extracting video key frames is already a relatively mature technology. It only needs to extract the video key frames (mature technology, no longer repeated), and perform complete img2img processing on the key frames, but the frames between the key frames do not need to be processed according to the img2img process. For this reason, this article proposes the following improved processing solution:

[0100] Processing process such as Fig.13 As shown in the figure, for the convenience of description, the processed image information tensor obtained by UNet and Scheduler is denoted as PIT, the Token embeddings obtained by Text Encoder is denoted as TE, and the key frames extracted by video key frame technology are denoted as V j There are N frames in total, each frame has its corresponding image information PIT j and the corresponding TE j , the result image Y is obtained through Image Decoder j .

[0101] (1) For a key frame V j , to the next key frame V j+1 There are K frames in total, and the image information of these K frames can be considered as the same as the key frame V j The image information is basically the same, and there is no need to go through the two steps of Text / Image Encoder and ImageInformation Creator every time. The key frame V j After the above two steps, we get the PIT containing image information. j , and the key frame V j and V j+1 The K frames contained in between are all related to the key frame V j have common or similar image information. (2) Key frame V j and V j+1 The K frames contained in between do not need to go through the Text / Image Encoder and Image Information Creator every time. For this purpose, a difference analysis network structure WarpNet is designed to identify V j and V j+1 A frame V between j,k and V j The difference F j,k , the PIT of the image information j Convert to the corresponding frame V through the TransferBlock2D modulej,k PIT of image information j,k , let PIT j,k After Image Decoder, the corresponding image result Y can be output j,k , further reducing the calculation time.

[0102] Among them, the detailed structure of WarpNet is as follows Fig.14 As shown, it contains 2 Encoders, and uses some common encoders, such as Fig.15 A pure convolutional encoder is shown, which will not be described here. At the same time, the 64x64 featuremaps in Encoder1 and Encoder2 are extracted and connected to the Decoder by concat to further learn the difference between the two frames. During the training process, Encoder1 and Encoder2 share parameters, that is, they have the same parameters. In addition, the PIDBlock2D mentioned above is introduced to improve its differentiated learning ability. The convolution downsampling block ConvStride2D represents convStride3x3+PRelu, convStride3x3 represents kernel=3x3, stride=2 convolution, followed by the activation layer PRelu, or other activation layers, where stride=2 means side convolution and side downsampling by 2 times. For example, the 1x80x64x64 Latent feature map in the figure becomes a 1x160x32x32 Latent feature map after ConvStride2D. The convolution block Conv2D represents conv3x3+PRelu (conv3x3 represents a convolution with kernel=3x3 and stride=1, followed by an activation layer PRelu, or other activation layers, but the activation layers followed by all convolution blocks and convolution downsampling blocks need to be consistent).

[0103] Experiments show that the feature map obtained after the Encoder in the WarpNet structure is similar to the deepest feature map. Fig.14 PIDBlock2D can capture the important positions and semantic information in the input feature map, so as to achieve comprehensive attention and processing of the input feature map. Because the integral (I) branch in PIDBlock2D considers the global features, the differential (D) branch considers the local features, and the coefficient (P) branch considers the original input feature information, the weight coefficients of each feature of the three branches can be learned through training. Different weight coefficients can be given to the global features, local features, and original features in the input feature map, thereby adjusting the degree of attention of the model to different parts.

[0104] The detailed structure of the transfer module TransferBlock2D is as follows Fig.16 As shown in the figure, it contains 1 PIDBlock2D to expand 4 channels to 80 channels and enrich the input feature information. The middle two feature information fusion modules are FeatureInformation Merge Block, the area surrounded by the dotted box, and 1 Conv2D. Each feature information fusion block contains Concat, PIDBlock2D, Conv2D and Transformer2DModel in sequence. Transformer2DModel is consistent with the Transformer2DModel in the UNet structure in the original SD, which can transform the text information TE j , key frame feature information PIT j,k And the difference information F between non-keyframes and keyframes j,k Effectively integrated, such as Fig.16 As shown in Figure 2, experiments show that two feature information fusion blocks are sufficient to learn the required feature information. Fusion modules can also be added, but increasing the number of fusion modules will increase the time consumption.

[0105] Through the above scheme, the continuity of style conversion between consecutive frames can be guaranteed, there will be no flickering between consecutive frames, and the conversion time can be greatly reduced.

[0106] Training of non-keyframe processing module

[0107] When training WarpNet and TransferBlock2D, the specific steps include:

[0108] (1) First train the UNet light network structure and related enhanced structures, fix the parameters of the network structure, and do not participate in this part of the training; (2) Train WarpNet and TransferBlock2D at the same time, and use PIT in the input j pass Figure 3 The original SD (parameters are fixed and not involved in training) is shown (iteration S times), and the result PIT obtained after WarpNet and TransferBlock2D training is shown (iteration S times). j,k , let PIT j,k go through Figure 3 The Image Decoder in the original SD shown can output the corresponding image result Y j,k , and V j,k and text message TE j go through Figure 3 The original SD process shown can get the corresponding result PIT j,k (Target) (same iteration S times) and Y j,k(Target), respectively as PIT j,k and Y j,k The specific loss function supervision refers to the original SD. The training scheme is relatively mature and will not be repeated here. (3) After step (2) training, the structure trained in step (1) is used to replace the relevant part of the SD process, and WarpNet and TransferBlock2D are fine-tuned. The fine-tuning learning rate is generally 10% to 20% of the learning rate in step (2) (which can be adjusted according to actual conditions). The purpose of this training step is to make WarpNet and TransferBlock2D better adapt to the UNet lightweight network structure and related enhanced structures, and obtain more stable results.

[0109] Embodiment three:

[0110] The present invention also provides an electronic device, which includes a processor and a memory, and a computer program stored in the memory and can be run on the processor, the processor includes one or more processing cores, the memory is connected to the processor via a bus, the memory is used to store program instructions, and the processor implements the steps in the above method embodiment of the first embodiment of the present invention when executing the computer program. Further, as an executable scheme, the electronic device can be a computer unit, and the computer unit can be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the above computer unit is only an example of a computer unit, and does not constitute a limitation on the computer unit, and may include more or less components than the above, or combine certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention. Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the computer unit, and various interfaces and lines are used to connect the various parts of the entire computer unit. The memory may be used to store the computer program and / or module, and the processor implements various functions of the computer unit by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0111] Embodiment 4:

[0112] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described above. Although the present invention is specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the present invention as defined in the appended claims, and all of these are within the scope of protection of the present invention.

Claims

1. A video style transfer method based on a stable diffusion network, wherein the stable diffusion network includes a text encoder and an image information creator component, wherein the image information creator component includes a Unet network and a scheduler, wherein: The stable diffusion network used is an enhanced stable diffusion network, and the method comprises the following steps: receiving a video of a first video style; According to the text indication information of the video style conversion, using the enhanced stable diffusion network to convert the video of the first video style into a video of the second video style; Wherein, the enhanced stable diffusion network is obtained by the following steps: S1: Obtaining a lightweight Unet network by iterative training using a distillation method, wherein the lightweight Unet network has a reduced number of network channels and a reduced number of cycles; S2: Enhance the obtained lightweight UNet network to obtain an enhanced lightweight UNet network structure, wherein the enhancement includes: At least one noise information restoration module is introduced, and the noise information restoration module is used to receive the feature map output by the lightweight UNet network and perform noise information restoration on the received feature map; the noise information restoration module includes a feature combination learning module, and the feature combination learning module includes: a coefficient branch, an integral branch and a differential branch; the integral branch is implemented by global average pooling; the differential branch is implemented by convolution; Introducing a semantic segmentation map module, the semantic segmentation map module is used to obtain a semantic segmentation map of the first video style video based on a semantic segmentation method; At least one feature information enhancement module is introduced, including: a pooling layer, a feature information connection module, and a semantic segmentation fusion module; the pooling layer processes the semantic segmentation map output by the semantic segmentation map module; the feature information connection module receives the output of the selected conversion module in the UNet lightweight network and the output of the noise information restoration module to connect the information of different input features and output the corresponding feature map; the semantic segmentation fusion module is used to fuse the semantic segmentation information output by the pooling layer with the information output by the feature information connection module; the feature information connection module and the semantic segmentation fusion module respectively include the feature combination learning module.

2. The video style transfer method according to claim 1, characterized in that Converting a video of the first video style into a video of the second video style using the enhanced stable diffusion network includes: Extract N frames of video key frames of the first video style from the video of the first video style, each of the video key frames V j Token embedding TE with the corresponding text encoder output j and the processed image information tensor PIT output by the image information creator component j , j = 1…N; Converting the extracted video key frames of the first video style into video key frames of the second video style using the enhanced stable diffusion network; For the adjacent video key frames V in the first video style video j and V j+1 Each V in the K non-key frames between j , k, where k = 1...K, using the non-key frame V j，k With the video key frame V j The difference between j，k , the video key frame V j The corresponding TE j and the PIT j To obtain the non-key frame V j，k The corresponding processed image information tensor PIT j，k , the PIT j，k Input the image decoder included in the stable diffusion network to obtain the non-key frame V j，k The corresponding video frame of the second video style.

3. The video style transfer method according to claim 1, characterized in that: In the feature combination learning module, the integration branch includes, in order, a 1x1 convolution layer + activation layer, a global average pooling layer, two fully connected layers, and a weight reorganization fitting layer; The differential branch includes, in order, a 3x3 convolution layer + activation layer, a dilated convolution layer with a dilation coefficient of 2 + activation layer, a dilated convolution layer with a dilation coefficient of 4 + activation layer, a dilated convolution layer with a dilation coefficient of 8 + activation layer, a connection layer, and a 3x3 convolution layer + activation layer; The coefficient branch includes a 1x1 convolution layer + an activation layer, and a scaling layer in sequence.

4. The video style transfer method according to claim 3, characterized in that: The weight reorganization fitting layer is used to receive the feature map PF of c channels output by the coefficient branch, the feature map IF of c channels output by the integral branch, and the feature map DF of c channels output by the differential branch, and output the feature map WF of c channels, wherein the PF, IF and DF of each channel respectively introduce c parameters for the WF of one channel, and each channel's WF corresponds to 3c parameters.

5. The video style transfer method according to claim 1, characterized in that: 3-5 feature information enhancement modules are introduced, each feature information enhancement module shares parameters, and each feature information enhancement module is connected to a feature combination learning module.

6. The video style transfer method according to claim 1, characterized in that: The noise information restoration module introduces two feature combination learning modules in the deepest feature layer and one feature combination learning module in the second deepest layer.

7. The video style transfer method according to claim 2, characterized in that: The difference F is obtained by differential analysis of the network structure. j，k , the difference analysis network structure includes: 2 encoders, respectively used to extract the non-key frame V j，k With the video key frame V j The feature map of A plurality of said feature combination learning modules; Multiple convolutional blocks; Multiple convolutional downsampling blocks + activation layers; 1 feature map nearest neighbor upsampling module; Multiple connection layers.

8. The video style transfer method according to claim 2, characterized in that: Use a predetermined conversion module to obtain the non-key frame V j，k The corresponding processed image information tensor PIT j，k , wherein the predetermined conversion module includes: 1 said feature combination learning module; 2 feature information fusion modules; and 1 convolution module; Among them, each of the feature information fusion blocks includes in sequence: a connection layer, the feature combination learning module, a convolutional layer and a Transformer2DModel module.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of the video style conversion method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the video style transfer method according to any one of claims 1 to 8 are implemented.

Images