Method and product for real-time video style conversion based on stable diffusion network

Abstract

The invention provides a method and a product for real-time video style conversion based on a stable diffusion network, the stable diffusion network comprises a text encoder, an image information creator assembly and an image decoder, the image information creator assembly comprises a Unet network and a scheduler, the image information creator component further comprises at least one first edge convolution module, a VGG network, an information fusion network and a fine tuning network, the information fusion network comprises a weight recombination fitting layer, the fine tuning network comprises at least one second edge convolution module, and the method comprises the following steps: receiving a first style video; processing the key frame and the non-key frame of the first-style video through a stable diffusion network to obtain a second-style video; performing fine tuning processing on the second-style video through a fine tuning network according to the first-style video to obtain the second-style video of which the continuity between adjacent frames meets a preset continuity condition; according to the technical scheme, the video conversion speed and quality can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and product for performing video style conversion based on a stable diffusion network. Background Art

[0002] Stable Diffusion is a latent diffusion model LDM for text to image txt2img, which was implemented and open-sourced by researchers from CompVis, Stability AI and LAION. Figure 1 The biggest advantage of SD is that it performs loop iterations in the latent space with relatively small feature map scale, which improves the computational efficiency of UNet.

[0003] Among them, S is the number of samples / iterations, which is generally 50-100 steps.

[0004] based on Figure 1 , describe the whole process of txt2image processing step by step:

[0005] (1) Input the prompt word "a photo of cartoon" and convert it into 77×768 TokenEmbeddings after being processed by the Text Encoder component. The Embeddings are then input into the Image Information Creator component.

[0006] (2) The random image information tensor is a 64×64 image representation randomly generated by a latent seed (Gaussian noise ~ N(0,1)). It represents a completely noisy image and serves as another initial input of the Image Information Creator component.

[0007] (3) Through the processing of the Image Information Creator component, a 64×64 processed image tensor containing image information is generated. This output contains the information of the image with the semantic information of the previous input prompt word. This process is called diffusion.

[0008] (4) The processed image tensor information generated in the previous step is processed by the ImageDecoder component to generate the final 512×512 image output related to the input Prompt word.

[0009] The denoising process of the scheduler is as follows: Figure 2 As shown below. The figure below is just a schematic diagram for easy understanding. The actual iterative process is implemented in the Latent space (that is, implicit features or potential features. Latent features can use a small amount of data to represent a certain type of information). Before the Latent feature map is generated, the model first generates a completely random Latent feature map in the Latent space, and then the noise predictor UNet starts working to remove the predicted noise from the Latent feature map. It is not actually a simple subtraction to remove noise, but a certain sampling calculation rule in the mathematical sense. With the continuous repetition of this step, we finally get a clear Latent feature map, which is actually a Latent feature map that incorporates text information.

[0010] The above describes the process of generating images from text. In the process of generating images from images, we first need to specify a reference image, and then add noise to this reference image after passing it through the image encoder to obtain the initial image information tensor. In fact, it replaces the random image information tensor mentioned above as another input of the Image Information Creator, such as Figure 3 shown.

[0011] Among them, the network structure of UNet is as follows Figure 4For the sake of convenience, the following description uses b×c×h×w to describe the size of the feature map, where b represents the batch size, i.e., the number of images processed at a time; c represents the number of channels, which is 3 for images; h represents the height of the feature map; and w represents the width of the feature map. The network structure diagram is drawn with an input image of size 2x3x512x512, where b=2, c=3, h=512, and w=512, to show the size of the output feature map of each layer. After being downsampled 8 times by Image Encoder, the initial image information tensor (Inial image informationtensor), after superimposing noise through Scheduler, this is used as the initial value of Latent feature, that is, implicit feature or potential feature. The subsequent Latent feature is iterated repeatedly through UNet and Scheduler. Each iteration obtains prediction noise (Prediction) through UNet network, and the Latent feature removes the prediction noise through Scheduler to obtain a new Latent feature; at this time, the noise of the Latent feature is removed or the noise level of the Latent feature is reduced; the new Latent feature is superimposed with noise again through Scheduler to obtain the input Latent feature map of the next iteration, until the prompt, that is, the text prompt information, is completely integrated into the Latent feature, and a feature map containing text information is obtained, that is Figure 3 The processed image information tensor shown.

[0012] Explanation of the prior art:

[0013] 1. Continuously superimposing noise is also to continuously inject text information into the Latent feature map.

[0014] 2. The latent features mentioned above are de-noised through a scheduler. In fact, a scheduler is a mathematical calculation formula or process. Different schedulers have different calculation processes. The scheduler superimposes noise, and then reverses the calculation process to remove noise.

[0015] 3. Take the input image of size 2x3x768x512 (b=2, c=3, h=512, w=512), downsample it 8 times after ImageEncoder to get the initial image information tensor (Inial imageinformation tensor) of size 2x4x96x64.

[0016] The detailed UNet network structure of the existing technology is as follows Figure 5 As shown, note that T1 to T16 in the figure represent each Transformer2DModel structure respectively. At the same time, the implicit feature maps (featuremaps) of the down-sampling outputs such as OutDown64x64A, OutDown64x64B, OutDown32x32A, OutDown32x32B, and the implicit feature maps of the up-sampling outputs such as OutUp64x64A, OutUp64x64B, OutUp32x32A, OutUp32x32B are marked in the figure for the convenience of subsequent explanation.

[0017] When the above-mentioned prior art image-to-image process is applied to the original video to convert it into smooth videos of various styles, such as comic-style videos, the following problems are bound to occur:

[0018] If every frame of the video is processed, it will inevitably lead to a sharp increase in video memory and serious time consumption. According to the current common NVIDIA V100 processing, each frame is processed in 1.5-2 seconds. A 10s video has 20 frames per second, which takes 6-7 minutes, pushing up the deployment cost of the server;

[0019] Due to the process of adding random noise, the video generation result with continuous frame rate will be seriously discontinuous, resulting in incoherent final video result and serious video flickering problem;

[0020] Since img2img takes up a lot of memory and takes a lot of time to process videos, if it is deployed on a personal PC or mobile terminal, not a server, the time consumption will increase dramatically, which may lead to serious memory shortage.

[0021] Diffusion models have revolutionized text-to-image generation with their superior quality and creativity. However, their multi-step sampling process is slow, often requiring dozens of inference steps to obtain satisfactory results. Distillation to increase sampling speed and reduce computational cost has not been very effective in practice, as distillation techniques, which reduce the number of parameters in the network structure, significantly reduce its original generative capabilities. The main problem with existing technologies stems from the suboptimal coupling of noise and images, which greatly hinders the distillation process. Summary of the invention

[0022] To solve at least one of the above problems, an embodiment of the present invention provides a method and product for video style conversion based on a stable diffusion network, so as to reduce the performance requirements of the device while completing the video style conversion, improve the conversion speed, and ensure the conversion quality of the image.

[0023] In order to achieve the above-mentioned object, on the one hand, a method for real-time video style transfer based on a stable diffusion network is provided, wherein the stable diffusion network includes a text encoder, an image information creator component and an image decoder, and the image information creator component includes a Unet network and a scheduler, wherein the image information creator component also includes at least one first edge convolution module, a VGG network, an information fusion network and a fine-tuning network, the information fusion network includes a weight reorganization fitting layer, and the fine-tuning network includes at least one second edge convolution module, and the method includes the following steps:

[0024] Receive the first style video;

[0025] Processing the key frames and non-key frames of the first style video through the stable diffusion network to obtain a second style video;

[0026] Fine-tune the second style video through the fine-tuning network according to the first style video to obtain a second style video whose continuity between adjacent frames satisfies a predetermined continuity condition;

[0027] The processing of the key frames of the first style video includes the following steps:

[0028] S1: extracting key frames of the first style video, inputting the key frames into the text encoder to obtain key frame feature maps, and inputting the key frame feature maps into the Unet network;

[0029] S2: Performing video style conversion on the key frame feature map through the Unet network to obtain a text information feature map;

[0030] S3: Inputting the intermediate feature map of the Unet network in the process of performing video style conversion on the key frame feature map into the first edge convolution module, enhancing the edge feature information of the intermediate feature map through the first edge convolution module, obtaining the intermediate feature map with enhanced edge feature information, and inputting the intermediate feature map with enhanced edge feature information into the information fusion network;

[0031] S4: Inputting the feature vector of the pre-trained VGG network into the information fusion network;

[0032] S5: fusing the feature vector and the edge feature information through the weight reorganization fitting layer of the information fusion network to enhance the intermediate feature map, thereby obtaining an edge information feature map;

[0033] S6: performing alhpa fusion on the text information feature map and the edge information feature map to obtain an alhpa feature map;

[0034] S7: using the alhpa feature map as the input of the Unet network, and executing steps S2-S6 again, and after executing steps S2-S6 for a predetermined number of times, obtaining a feature map that has been fused multiple times with respect to text information and edge information;

[0035] S8: inputting the feature map obtained in step S7 and subjected to multiple fusion into the image decoder, and using the image decoder to decode the feature map and subjected to multiple fusion to obtain a key frame of a second style video;

[0036] The processing of the non-key frames of the first style video includes the following steps:

[0037] S9: extracting the intermediate feature map of the key frame input into the first edge convolution module when the key frame is last iterated in the UNet network;

[0038] Inputting a non-key frame of the first style video and an intermediate feature map input to the first edge convolution module at the last iteration into the information fusion network, and outputting a feature map corresponding to the non-key frame after processing by the information fusion network;

[0039] S10: inputting the feature map corresponding to the non-key frame into the image decoder, and using the image decoder to decode the feature map corresponding to the non-key frame to obtain a non-key frame of the second style video corresponding to the non-key frame of the first style video;

[0040] The fine-tuning of the second style video by the fine-tuning network according to the first style video includes:

[0041] Inputting the first style video and the second style video into the second edge convolution module, and converting through the second edge convolution module to obtain a first feature map corresponding to the first style video and a second feature map corresponding to the second style video;

[0042] The first feature map and the second feature map are fused and input into the fine-tuning network. After sampling and learning edge information between adjacent frames in the fine-tuning network, a second style video in which the continuity between adjacent frames meets a predetermined continuity condition is output.

[0043] Preferably, in the method for real-time video style conversion based on a stable diffusion network, the information fusion network comprises in sequence: a convolution block, a convolution upsampling block and the weight reorganization fitting layer; wherein the convolution upsampling block comprises: a nearest neighbor upsampling block and a convolution block.

[0044] Preferably, in the method for real-time video style conversion based on a stable diffusion network, in step S5, the intermediate feature map enhanced by fusing the feature vector and edge feature information through the weight reorganization fitting layer of the information fusion network includes:

[0045] The weight reorganization fitting layer receives the intermediate feature map EF enhanced by the output edge feature information of the first edge convolution module, the feature map LF output by the convolution block or the convolution upsampling block in the information fusion network, and the feature vector VM output by the VGG network, and outputs a feature map PF with edge feature information and semantic text information after calculation by the weight re-fitting layer;

[0046] The calculation of the weight refitting layer includes:

[0047] The channels of the intermediate feature map EF are denoted by c, which is the number of channels;

[0048] The channel of the feature map LF is denoted as

[0049] The channel of the feature vector VM is denoted as

[0050] The channel of the feature map PF is denoted as PF j , where j = 0, 1, …, c-1;

[0051] The PF j It is obtained by the following calculation formula:

[0052]

[0053] Among them, Σ represents the summation, the aw j,k , bw j,k , cw j,k , dw j,k ,ew j,k , fw j,k Parameters learned for training.

[0054] Preferably, in the method for real-time video style conversion based on a stable diffusion network, the fine-tuning network further comprises:

[0055] Multiple concat layers;

[0056] Multiple convolutional blocks;

[0057] Multiple convolutional downsampling blocks;

[0058] Multiple nearest neighbor upsampling blocks.

[0059] Preferably, in the method for real-time video style conversion based on a stable diffusion network, the first edge convolution module and the second edge convolution module respectively include:

[0060] 4 directional convolution modules;

[0061] Multiple concat layers;

[0062] Multiple convolution blocks + activation layers;

[0063] Multiple edge computing formula modules, wherein the processing of the feature graph by the edge computing formula modules includes:

[0064]

[0065] Among them, the feature maps output by the four directional convolution modules are FM X , FM Y , FM C1 , FM C2 , where FM X , FM Y , FM C1 , FM C2 The corresponding channels are represented by FM X,m , FM Y,m , FM C1,m , FM C2,m , where m = 0, 1, 2, ..., c-1, the output feature map corresponding to the edge computing formula module is FM edge , the feature map FM edge Each channel is represented by FM edge,m .

[0066] On the other hand, an electronic device is provided, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of any of the above-described methods for performing video style conversion based on a stable diffusion network.

[0067] On the other hand, a computer-readable storage medium is provided, characterized in that at least one program is stored in the storage medium, and the at least one program is executed by a processor to implement the method for real-time video style transfer based on a stable diffusion network as described in any one of claims 1 to 5.

[0068] On the other hand, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method for performing video style transfer based on a stable diffusion network as described above are implemented.

[0069] The above technical solution has the following technical effects: the technical solution of the embodiment of the present invention provides an enhanced stable diffusion network by introducing the information structure network structure InfoMergeNet and the VGG network, reduces the number of iterations of the original SD network, and improves the image processing speed. For example, in the preferred embodiment, the actual measured time spent is 10% to 20% of the original time, which greatly reduces the memory consumption; and the technical solution of the embodiment of the present invention does not need to rely on the distillation technology during training;

[0070] In a further embodiment, for redundant information of the video, through the key frame technology, the non-key frames between the key frames only need to execute the information fusion network structure InfoMergeNet once, and then decoded by the image decoder (Image Decoder), so that continuous results can be obtained, avoiding repeated processing between adjacent frames, which can further shorten the processing time, reduce memory, and reduce flicker problems;

[0071] In a further embodiment, based on the above processing, an edge refinement network EdgeRefineNet is introduced between adjacent frames to learn the difference information between adjacent frames, so that the adjacent frames maintain better consistency and avoid the flickering problem between adjacent frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a flow chart of the Stable Diffusion network Chinese raw image of the prior art;

[0073] Figure 2 This is a diagram of the working process of Scheduler step-by-step denoising in the Stable Diffusion network of the prior art;

[0074] Figure 3 It is a flowchart of the image generation process in the Stable Diffusion network of the prior art;

[0075] Figure 4 A brief network structure diagram of the original UNet in the Stable Diffusion network used in the prior art;

[0076] Figure 5 The detailed network structure diagram of the original UNet in the Stable Diffusion network used in the prior art;

[0077] Figure 6 It is a structural diagram of the InfoMergeNet information fusion network in the method for real-time video style transfer based on a stable diffusion network according to an embodiment of the present invention;

[0078] Figure 7It is a structural diagram of the EdgeConv2D edge convolution module in the method for real-time video style conversion based on a stable diffusion network according to an embodiment of the present invention;

[0079] Figure 8 It is a structural diagram of a DirectConv directional convolution module in a method for real-time video style conversion based on a stable diffusion network according to an embodiment of the present invention;

[0080] Fig. 9 This is a structural diagram of a weight reorganization fitting layer PolyFit2D in a method for real-time video style conversion based on a stable diffusion network according to an embodiment of the present invention;

[0081] Fig.10 It is a flowchart of processing non-key frames in a method for real-time video style conversion based on a stable diffusion network according to an embodiment of the present invention;

[0082] Fig.11 This is a structural diagram of a fine-tuning network in a method for real-time video style transfer based on a stable diffusion network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0083] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0084] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0085] Embodiment 1:

[0086] In a method for real-time video style conversion based on a stable diffusion network according to an embodiment of the present invention, the stable diffusion network includes a text encoder, an image information creator component and an image decoder, the image information creator component includes a Unet network and a scheduler, wherein the image information creator component also includes at least one first edge convolution module, a VGG network, an information fusion network and a fine-tuning network, the information fusion network includes a weight reorganization fitting layer, and the fine-tuning network includes at least one second edge convolution module, and the method includes the following steps:

[0087] Receive the first style video;

[0088] Processing the key frames and non-key frames of the first style video through a stable diffusion network to obtain a second style video;

[0089] Fine-tune the second style video through a fine-tuning network according to the first style video to obtain a second style video in which continuity between adjacent frames satisfies a predetermined continuity condition;

[0090] The processing of the key frames of the first style video includes the following steps:

[0091] S1: extract the key frame of the first style video, input the key frame into the text encoder, obtain the key frame feature map, and input the key frame feature map into the Unet network;

[0092] S2: Perform video style conversion on the key frame feature map through the Unet network to obtain the text information feature map;

[0093] S3: Inputting the intermediate feature map of the Unet network in the process of video style conversion of the key frame feature map into the first edge convolution module, enhancing the edge feature information of the intermediate feature map through the first edge convolution module, obtaining the intermediate feature map with enhanced edge feature information, and inputting the intermediate feature map with enhanced edge feature information into the information fusion network;

[0094] S4: Input the feature vector of the pre-trained VGG network into the information fusion network;

[0095] S5: The feature vector and edge feature information are fused through the weight reorganization fitting layer of the information fusion network to enhance the intermediate feature map, thereby obtaining the edge information feature map;

[0096] S6: Perform alhpa fusion on the text information feature map and the edge information feature map to obtain an alhpa feature map;

[0097] S7: using the alhpa feature map as the input of the Unet network, and executing steps S2-S6 again. After executing steps S2-S6 for a predetermined number of times, a feature map that has been fused multiple times with respect to text information and edge information is obtained;

[0098] S8: inputting the feature map obtained in step S7 and subjected to multiple fusion into an image decoder, and using the image decoder to decode the feature map and subjected to multiple fusion to obtain a key frame of the second style video;

[0099] The processing of the non-key frames of the first style video includes the following steps:

[0100] S9: Extract the intermediate feature map of the edge convolution module when the key frame is input into the last iteration of the UNet network;

[0101] The non-key frame of the first style video and the intermediate feature map input to the edge convolution module at the last iteration are input into the information fusion network, and the feature map corresponding to the non-key frame is output after being processed by the information fusion network;

[0102] S10: inputting a feature map corresponding to the non-key frame into an image decoder, and using the image decoder to decode the feature map corresponding to the non-key frame to obtain a non-key frame of the second style video corresponding to the non-key frame of the first style video;

[0103] The fine-tuning of the second style video through the fine-tuning network according to the first style video includes:

[0104] Inputting the first style video and the second style video into the second edge convolution module, and converting them through the second edge convolution module to obtain a first feature map corresponding to the first style video and a second feature map corresponding to the second style video;

[0105] The first feature map and the second feature map are fused and input into a fine-tuning network. After sampling and learning edge information between adjacent frames in the fine-tuning network, a second style video in which the continuity between adjacent frames meets a predetermined continuity condition is output.

[0106] Embodiment 2:

[0107] In this embodiment, for ease of understanding, a unified description is first given:

[0108] 1. The output of the intermediate graph of the network structure is generally called a feature map. The feature map can be single-channel or multi-channel; Figure 1 Generally refers to the description in the latent space.

[0109] 2. In this embodiment, the size of the feature map is described by b×c×h×w, where b represents the batch size, that is, the number of images processed at a time or the number of images processed in parallel; c represents the number of channels, where for images, the number of channels is 3 channels; h represents the height of the feature map, and w represents the width of the feature map.

[0110] 3. The convolution block Conv2D represents conv3x3+PRelu, where conv3x3 represents a convolution with kernel=3x3 and stride=1, followed by an activation layer PRelu; the convolution downsampling block ConvStride2D represents convStride3x3+PRelu, where convStride3x3 represents a convolution with kernel=3x3 and stride=2, followed by an activation layer PRelu, where stride=2 represents side convolution and side downsampling by 2 times, such as 2x3x512x512, which becomes 2x16x256x256 after ConvStride2D.

[0111] In a convolution block or a convolution downsampling block, the number of channels can vary. The result obtained by the convolution block is generally called a feature map.

[0112] 4. FC represents the fully connected layer + activation function PRelu. Other activation functions can be selected but must remain consistent. The result is called a feature vector. The size of the feature vector is generally represented by b×c. b represents the batch size, that is, the number of images processed at a time or the number of parallel processing; c represents the vector size, which can also be called the number of channels for convenience. For convenience, the feature vector is also called a feature map of size b×c×1×1.

[0113] 5. UpConv2D is mainly composed of UpNearest2D+Conv2D. UpNearest2D means the nearest neighbor upsampling of the feature map, and the number of channels remains unchanged. Here it means upsampling by 2 times each time, such as 2x320x32x32, the nearest neighbor upsampling is 2 times, becoming 2x320x64x64, keeping the number of channels unchanged.

[0114] 6. The convolution downsampling block ConvStride2D represents convStride3x3+Prelu; wherein convStride3x3 represents a convolution with kernel=3x3 and stride=2, followed by an activation layer PRelu, where stride=2, representing a side convolution and side downsampling by a factor of 2, such as 2x3x512x512, which becomes 2x16x256x256 after ConvStride2D.

[0115] To solve the problem that the existing img2img method is time-consuming and memory-intensive for video conversion of various styles, the following solution is proposed:

[0116] From the img2img process, the time consumption is mainly concentrated in the loop iteration of UNet and Scheduler. The time consumption of Scheduler is very low. The network structure of UNet is relatively large. The time consumption of 50 cycles accounts for 90%. For this reason, a lightweight UNet network structure and its training are designed, and the number of cycles is reduced at the same time, and multiple enhancement results are generated at one time. In the diffusion process, it is essentially a process of continuously restoring image information from noise. For this reason, a solution is designed to reduce the number of cycles, which can naturally reduce the time consumption and shorten the number of iterations, so that noise restoration can be completed in 6-8 times. The existing structure generally takes 30-50 iterations to achieve a better generation effect. Doing so means that the speed of noise restoration needs to be accelerated. For this reason, one embodiment of the present invention introduces another information fusion network structure Information Merge Net, abbreviated as InfoMergeNet, based on the existing network structure. Specifically, Figure 6 shown.

[0117] 1. Extract the intermediate features output by each Transformer2DModel of the Unet network in the original SD structure and input them into the InfoMergeNet structure through the edge convolution module (EdgeConv2D), such as Figure 6 shown.

[0118] 2. Introduce the trained VGG network. The VGG network finally has three fully connected layers FC-4096, FC-4096, and FC-1000, which are recorded as VGG-FC1, VGG-FC2, and VGG-FC3 respectively. Select the second fully connected layer VGG-FC2 as the input of the InfoMergeNet structure. It is divided into two branches. One branch connects the fully connected layer to obtain a feature vector of size 2x(1280x64), which is transformed into a feature map of size 2x1280x8x8 through Reshape2D; the other branch connects 8 fully connected layers FC in sequence, and the feature vector obtained by each fully connected layer is integrated into the InfoMergeNet structure through PolyFit2D.

[0119] Among them, Reshape2D is a common operation in neural network structure and also a conventional professional term. It is used to change the distribution of data structure, generally changing the last three dimensions in bxcxhxw, i.e. the three dimensions corresponding to c, h, and w; for example, transforming a feature vector of 2x81920 (81920=1280x64) into a feature map of size 2x1280x8x8 (81920=1280x8x8); or transforming a feature map of size 2x1280x8x8 (81920=1280x8x8) into a feature map of size 2x1280x64x1, or a feature vector of 2x81920.

[0120] 3. The information fusion network structure consists of Conv2D, UpConv2D and PolyFit2D, and finally outputs the implicit feature InfoLatent, i.e., the edge information feature map, which contains semantic information and strengthens edge information. InfoLatent can generate the final image through the Image Decoder in the SD structure; that is, the InfoLatent obtained here and the Latent obtained by the UNet+Scheduler loop iteration in the original SD structure can both generate the final image through the Image Decoder. InfoLatent integrates the intermediate features in the UNet structure and the semantic information features of the VGG structure, and the generated image has the ability to maintain the stability of edge information.

[0121] In the UNet+Scheduler loop iteration, in each iteration, the InfoLatent obtained by the information fusion network structure and the Latent obtained by the UNet+Scheduler loop iteration can be simply fused with alhpa, InfoLatent×alpha+Latent×(1.0-alpha), and alpha=0.5 can be taken to obtain the alpha feature map and use it as input to participate in the iteration again. This can accelerate its generation capability and reduce the original 30-50 iterations to less than 10 times, so as to have basically consistent generation effects. At the same time, it has the ability to maintain edge stability in video style generation to avoid flickering.

[0122] Figure 7 FIG. 1 is a structural diagram of an EdgeConv2D edge convolution module in a method for performing video style conversion based on a stable diffusion network according to an embodiment of the present invention. Figure 7As shown in the figure, by strengthening the edge information of the feature map and combining it with the original features, the feature information is further strengthened. Due to the edge information learned during the training process, the contour of the final generated image is stable, reducing the problem of video flickering. The input feature map is convolved Conv1x1+PRelu to obtain the feature map, and the number of channels is changed; it can be considered that the information of the original feature map is maintained, and the number of channels can be changed by 1x1 convolution; then the horizontal feature map, the vertical feature map, the oblique C1 feature map, and the oblique C2 feature map are obtained through the directional convolution modules DirecConvX, DirecConvY, DirecConvC1, and DirecConvC2, respectively. The 4 directional feature maps are calculated by the edge calculation formula EdgeFormula2D to obtain the edge feature map, and then concat the feature map obtained by the convolution Conv1x1+PRelu to obtain the combined feature map, and the final output result is obtained after the convolution Conv3x3+PRelu. If the input feature map is of size, the output size of each intermediate layer (the number of channels is c) is also given, and the final output feature map is of size b×c1×h×w. Figure 7 , where the number of channels c0, c and c1 may not be equal.

[0123] Figure 8 FIG. 1 is a structural diagram of a DirectConv directional convolution module in a method for performing video style conversion based on a stable diffusion network according to an embodiment of the present invention. Figure 8 As shown in the figure, the directional convolution (DirectConv) includes 4 directions, among which DirecConvX is actually a 1x5 convolution, and DirecConvY is actually a 5x1 convolution; and the unidirectional convolution is common in the current convolution structure; the two convolution directions DirecConvC1 and DirecConvC2 introduced in the embodiment of the present invention are just located at the 45-degree direction. Specifically, the direction represented by the red number 1 in the figure is the 5 parameters to be learned during network training, and the gray blocks are only for convenience of display and do not actually exist; the number of channels of the input and output feature maps of the directional convolution is consistent, and each channel of the input feature map is convolved separately to obtain the corresponding output channel, that is, each channel has its own convolution kernel; wherein each convolution kernel has 5 parameters. For example, the input is b×c×h×w, where b only represents the number of parallel operations. During forward reasoning, if only one image is processed, b=1, and the mth (m=0,1,2,…,c-1)th channel is convolved separately to obtain the corresponding output channel. The convolution kernel corresponding to each channel is learned through training, and the parameters of the convolution kernel learned for each channel are different, which is the grouped convolution often mentioned in the convolution structure.

[0124] Furthermore, the meaning of the edge calculation formula EdgeFormula2D is that the feature maps in the four directions are FM X , FM Y , FM C1 , FM C2 , the channels of each feature map are represented as FM X,m , FM Y,m , FM C1,m , FM C2,m , where m = 0, 1, 2, ..., c-1, and the corresponding output feature map is FM edge , each channel of the output feature map is represented as FM edge,m , then the following calculation formula is obtained

[0125]

[0126] Above Represents square root. Through sufficient training and learning of the directional convolution in the above four directions, the edge information characteristics of the feature map can be learned, so that it can retain the ability of edge information, which helps to maintain the stability of the edge contour of the final video style conversion result and reduce the flicker problem between consecutive frames. The above convolution kernel can use 7 parameters, 9 parameters, 11 parameters, and so on, but it will increase the time consumption. Experiments have proved that the convolution kernel of size 5 can fully learn the edge feature information.

[0127] Fig. 9 : is a structural diagram of a weight reorganization fitting layer PolyFit2D in a video style conversion method based on a stable diffusion network according to an embodiment of the present invention, such as Fig. 9 As shown in the figure, this module can enhance the ability of information integration and effectively integrate the text information contained in Transformer2DModel and the image semantic information contained in VGG. The weight reorganization fitting layer PolyFit2D has three inputs, namely the output feature map EF of the edge convolution module EdgeConv2D, the feature map LF output by the previous layer (Last Layer) of the network structure PolyFit2D, and the feature vector VM output by the fully connected layer. The output feature map is PF.

[0128] The channels of the output feature map PF are denoted as {PF0, PF1…, PF c-1}, each channel PF j (j=0,1,…,c-1) is actually a single channel feature map of size h×w, for each channel PF j Construct the following training learning parameters:

[0129] The channels of the feature map EF are denoted as {EF0, EF1…, EF c-1}, each channel EF k(k=0,1,…,c-1) is actually a single-channel feature map of size h×w.

[0130] The channels of the feature map LF are denoted as {LF0, LF1…, LF c-1}, each channel LF k (k=0,1,…,c-1) is actually a single-channel feature map of size h×w.

[0131] The channels of the feature vector VM are denoted as {VM0, VM1…, VM c-1}, each channel VM k (k=0,1,…,c-1) is actually a number.

[0132] PF j The calculation formula is:

[0133]

[0134] Among them, Σ represents the summation, aw j,k , bw j,k , cw j,k , dw j,k ,ew j,k , fw j,k are the parameters involved in training and learning.

[0135] Each output channel PF j Corresponding to 6c parameters, the output has c channels, and the required learning parameters are 6c in total 2 The specific weight parameters of the weight reorganization fitting layer PolyFit2D include 6c 2 In training, let these 6c 2 Parameters are involved in learning to achieve the best feature information fitting effect, so that the final output latent has text information and edge feature information. The quadratic term is used because the quadratic term information can enhance the edge effect, because the edge often has a higher gradient, and the quadratic term can enhance the gradient information.

[0136] During training, the specific steps are:

[0137] 1. The parameters in the consolidated SD structure do not participate in training, and only train and learn the information fusion network structure InfoMergeNet.

[0138] 2. During the initial training, InfoLatent obtained by InfoMergeNet and Latent obtained by UNet+Scheduler iteration are randomly alpha-pasted, so that InfoLatent also has the characteristics of text information and edge information.

[0139] 3. Next, the learning rate is adjusted to 10% to 20% of the learning rate in step 2, and InfoMergeNet is fine-tuned to further improve its ability to generate single image information features.

[0140] Processing of non-key frames:

[0141] When converting videos, there is actually redundant information in the video. If each frame is processed according to the img2img process, the calculation time will inevitably increase, and there will still be a certain amount of flicker between consecutive frames. In fact, extracting video key frames is already a relatively mature technology. We only need to extract video key frames (mature technology, no longer repeat), and perform complete img2img processing on the key frames. When the key frames are undergoing complete img2img processing, during the UNet+Scheduler loop iteration, InfoMergeNet still participates in the loop iteration, and the Latent obtained from the two is fused with alhpa, and alpha is 0.5, which can reduce the number of iterations from 30-50 times to less than 10 times. At the same time, the generated style result image and the input original image can maintain consistent edge information to avoid edge jumps and flickering. The edge information features are controlled by InfoMergeNet.

[0142] However, the frames between the key frames do not need to be processed according to the img2img process. For this reason, this article proposes the following improved processing solution.

[0143] Processing process such as Fig.10 As shown in the figure, for the convenience of description, the processed image information tensor obtained by UNet and Scheduler is denoted as PIT, the Token embeddings obtained by Text Encoder is denoted as TE, and the key frames extracted by video key frame technology are denoted as V j There are N frames in total, each frame has its corresponding image information PIT j and the corresponding TE j , the result image Y is obtained through Image Decoder j .

[0144] 1. For a key frame V j , to the next key frame V j+1 There are K frames in total, and the image information of these K frames can be considered as the same as the key frame V j The image information is basically the same, and there is no need to go through the two steps of Text / Image Encoder and ImageInformation Creator every time. The key frame V j After the above two steps, we get the PIT containing image information. j, and the key frame V j and V j+1 The K frames contained in between are all related to the key frame V j Have common or similar image information.

[0145] 2. Keyframe V j and V j+1 The K frames contained in between do not need to go through the Text / Image Encoder and Image Information Creator every time. j,k Input the information fusion network InfoMergeNet and extract the key frame V j In the last iteration of the UNet+Scheduler loop, the corresponding intermediate feature map in UNet is input into InfoMergeNet. Only one execution of InfoMergeNet is required to obtain the corresponding frame V j,k PIT of image information j,k , let PIT j,k After Image Decoder, the corresponding image result Y can be output j,k , further reducing the calculation time.

[0146] Image fine-tuning processing:

[0147] The results obtained by the original SD-img2img are uncontrollable, lacking style continuity between adjacent frames and having serious transition discontinuities.

[0148] For the result image Y obtained by the key frame j and through non-keyframe V j,k The resulting image Y j,k ,Through the information fusion network InfoMergeNet enhanced by edge information, the flicker problem of the video has been greatly reduced, and the continuity of adjacent frames of the video style transfer has been improved.

[0149] Furthermore, the following fine-tuning network structure EdgeRefineNet is used to improve the continuity between adjacent frames and completely avoid the flicker problem. For continuous video, each frame is denoted as V t , t represents the tth frame of the continuous video, and the two adjacent frames are V t and V t+1 , the output results Y are obtained through the above InfoMergeNet and Image Decoder respectively t and Y t+1 , these two results have good continuity. The structure of EdgeRefineNet is as follows Fig.11 shown.

[0150] The EdgeRefineNet structure mentioned above is a general structure. t and Y t After merging, we get the feature map through EdgeConv2D. The t+1 frame V t+1 and Y t+1 After merging, EdgeConv2D is used to obtain the feature map. The two sets of feature maps are merged again as the input of the network. After 4 downsampling, 4 upsampling are performed at the same time, and finally Y is realized. t+1 In the structure, the edge convolution module EdgeConv2D is introduced when downsampling ConvStirde2D, and EdgeConv2D is also introduced after upsampling UpNearest2D, which can fully learn the edge differences between adjacent frames, so as to obtain the fine-tuned next frame result Refine Y t+1 , Y t+1 and Y t It can maintain continuous consistency and completely avoid flickering problems. The number of parameters of the fine-tuning network is relatively small and the time consumption is relatively low.

[0151] Through the above scheme, the continuity of style conversion between consecutive frames can be guaranteed, flickering between consecutive frames will not occur, and the conversion time can be greatly reduced.

[0152] Embodiment three:

[0153] The present invention also provides an electronic device, which includes a processor and a memory, and a computer program stored in the memory and executable on the processor, wherein the processor includes one or more processing cores, the memory is connected to the processor via a bus, and the memory is used to store program instructions, and when the processor executes the computer program, the steps in the above-mentioned method embodiment of the first embodiment of the present invention are implemented.

[0154] Further, as an executable solution, the electronic device may be a computer unit, and the computer unit may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art may understand that the composition structure of the above-mentioned computer unit is only an example of a computer unit and does not constitute a limitation on the computer unit, and may include more or less components than the above-mentioned, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.

[0155] Further, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, and various interfaces and lines are used to connect the various parts of the entire computer unit.

[0156] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer unit by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0157] Embodiment 4:

[0158] The present invention also provides a computer-readable storage medium, wherein at least one program is stored in the storage medium, and the at least one program is executed by a processor to implement the method for real-time video style conversion based on a stable diffusion network as described above.

[0159] Embodiment five:

[0160] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described above are implemented.

[0161] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A method for real-time video style transfer based on a stable diffusion network, wherein the stable diffusion network includes a text encoder, an image information creator component and an image decoder, wherein the image information creator component includes a Unet network, a scheduler, It is characterized in that The image information creator component also includes at least one first edge convolution module, a VGG network, an information fusion network and a fine-tuning network, wherein the information fusion network includes a weight reorganization fitting layer, and the fine-tuning network includes at least one second edge convolution module. The method includes the following steps: Receive the first style video; Processing the key frames and non-key frames of the first style video through the stable diffusion network to obtain a second style video; Fine-tune the second style video through the fine-tuning network according to the first style video to obtain a second style video whose continuity between adjacent frames satisfies a predetermined continuity condition; The processing of the key frames of the first style video includes the following steps: S1: extracting key frames of the first style video, inputting the key frames into the text encoder to obtain key frame feature maps, and inputting the key frame feature maps into the Unet network; S2: Performing video style conversion on the key frame feature map through the Unet network to obtain a text information feature map; S3: Inputting the intermediate feature map of the Unet network in the process of performing video style conversion on the key frame feature map into the first edge convolution module, enhancing the edge feature information of the intermediate feature map through the first edge convolution module, obtaining the intermediate feature map with enhanced edge feature information, and inputting the intermediate feature map with enhanced edge feature information into the information fusion network; S4: Inputting the feature vector of the pre-trained VGG network into the information fusion network; S5: fusing the feature vector and the edge feature information through the weight reorganization fitting layer of the information fusion network to enhance the intermediate feature map, thereby obtaining an edge information feature map; S6: performing alhpa fusion on the text information feature map and the edge information feature map to obtain an alhpa feature map; S7: using the alhpa feature map as the input of the Unet network, and executing steps S2-S6 again, and after executing steps S2-S6 for a predetermined number of times, obtaining a feature map that has been fused multiple times with respect to text information and edge information; S8: inputting the feature map obtained in step S7 and subjected to multiple fusion into the image decoder, and using the image decoder to decode the feature map and subjected to multiple fusion to obtain a key frame of a second style video; The processing of the non-key frames of the first style video includes the following steps: S9: extracting the intermediate feature map of the key frame input into the first edge convolution module when the key frame is last iterated in the UNet network; Inputting a non-key frame of the first style video and an intermediate feature map input to the first edge convolution module at the last iteration into the information fusion network, and outputting a feature map corresponding to the non-key frame after processing by the information fusion network; S10: inputting the feature map corresponding to the non-key frame into the image decoder, and using the image decoder to decode the feature map corresponding to the non-key frame to obtain a non-key frame of the second style video corresponding to the non-key frame of the first style video; The fine-tuning of the second style video by the fine-tuning network according to the first style video includes: Inputting the first style video and the second style video into the second edge convolution module, and converting through the second edge convolution module to obtain a first feature map corresponding to the first style video and a second feature map corresponding to the second style video; The first feature map and the second feature map are fused and input into the fine-tuning network. After sampling and learning edge information between adjacent frames in the fine-tuning network, a second style video in which the continuity between adjacent frames meets a predetermined continuity condition is output.

2. The method for real-time video style transfer based on a stable diffusion network according to claim 1, It is characterized in that The information fusion network includes in sequence: a convolution block, a convolution upsampling block and the weight reorganization fitting layer; wherein the convolution upsampling block includes: a nearest neighbor upsampling block and a convolution block.

3. The method for real-time video style transfer based on a stable diffusion network according to claim 2, It is characterized in that In step S5, the intermediate feature map enhanced by fusing the feature vector and edge feature information through the weight reorganization fitting layer of the information fusion network includes: The weight reorganization fitting layer receives the intermediate feature map EF enhanced by the output edge feature information of the first edge convolution module, the feature map LF output by the convolution block or the convolution upsampling block in the information fusion network, and the feature vector VM output by the VGG network, and outputs a feature map PF with edge feature information and semantic text information after calculation by the weight re-fitting layer; The calculation of the weight refitting layer includes: The channel of the intermediate feature map EF is denoted as EF k , where k = 0, 1, ..., c-1, c is the number of channels; The channel of the feature map LF is denoted as LF k , where k = 0, 1, ..., c-1; The channel of the feature vector VM is denoted as VM k , k=0,1,…,c-1; The channel of the feature map PF is denoted as PF j , where j = 0, 1, …, c-1; The PF j It is obtained by the following calculation formula: Among them, Σ represents the summation, the aw j,k , bw j,k , cw j,k , dw j,k ,ew j,k , fw j,k Parameters learned for training.

4. The method for real-time video style transfer based on a stable diffusion network according to claim 1, It is characterized in that The fine-tuning network also includes: Multiple concat layers; Multiple convolutional blocks; Multiple convolutional downsampling blocks; Multiple nearest neighbor upsampling blocks.

5. The method for real-time video style transfer based on a stable diffusion network according to claim 4, It is characterized in that The first edge convolution module and the second edge convolution module respectively include: 4 directional convolution modules; Multiple concat layers; Multiple convolution blocks + activation layers; Multiple edge computing formula modules, wherein the processing of the feature graph by the edge computing formula modules includes: Among them, the feature maps output by the four directional convolution modules are FM X , FM Y , FM C1 , FM C2 , where FM X , FM Y , FM C1 , FM C2 The corresponding channels are represented by FM X,m , FM Y,m , FM C1,m , FM C2,m , where m = 0, 1, 2, ..., c-1, the output feature map corresponding to the edge computing formula module is FM edge , the feature map FM edge Each channel is represented by FM edge,m .

6. An electronic device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of the method for real-time video style conversion based on a stable diffusion network as described in any one of claims 1 to 5.

7. A computer-readable storage medium, It is characterized in that The storage medium stores at least one program, and the at least one program is executed by a processor to implement the method for real-time video style conversion based on a stable diffusion network as described in any one of claims 1 to 5.

8. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method for real-time video style conversion based on a stable diffusion network are implemented as described in any one of claims 1 to 5.

Images