Animation coloring method and device based on video generation model

Abstract

The present invention discloses an animation coloring method and device based on a video generation model, which relates to the technical field of automatic image coloring, and solves the technical problem that the current coloring method produces different results for each operation of the same line draft, resulting in poor continuity between the previous and next frames. The method includes extracting the edge line draft of the middle frame image of the video to be colored, and inputting the edge line draft into the ControlNet model to obtain the edge features of the middle frame image; inputting the first frame image and the last frame image of the video to be colored into the video generation model to obtain the image features of the first frame image and the last frame image, and combining the image features and the edge features to obtain edge condition information; performing feature fusion on the first frame image and the last frame image according to the edge condition information to obtain fusion features; and coloring the middle frame image through the fusion features. The present invention can enhance the continuity of the front and back frames of the video, and realize accurate coloring of the animation video.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic image coloring, and in particular to an animation coloring method and device based on a video generation model. Background Art

[0002] In animation production, each frame of the animation line drawing needs to be colored. Initially, it was done manually by artists. However, in just a few minutes of animation video, there are hundreds of lines that need to be colored. Manual coloring not only tests the artist's level, but is also very inefficient. With the rapid development of artificial intelligence, computer vision, image processing and other technologies have been widely used to achieve automatic animation coloring.

[0003] Traditional automatic coloring methods include: automatic coloring methods based on generative adversarial networks, but this method can only generate colors randomly. In actual industrial production, the characters and objects in the animation have set colors, and the colors will also change in brightness as the scene changes. Obviously, randomly generated colors cannot meet the requirements of industrial generation. There are also coloring methods based on image segmentation, but if the image line draft carries less information, it will lead to the inability to achieve accurate segmentation, thus affecting the accuracy of coloring. Therefore, most coloring methods based on generative models are used to overcome the above problems.

[0004] At present, commonly used generative models lack precise control and are therefore difficult to apply to industrial generation. For example, the ATTENTION-AWARE model (a deep learning model based on the attention mechanism), the ANIME LINE model (a specific model in the StableDiffusion model, mainly used for coloring line drawings), the DRAW model (Deep Recurrent Attentive Writer, an image generation model based on recurrent neural networks and autoencoders), and the COLORIZATION model (a technology for automatically coloring grayscale images). Although these models can colorize in a targeted manner through the provided reference frames, each run of the same line drawing will produce different results, resulting in poor continuity between the previous and next frames.

[0005] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0006] The current coloring method will produce different results for each coloring of the same line drawing, resulting in poor continuity between the previous and next frames. Summary of the invention

[0007] The purpose of the present invention is to provide an animation coloring method and device based on a video generation model, so as to solve the technical problem that the current coloring method in the prior art produces different results for the coloring of the same line draft each time it is run, resulting in poor continuity between the previous and next frames. The current coloring method produces different results for the coloring of the same line draft each time it is run, resulting in poor continuity between the previous and next frames. The various technical effects that can be produced by the preferred technical solution among the various technical solutions provided by the present invention are described in detail below.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention provides an animation coloring method based on a video generation model, comprising the following steps: extracting edge line drafts of an intermediate frame image of a video to be colored, and inputting the edge line drafts into a ControlNet model to obtain edge features of the intermediate frame image; inputting a first frame image and an end frame image of the video to be colored into a video generation model to obtain image features of the first frame image and the end frame image, and combining the image features and the edge features to obtain edge condition information; wherein the first frame image and the end frame image are both color images that have been colored; performing feature fusion on the first frame image and the end frame image according to the edge condition information to obtain fusion features; and coloring the intermediate frame image using the fusion features.

[0010] Optionally, the step of extracting edge line drafts of the intermediate frame images of the video to be colored, and inputting the edge line drafts into a ControlNet model to obtain edge features of the intermediate frame images comprises: extracting edge line drafts of the intermediate frame images of the video to be colored by an edge detection algorithm, inputting the edge line drafts into the ControlNet model, and obtaining output features c of each block of the ControlNet model; wherein the output features c of each block are edge features of the intermediate frame images.

[0011] Optionally, the first frame image and the last frame image of the video to be colored are input into a video generation model to obtain image features of the first frame image and the last frame image, and edge condition information is obtained by combining the image features and the edge features, including: inputting the first frame image and the last frame image of the video to be colored into a video generation model to obtain output features f of each block of the video generation model, and adding the output features f to the output features c to obtain the edge condition information f'.

[0012] Optionally, the feature fusion of the first frame image and the tail frame image is performed according to the edge condition information to obtain a fused feature, including: taking the edge condition information as the bias feature of the video generation model, and obtaining the first frame latent space feature Z1 of the first frame image and the tail frame latent space feature Z2 of the tail frame image through the video generation model; respectively dimensionally splicing the first frame latent space feature Z1 and the tail frame latent space feature Z2 with Gaussian noise, and then denoising to obtain a first frame denoising feature Z1' and a tail frame denoising feature Z2'; fusing the first frame denoising feature Z1' and the tail frame denoising feature Z2' to obtain a fused feature.

[0013] Optionally, in the step of fusing the first frame denoising feature Z1' and the last frame denoising feature Z2' to obtain a fused feature, a formula for obtaining the fused feature is:

[0014] ;

[0015] in, is the fusion feature; is a weight vector that increases linearly from 0 to 1, and the value of each increase of the weight vector is , N is the number of frames of the video to be colored that is denoised each time.

[0016] Optionally, after fusing the first frame denoising feature Z1' and the last frame denoising feature Z2' to obtain a fused feature, the method further includes: repeatedly performing the operation of adding noise and then denoising on the fused feature to obtain a final fused feature.

[0017] Optionally, coloring the intermediate frame image using the fusion feature includes: decoding the fusion feature through a decoder to obtain a colored video generated according to the first frame image, the last frame image and the intermediate frame image.

[0018] Optionally, coloring the intermediate frame images through the fusion features also includes: calculating all connected areas in the edge line draft of each of the intermediate frame images; wherein each of the connected areas corresponds to a colored area in a reference color draft; the reference color draft is obtained by fusing the features of the first frame image and the last frame image; and based on the first frame image and the last frame image, flatly coloring each connected area through a voting operation.

[0019] Optionally, based on the first frame image and the last frame image, each connected area is flatly colored through a voting operation, including: obtaining the color value of the reference color draft corresponding to each of the connected areas, recorded as RGB1; calculating the distance between each pixel point and all color values ​​in the RGB1 in the connected area, and assigning the color value closest to the pixel point to the pixel point; and flatly coloring the color value with the highest frequency of occurrence in the connected area to the entire connected area.

[0020] An animation coloring device based on a video generation model, comprising:

[0021] one or more processors;

[0022] A memory for storing one or more computer programs, and one or more processors for executing the one or more computer programs stored in the memory, so that the one or more processors perform an animation coloring method based on a video generation model as described in any one of the above

[0023] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects:

[0024] The present invention improves the sampling algorithm of the video generation model, takes the first frame and the last frame of the colored video as the coloring reference images, introduces the information of the second reference image, and jointly generates the color video, thereby strengthening the control over the video generation. Due to the image information of the first frame and the last frame, the required color can be accurately filled into the middle frame, the coherence of the front and rear frames of the video is strengthened, and the coloring work in industrial production can be solved by aligning with the industrial generation. The edge information of the line draft is also introduced into the video production model through the ControlNet control model, so as to further strengthen the intermediate control process of the video generation model and improve the coloring effect of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 is a flow chart of an animation coloring method based on a video generation model according to a first embodiment of the present invention;

[0027] Figure 2 This is a flowchart of step S3 of the animation coloring method based on the video generation model in the first embodiment of the present invention;

[0028] Figure 3This is a flowchart of step S4 of the animation coloring method based on the video generation model in the first embodiment of the present invention;

[0029] Figure 4 It is a flowchart of step S42 of the animation coloring method based on the video generation model in the first embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. that are consistent with some aspects of the present disclosure as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present invention.

[0031] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0032] In order to illustrate the technical solution of the present invention, a specific embodiment is used below for description, and only the parts related to the embodiment of the present invention are shown.

[0033] Embodiment 1:

[0034] like Figure 1As shown, the present invention provides an animation coloring method based on a video generation model, comprising the following steps: S1, extracting edge line drafts of the middle frame image of the video to be colored, and inputting the edge line drafts into the ControlNet model to obtain edge features of the middle frame image; S2, inputting the first frame image and the last frame image of the video to be colored into the video generation model to obtain image features of the first frame image and the last frame image, and combining the image features and the edge features to obtain edge condition information; wherein the first frame image and the last frame image are both color images that have been colored; S3, performing feature fusion on the first frame image and the last frame image according to the edge condition information to obtain fused features; S4, coloring the middle frame image by using the fused features.

[0035] This embodiment improves the sampling algorithm of the video generation model, takes the first frame and the last frame of the colored video as the coloring reference images, introduces the information of the second reference image, and jointly generates a color video, thereby strengthening the control over video generation. Due to the image information of the first frame and the last frame, the required color can be accurately filled into the middle frame, the continuity of the previous and next frames of the video is strengthened, and it is aligned with the industrial generation, which can solve the coloring work in industrial production. The edge information of the line draft is also introduced into the video production model through the ControlNet control model, so as to further strengthen the intermediate control process of the video generation model and improve the coloring effect of the video.

[0036] Next, combine Figure 1-Figure 4 The specific implementation steps of the animation coloring method based on the video generation model provided in this embodiment are introduced in detail:

[0037] First, execute step S1, extract the edge line draft of the intermediate frame image of the video to be colored, and obtain the edge features of the intermediate frame image by inputting the edge line draft into the ControlNet model. The ControlNet model is a neural network structure that can control the diffusion model, especially the Stable Diffusion model, by adding additional conditions. In this embodiment, the edge line draft of the intermediate frame image is added to use the ControlNet model to achieve precise control of the intermediate process of video generation.

[0038] Specifically, step S1 includes: extracting edge line drawings of the middle frame image of the video to be colored by an edge detection algorithm, inputting the edge line drawings into the ControlNet model, and obtaining output features c of each block of the ControlNet model; wherein the output features c of each block are edge features of the middle frame image. In the ControlNet model, block is the basic unit constituting the entire network structure, which is used to process and convert input features, and control the generation process by processing and converting input features.

[0039] Furthermore, the edge line draft of the intermediate frame image can be extracted by edge detection algorithms such as the Sobel operator, Canny operator or Laplacian operator. In this embodiment, the Canny operator is preferably used to extract the edge line draft. The Canny operator can accurately detect the real edge in the image, and carefully locate the edge pixels. It can also effectively suppress the interference of noise and improve the accuracy of edge detection, thereby improving the accuracy of introducing edge condition information and improving the quality of the generated video.

[0040] Then, step S2 is executed, the first frame image and the last frame image of the video to be colored are input into the video generation model, the image features of the first frame image and the last frame image are obtained, and the edge condition information is obtained by combining the image features and the edge features. The coloring video is generated by the color information of the first frame image and the last frame image, and the line draft information of the middle frame image, so that the quality of the generated video can be enhanced.

[0041] Specifically, step S2 includes: inputting the first frame image and the last frame image of the video to be colored into the video generation model, obtaining the output feature f of each block of the video generation model, adding the output feature f to the output feature c, and obtaining the edge condition information f'. It should be noted that the block of the video generation model is also the basic unit that constitutes the entire network structure, and is also used to process and convert input features; for the feature f output by each block of the video generation model, an output feature c of the block of the ControlNet model is added, that is, a bias feature is added to the video generation model, and adding this bias feature is equivalent to introducing the edge condition information f'=f+c; thereby improving the expressiveness and flexibility of the video generation model, and generating color videos that meet industrial generation needs.

[0042] It should be noted that models such as Stable Video Diffusion (SVD), EasyAnimate, DreamMachine (Luma AI), Image-to-Video or MOFA-Video can be used as the video generation model of this embodiment. Preferably, the SVD model is used. The SVD model has high flexibility and strong visual expression ability. The SVD model is an open source model, has extremely strong ease of use, and is easy to cooperate with ControlNet to achieve control of the intermediate process of video generation.

[0043] Next, step S3 is executed to fuse the features of the first frame image and the last frame image according to the edge condition information to obtain the fused features; wherein the first frame image and the last frame image are both colorized images. At this step, the user only needs to provide the first frame and the last frame images to accurately color the intermediate frame image.

[0044] Specifically, Figure 2 As shown, step S3 includes: S31, using edge condition information as the bias feature of the video generation model, obtaining the first frame latent space feature Z1 of the first frame image and the last frame latent space feature Z2 of the last frame image through the video generation model; S32, dimensional splicing the first frame latent space feature Z1 and the last frame latent space feature Z2 with Gaussian noise respectively, and then denoising to obtain the first frame denoising feature Z1' and the last frame denoising feature Z2'; S33, fusing the first frame denoising feature Z1' and the last frame denoising feature Z2' to obtain the fused feature. In step S33, the formula for obtaining the fused feature is:

[0045] ;

[0046] in, For fusion features; is a weight vector that increases linearly from 0 to 1. The value of each increase in the weight vector is , N is the number of frames to be denoised each time in the video to be colored.

[0047] For example, the first frame image and the last frame image (x1, x2) are input to the video generation model, and the latent space features (Z1, Z2) of the two images are obtained after encoding through the variational autoencoder (VAE); then the noise Zt that obeys the Gaussian noise distribution is sampled, and the latent space features (Z1, Z2) are concatenated with the Gaussian noise Zt, and then denoised using the UNet network to obtain , Z2' ,in, represents the next moment (i.e. t-1) at time t, based on the latent space features after denoising the first frame image Z1 (i.e. the first frame denoising features Z1'); It represents the next moment of time t, based on the latent space features after denoising the last frame image Z2 (i.e., the last frame denoising feature Z2').

[0048] Next, the first frame denoising feature Z1' and the last frame denoising feature Z2' are fused. Specifically, a weight vector is generated. , It increases linearly from 0 to 1. Since both the first frame and the last frame are input into the video generation model, the video generation model will use both the first frame and the last frame as the first frame of the generated video. Therefore, Z2' is time-reversed (it should be noted that due to the addition of the bias feature, when the latent space feature Z2' of the last frame is time-reversed, the bias feature must also be time-reversed), and the last frame becomes the last frame of the generated video. The weight of Z1' as the first frame is slowly reduced from 1 to 0, and the weight of Z2' as the last frame is slowly increased from 0 to 1, and then these two latent space features are added to obtain the fusion feature. .

[0049] In order to strengthen the resonance of the two latent spaces, after fusing the first frame denoising feature Z1' and the last frame denoising feature Z2' to obtain the fused feature, the method further includes: repeatedly performing the denoising operation on the fused feature for multiple times to obtain the final fused feature. Preferably, the denoising operation is repeated three times for each fused feature Z to obtain the final fused feature Z0. The fused feature finally obtained adds the reference information of the last frame, which can ensure the continuity of the entire video frame and the accuracy of the video coloring.

[0050] Finally, step S4 is executed to color the intermediate frame image by fusion features. Specifically, step S4 includes: decoding the fusion features by a decoder to obtain a colorized video generated according to the first frame image, the last frame image, and the intermediate frame image. Preferably, a VAE decoder can be used to decode according to the fusion features to obtain the final colorized video.

[0051] This embodiment generates the color value of the intermediate frame and calculates the connected area of ​​the line drawing, and uses a voting operation to color the intermediate frame image. Specifically, Figure 3 As shown, it includes: S41, calculating all connected areas in the edge line draft of each intermediate frame image; wherein each connected area corresponds to a color area in the reference color draft; the reference color draft is obtained by feature fusion of the first frame image and the last frame image; the connected area is an image area composed of foreground pixels with the same pixel value and adjacent positions in the image; in a binary image, the connected domain is usually composed of pixels with a value of 1 (foreground), while pixels with a value of 0 (background) are not considered as part of the connected domain. The connected areas of the intermediate frame image are similar to the connected areas in the reference color draft image, so for each connected area of ​​the intermediate frame image, a corresponding color area can be found on the reference color draft. S42, based on the first frame image and the last frame image, each connected area is flatly colored by voting. The voting operation can accurately and quickly color the intermediate frame image.

[0052] Further, if Figure 4As shown, step S42 includes: S421, obtaining the color value of the reference color draft corresponding to each connected area, recorded as RGB1; RGB1 is a set of color values ​​of all possible connected areas of the intermediate frame obtained according to the reference color draft; S422, calculating the distance between each pixel point and all color values ​​in RGB1 in the connected area, and assigning the color value closest to the pixel point to the pixel point; in this way, each pixel point in the connected area is assigned a color value, and the color value set of all pixels in the connected area can be recorded as RGB2; S423, applying the color value with the highest frequency in the connected area to the entire connected area. Finally, count all the color values ​​in RGB2, and take the color value with the largest count as the color of the entire connected area. The above operation is performed on each connected area to complete the coloring of the intermediate frame, thereby obtaining a natural and realistic coloring effect. The method of this embodiment can be connected to the production process of frame-by-frame animation, greatly improving industrial production efficiency.

[0053] The embodiment is only a specific example and does not represent only one implementation mode of the present invention.

[0054] Embodiment 2:

[0055] The present invention also provides an embodiment of an animation coloring device based on a video generation model, comprising one or more processors and a memory; wherein the memory is used to store one or more computer programs, and the one or more processors are used to execute one or more computer programs stored in the memory, so that the processor performs the features / steps of the above-mentioned embodiment of an animation coloring method based on a video generation model.

[0056] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. An animation coloring method based on a video generation model, characterized in that: The following steps are involved: Extracting edge line drawings of the middle frame image of the video to be colored, and inputting the edge line drawings into the ControlNet model to obtain edge features of the middle frame image; Inputting the first frame image and the last frame image of the video to be colored into the video generation model, obtaining image features of the first frame image and the last frame image, and combining the image features with the edge features to obtain edge condition information; wherein the first frame image and the last frame image are both color images that have been colored; Performing feature fusion on the first frame image and the last frame image according to the edge condition information to obtain fusion features; Coloring the intermediate frame image by using the fusion feature; The step of extracting edge line drawings of the intermediate frame image of the video to be colored, and inputting the edge line drawings into the ControlNet model to obtain edge features of the intermediate frame image includes: Extracting edge line drawings of the intermediate frame image of the video to be colored by an edge detection algorithm, inputting the edge line drawings into the ControlNet model, and obtaining output features c of each block of the ControlNet model; wherein the output features c of each block are edge features of the intermediate frame image; The step of inputting the first frame image and the last frame image of the video to be colored into a video generation model, obtaining image features of the first frame image and the last frame image, and combining the image features with the edge features to obtain edge condition information includes: Input the first frame image and the last frame image of the video to be colored into a video generation model to obtain an output feature f of each block of the video generation model, and add the output feature f to the output feature c to obtain the edge condition information f'; The step of fusing the first frame image and the last frame image according to the edge condition information to obtain a fused feature includes: The edge condition information is used as a bias feature of the video generation model, and a first frame latent space feature Z1 of the first frame image and a last frame latent space feature Z2 of the last frame image are obtained through the video generation model; The first frame latent space feature Z1 and the last frame latent space feature Z2 are respectively dimensionally concatenated with Gaussian noise, and then denoised to obtain the first frame denoised feature Z1' and the last frame denoised feature Z2'; The first frame denoising feature Z1' and the last frame denoising feature Z2' are fused to obtain a fused feature; In the step of fusing the first frame denoising feature Z1' and the last frame denoising feature Z2' to obtain the fused feature, the formula for obtaining the fused feature is: ； in, is the fusion feature; is a weight vector that increases linearly from 0 to 1, and the value of each increase of the weight vector is , N is the number of frames of the video to be colored that is denoised each time.

2. The method for coloring an animation based on a video generation model according to claim 1, characterized in that: After fusing the first frame denoising feature Z1' and the last frame denoising feature Z2' to obtain a fused feature, the method further includes: repeatedly performing the operation of adding noise and then denoising on the fused feature to obtain a final fused feature.

3. The method for coloring an animation based on a video generation model according to claim 1, characterized in that: The coloring of the intermediate frame image by using the fusion feature includes: decoding the fusion feature by a decoder to obtain a colored video generated according to the first frame image, the last frame image and the intermediate frame image.

4. The method for coloring an animation based on a video generation model according to claim 1, characterized in that: The coloring of the intermediate frame image by using the fusion feature further includes: Calculate all connected areas in the edge line draft of each intermediate frame image; wherein each connected area corresponds to a color area in the reference color draft; the reference color draft is obtained by fusing the features of the first frame image and the last frame image; Based on the first frame image and the last frame image, each connected area is flatly colored through a voting operation.

5. The method for coloring an animation based on a video generation model according to claim 4, characterized in that: The step of flat-painting each connected region based on the first frame image and the last frame image through a voting operation includes: Obtaining the color value of the reference color draft corresponding to each of the connected regions, recorded as RGB1; Calculate the distance between each pixel point and all color values ​​in RGB1 in the connected area, and assign the color value closest to the pixel point to the pixel point; The color value that appears most frequently in the connected area is evenly applied to the entire connected area.

6. An animation coloring device based on a video generation model, characterized in that: include: one or more processors; A memory for storing one or more computer programs, and one or more processors for executing the one or more computer programs stored in the memory, so that one or more processors execute an animation coloring method based on a video generation model as described in any one of claims 1-5.

Images