A deep learning-based animation video line drawing coloring method

Abstract

The application belongs to the field of image coloring in computer vision, and relates to an animation video line drawing coloring method based on deep learning, which can be applied to the image processing scene. According to reference frames and target frames, the coloring algorithm uses a multi-level moving information feature distillation module to obtain coarse-grained moving information and multi-scale target frame preliminary color distribution information between the reference frames and the target frames, eliminates the interference of low-confidence moving fine information through a confidence estimation module based on moving information, and finally uses a coloring module to complete high-quality and fine-grained target frame coloring. The multi-level moving information feature distillation module proposed in the application can obtain high-quality moving information, and a confidence estimation module based on moving information is designed, further improving the semantic consistency of the animation video. Experimental results show that compared with other animation line drawing and video animation line drawing coloring methods, the method can bring more detailed coloring effect.

Description

Technical Field

[0001] This invention belongs to the field of image coloring in computer vision, and relates to a deep learning-based algorithm for coloring animated video line art, which can be applied to image processing scenarios. Background Technology

[0002] Anime line art video coloring refers to generating a colored image on a given target line art image and a reference image. In animation production, this involves a significant amount of repetitive manual work, as line art with similar content is repeatedly colored. Therefore, many automatic coloring methods have been developed. These non-learning-based methods generally work well when the number of regions remains constant and the variation between regions is small. In recent years, AI-powered line art video coloring methods have shown great potential in this task. Currently, many deep learning-based line art coloring methods have made significant progress, but when simply applied to coloring animated line art images, they can lead to inconsistencies in color between frames if the movement between animation frames is not considered. Therefore, maintaining the consistency of semantic information between frames is crucial. Summary of the Invention

[0003] This invention provides a deep learning-based method for coloring line art in animated videos. The coloring algorithm uses an inter-frame motion information feature distillation module to obtain coarse-grained motion information between a reference frame and a target frame, along with the initial color layout of the target frame. A motion information-based confidence estimation module eliminates interference from low-confidence motion information. Finally, a coloring module completes high-quality, fine-grained coloring of the target frame. The motion information feature distillation module proposed in this invention can obtain high-quality motion information, and the designed motion information-based confidence estimation module further improves the semantic consistency of the animated video.

[0004] The technical solution of the present invention:

[0005] A deep learning-based method for coloring animated video line art is characterized by being implemented through three modules, as follows:

[0006] The first module is used to improve the accuracy of semantic alignment of movement information between line art frames by distilling the movement information between color frames.

[0007] The first module comprises two sub-networks: a RAFT motion estimation network for color images and a RAFT motion estimation network for line art images; both sub-networks use the same Encoder feature extractor; the specific steps are as follows:

[0008] Step 1: Two color frames are processed by a RAFT motion estimation network for color images to obtain inter-frame motion information features.

[0009] Color reference frames and color target frames are input into the RAFT motion estimation network for color images; the color reference frames and color target frames are training frame pairs extracted from single scenes in different animated films; the RAFT motion estimation network for color images is supervised by Warp Loss and Structural Loss to obtain the motion information feature T′ from the color reference frame to the color target frame. i The obtained T′ i Optical flow TF is obtained through multiple iterations of a RAFT motion estimation network based on color images. i Using the obtained optical flow TF i The color reference frame is warped using the grid sample function to obtain the distorted image w. Tc , that is, the target color reference frame;

[0010] Step 2: The motion information features T′ between color frames in the RAFT motion estimation network of the color image are... i Distilled into the RAFT motion estimation network of the line drawing image

[0011] The RAFT motion estimation network for line art images takes a reference frame and a target frame as input. The reference frame and target frame are line art pairs extracted from real color animation frames using a line art extractor. The RAFT motion estimation network then passes the reference frame and target frame through the line art images to obtain the inter-frame motion information feature S′. i The RAFT motion estimation network structure for the line drawing images is the same as that for the color images, but the two networks do not share parameters. The motion information features T′ between color frames from step 1 are used. i As a soft label, the inter-frame motion information feature S′ of the line art image generated by the RAFT motion estimation network is used for the line art image. i Supervision was performed using mean squared error loss, Warp Loss, and Structural Loss to extract the motion information features T′ between color frames from step 1. i Distilled to the movement information feature S′ between line drawing frames i The obtained S′ i Optical flow SF is obtained through multiple iterations of a RAFT motion estimation network based on line art images. i Using the obtained optical flow SF i The color reference frame from step 1 is warped using the grid sample function to obtain the distorted image w. Sc That is, the initial color layout map of the target frame;

[0012] To ensure the accuracy of semantic color alignment, the RAFT motion estimation network for color images is first pre-trained using Warp Loss and Structural Loss. After the RAFT motion estimation network for color images is stabilized, Warp Loss, Structural Loss, and mean squared error loss are used to jointly train the RAFT motion estimation network for line art images and the RAFT motion estimation network for color images, so that the RAFT motion estimation network for line art images can converge faster.

[0013] The loss function used in this module is:

[0014] Mean squared error loss: The purpose is to ensure that the inter-frame motion information features of the RAFT motion estimation network for line art images and the RAFT motion estimation network for color images are the same.

[0015] MSE=||ρ(S′ i )-ζ(T′ i )||2Formula 1

[0016] Where i represents the i-th level, S′ i T′ represents the motion information features between the online frames of the i-th level. i ρ and ζ represent the motion information features between color frames at the i-th level, and are convolutional layers used to map these two feature maps onto a unified feature space.

[0017] Distortion loss (Warp Loss): Constrains the network to learn a semantically aligned color layout.

[0018] L warp =||w Tc (j,k)-T C (j,k)||1 Formula 2 Where, w Tc T represents the generated distorted image. C (j,k) represents the color target frame image; (j,k) represents the spatial location of the distorted image.

[0019] Structural Loss: The purpose is to preserve the overall outline structure of the line art, so that the line art extracted by the SketchKeras line art extractor is closer to the real line art.

[0020] L structure =||φ t (T′ c )-φ i (T c Formula 3, where T′ c It is the target color reference frame generated in step 1, T c It is a color target frame, φ i (T′c ) represents the i-th layer line art feature generated by the SketchKeras line art extractor, φ i (T c ) represents the i-th layer line art feature generated by the SketchKeras line art extractor after processing the color target frame, φ i This represents the i-th layer feature extracted by the pre-trained SketchKeras network for extracting line art.

[0021] The second module, based on the confidence assessment of mobile information, obtains an accurate high-confidence color distribution reference map. Step 3 involves acquiring the confidence distribution feature map.

[0022] The initial color layout map of the target frame obtained by the RAFT motion estimation network of the line drawing image in step 2 is used. Sc Line drawing frame movement information feature S′ i Optical flow SF i The target frame's initial color layout map w is used as input to this module, along with a color reference frame. Sc The color reference frame and the image were processed by a feature extraction network, which contained a 1×1 convolutional layer and a ReLU layer; the inter-frame motion information feature S′ was used for the line art. i First, the matching rate is calculated using the softmax function. Then, the probability of the top-K matches is extracted using a Top-K pooling layer, removing the low-confidence parts. Finally, a SoftArgmax layer is used to obtain new inter-frame motion information features of the line art. These new inter-frame motion information features are then used by a feature extraction network to extract the initial color layout map w of the target frame. Sc Features extracted from color reference frames and optical flow SF i The features are concatenated at the channel level, and feature fusion is performed using three convolutional layers and a sigmoid activation function to generate a confidence distribution feature map CN(j,k).

[0023] This step proposes a binarized cross-entropy loss Lc, which aims to identify regions with incorrect initial color layout.

[0024]

[0025] in, For the pseudo-confidence graph CN^(j,k), y i Generate a confidence distribution feature map CM(j,k) for step 3. The pseudo-confidence map CM^(j,k) is obtained by comparing it with the initial color layout map w of the target frame. Sc and the color target frame image T in step 1 C get.

[0026] The formula for obtaining a pseudo-confidence plot is:

[0027]

[0028]

[0029] T C w represents the color target frame image in step 1. Sc The initial color layout map of the target frame is represented by (j,k), where (j,k) represents the image pixel coordinates, α and β represent the maximum and minimum pixel values ​​of the confidence map, and θ is a hyperparameter.

[0030] Step 4: Obtain a high-confidence color distribution reference image

[0031] By performing a Hadamard product between the initial color layout map of the target frame obtained in step 2 and the confidence distribution feature map obtained in step 3, a high-confidence color distribution reference map can be obtained.

[0032] The third module is a colorization network that generates target color frames based on attention mechanisms and multi-scale feature fusion.

[0033] Step 5: Generate the target color frame

[0034] An attention module is introduced into the encoder part of the U-Net structure. The line art target frame in step 2 is input into the encoder of the U-Net structure to obtain multi-scale features. At the same time, the color reference frame in step 1 and the high-confidence color distribution reference map obtained in step 4 are input into the attention module. The attention mechanism is used to calculate the cosine similarity between the high-confidence color distribution map features at the corresponding scale and the transpose of the color reference frame features in step 1. The cosine similarity is multiplied by the color reference frame features in step 1 to obtain the attention feature map. Then, the obtained multi-scale attention features are concatenated with the multi-scale features of the line art target frame obtained by the encoder of the U-Net structure at the channel level. Finally, the concatenated feature map is used to generate the target color frame through the decoder of the U-Net structure.

[0035] The loss function used in this module is:

[0036] L1 loss aims to ensure that the content of the generated color target frame and the real color target frame are as similar as possible.

[0037] L1=||T′ c -T c ||1 Formula 7 Where, T′ c T represents the generated color target frame. c Represents a true color target frame;

[0038] Perceptual loss: Using perceptual loss to constrain the generated color target frame T′ cand true color target frame T c Feature reconstruction matching between:

[0039]

[0040] in, It is the distribution of the generated color target frames. It is the distribution of true color target frames, φ i Let represent the i-th layer features from the pre-trained VGG-19 network, and chw represent the size of the feature map of the i-th layer;

[0041] Style loss: The Gram matrix is ​​used to measure the differences in high-dimensional feature maps.

[0042]

[0043] in, It is the generated color target frame T′ c distributed, It is a true color target frame T c distributed, It is the activation graph φ of the i-th layer of the pre-trained VGG-19 network. i Calculate the Gram matrix, and then calculate the Euclidean distance between corresponding layers;

[0044] The overall loss function of the model is defined as:

[0045]

[0046] Where λ mse , λ warp , λ structure , λ style , λ c , λ perceptual ,and These are the parameters for mean squared error loss, distortion loss, structure loss, style loss, binary cross-entropy loss, perceptual loss, and L1 loss, respectively.

[0047] The beneficial effects of this invention are as follows: This invention enables high-quality coloring of animation line art frames based on color reference frames, generating coloring results with high-quality color semantic alignment. Specifically, it generates colors that match the corresponding areas of the reference frame, and the overall coloring style is the same as the color reference frame. Furthermore, this invention significantly improves the coloring results of animation line art and has strong adaptability to different line art coloring scenarios. Attached Figure Description

[0048] Figure 1 This is the overall flowchart of the present invention.

[0049] Figure 2This is a flowchart of the inter-frame motion information feature extraction module.

[0050] Figure 3 This is a flowchart of the confidence estimation module based on mobile information.

[0051] Figure 4 This is a flowchart of the coloring module. Detailed Implementation

[0052] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0053] This invention builds an overall algorithm network based on the PyTorch deep learning framework, and the basic data processing is implemented using the Python language.

[0054] Step 1: Create training sets for both color animation frames and line art animation frames. Using FFmpeg's scene transition detector, 5971 scenes (approximately 26 frames per scene) were extracted from ten Miyazaki films, totaling 155250 color animation frames for training. The color reference frames and color target frames input to the RAFT motion estimation network for color images in the first module are training frame pairs extracted at five-frame intervals from individual scenes in different animated films. The SketchKeras line art extractor was used to extract line art pairs from real color animation frames, which served as input to the RAFT motion estimation network for line art images in the first module.

[0055] Step 2: First, the RAFT motion estimation network for the color images in the first module is pre-trained using Warp Loss and Structural Loss to ensure the accuracy of semantic color alignment. The total number of training iterations is 10. 6 After the RAFT motion estimation network for color images converges, the RAFT motion estimation networks for both color and line art images are jointly trained to accelerate the convergence of the line art image RAFT motion estimation network. For the confidence estimation module based on motion information, the network is pre-trained separately until convergence, and then its parameters are fixed. The overall network structure is then built, and all loss functions proposed in this invention are used for end-to-end training until convergence. The Adam optimizer is set to a learning rate of 1e-4, and the total training iterations are 6*10. 6 The momentum parameters β1 = 0.5, β2 = 0.9, and the batch size is 2. We set the hyperparameter θ = 128 for the confidence estimation module based on motion information. During the testing phase, the RAFT motion estimation network for the color images in the first module was removed, while the RAFT motion estimation network for the line art images in the first module and the confidence evaluation network structure based on motion information in the second module, as well as the colorization network for generating the target color frame in the third module, were retained.

[0056] The specific steps are as follows:

[0057] Step 3: Construct a RAFT motion estimation network for color images to extract inter-frame motion information features. The color reference frame and the color target frame are the inputs to the RAFT motion estimation network for color images in the first module. The RAFT motion estimation network for color images is supervised by Warp Loss and Structural Loss to obtain the motion information feature T′ from the color reference frame to the color target frame. i The obtained T′ i Optical flow TF is obtained through multiple iterations of a RAFT motion estimation network based on color images. i Using the obtained optical flow TF i The color reference frame is warped using the grid sample function to obtain the distorted image w. Tc This refers to the target color reference frame. The RAFT motion estimation network for the color image is optimized using the following loss function:

[0058] Distortion loss (Warp Loss): Constrains the network to learn a semantically aligned color layout.

[0059] L warp =||w Tc (j,k)-T C (j,k)||1 Formula 2 Where, w Tc T represents the generated distorted image. C (j,k) represents the color target frame image; (j,k) represents the spatial location of the distorted image.

[0060] Structural Loss: The purpose is to preserve the overall outline structure of the line art, so that the line art extracted by the SketchKeras line art extractor is closer to the real line art.

[0061] L structure =||φ i (T′ c )-φ i (T c Formula 3, where T′ c It is the target color reference frame generated in step 1, T c It is a color target frame, φ i (T′ c ) represents the i-th layer line art feature generated by the SketchKeras line art extractor, φ i (T c ) represents the i-th layer line art feature generated by the SketchKeras line art extractor after processing the color target frame, φ iThis represents the i-th layer feature extracted by the pre-trained SketchKeras network for extracting line art.

[0062] Step 4: Construct a RAFT motion estimation network for the line art images to extract motion information features between line art frames. The RAFT motion estimation network for the line art images uses the motion information features T′ between color frames obtained from the line art reference frame, the line art target frame, and the RAFT motion estimation network for the color images in Step 3. i The color reference frame in Step 3 is used as input; the line art reference frame and the line art target frame are passed through the RAFT motion estimation network of the line art image with the same structure as the RAFT motion estimation network of the color image to obtain the inter-frame motion information feature S′ of the line art. i ; where the motion information feature T′ between color frames in Step 3 i As a soft label, the inter-frame motion information feature S′ of the line art image generated by the RAFT motion estimation network is used for the line art image. i Supervision was performed using mean squared error loss, Warp Loss, and Structural Loss to extract the motion information features T′ between color frames in Step 3. i Distilled to the movement information feature S′ between line drawing frames i The obtained S′ i Optical flow SF is obtained through multiple iterations of a RAFT motion estimation network based on line art images. i Using the obtained optical flow SF i The color reference frame in Step 3 is warped using the grid sample function to obtain the distorted image w. sc That is, the initial color layout map of the target frame;

[0063] The RAFT motion estimation network for the line art image is optimized using the following loss function, along with Warp Loss and Structural Loss from Step 3:

[0064] Mean squared error loss: The purpose is to force the inter-frame motion information features of the RAFT motion estimation network for line art images and the RAFT motion estimation network for color images to be the same;

[0065] MSE=||ρ(S′ i )-ζ(T′ i )||2Formula 1

[0066] Where i represents the i-th level, S′ i T′ represents the motion information features between the online frames of the i-th level. i ρ and ζ represent the motion information features between color frames at the i-th level, and are convolutional layers used to map these two feature maps onto a unified feature space.

[0067] Step 5: Construct a confidence estimation module based on motion information, which uses the initial color layout map w of the target frame obtained from the RAFT motion estimation network of the line drawing image in Step 4. Sc Line drawing frame movement information feature S′ i Optical flow SF i The target frame's initial color layout map w is used as input to this module, along with a color reference frame. Sc The color reference frame and the image were processed by a feature extraction network, which contained a 1×1 convolutional layer and a ReLU layer; the inter-frame motion information feature S′ was used for the line art. i First, the matching rate is calculated using the softmax function. Then, the probability of the top-K matches is extracted using a Top-K pooling layer, removing the low-confidence parts. Finally, a Soft Argmax layer is used to obtain new inter-frame motion information features of the line art. These new inter-frame motion information features are then used by a feature extraction network to extract the initial color layout map w of the target frame. Sc Features extracted from color reference frames and optical flow SF i The features are concatenated at the channel level, and feature fusion is performed using three convolutional layers and a sigmoid activation function to generate a confidence distribution feature map. The confidence distribution feature map is then multiplied with the initial color layout map of the target frame obtained in Step 4 to obtain a high-confidence color distribution reference map.

[0068] The confidence estimation module is optimized using the following loss function:

[0069] Binarized cross-entropy loss: The purpose is to identify regions where the initial color layout is incorrect.

[0070]

[0071] in, For the pseudo-confidence graph CM^(j,k), y i Generate the confidence distribution feature map CM(j,k) for step 5. Among them, The pseudo-confidence map CM^(j,k) is obtained by comparing it with the initial color layout map w of the target frame. Sc and the color target frame image T in step 1 C get.

[0072] The formula for obtaining a pseudo-confidence plot is:

[0073]

[0074]

[0075] TC w represents the color target frame image in step 1. Sc The initial color layout map of the target frame is represented by (j,k), where (j,k) represents the image pixel coordinates, α and β represent the maximum and minimum pixel values ​​of the confidence map, and θ is a hyperparameter.

[0076] Step 6: Construct a colorization module based on attention mechanism and multi-scale feature fusion to generate target color frames. Input the line art target frame into a U-Net encoder to obtain multi-scale features. At the same time, input the color reference frame from Step 1 and the high-confidence color distribution reference map obtained in Step 5 into the attention module. The resulting multi-scale attention feature map is concatenated with the multi-scale features of the line art target frame obtained from the U-Net encoder at the channel level. Finally, the concatenated feature map is used to generate the target color frame through a U-Net decoder.

[0077] Step 7: We use the following loss function to quantify and train the difference between the color target frame generated by the generator and the real color target frame:

[0078] The coloring module is optimized using the following loss function:

[0079] L1 loss aims to ensure that the content of the generated color target frame and the real color target frame are as similar as possible.

[0080] L1=||T′ c -T c ||1 Formula 7 Where, T′ c T represents the generated color target frame. c Represents a true color target frame;

[0081] Perceptual loss: Using perceptual loss to constrain the generated color target frame T′ c and true color target frame T c Feature reconstruction matching between:

[0082]

[0083] in, It is the distribution of the generated color target frames. It is the distribution of true color target frames, φ i Let represent the i-th layer features from the pre-trained VGG-19 network, and chw represent the size of the feature map of the i-th layer;

[0084] Style loss: The Gram matrix is ​​used to measure the differences in high-dimensional feature maps.

[0085]

[0086] in, It is the generated color target frame T′ c distributed, It is a true color target frame T c distributed, It is the activation graph φ of the i-th layer of the pre-trained VGG-19 network. i Calculate the Gram matrix, and then calculate the Euclidean distance between corresponding layers;

[0087] The overall loss function of the model is defined as:

[0088]

[0089] Where λ mse , λ warp , λ structure , λ style , λ c , λ perceptual ,and These are the parameters for mean squared error loss, distortion loss, structure loss, style loss, binary cross-entropy loss, perceptual loss, and L1 loss, respectively.

[0090] Step 8: After training the model to convergence according to the above steps, use the line art reference image, the line art target image, and the color reference frame as input to the network to obtain the result of automatic coloring of the animation line art, and finally generate a complete color animation scene.

[0091] Step 9: Comparison and Analysis of Experimental Results

[0092]

[0093] Image evaluation metrics:

[0094] SSMI: Structural Similarity Index, a metric that measures the similarity between two images, comparing brightness, contrast, and structure. A higher SSMI value is better.

[0095] PSNR: Peak Signal-to-Noise Ratio. When evaluating image quality, a higher PSNR value indicates better image quality.

[0096] FID: Image quality assessment metric, a method to measure the similarity between two image datasets. The more similar our generated image is to the real image, the better; the smaller the FID value, the better.

[0097] We use structural similarity, peak signal-to-noise ratio, and image quality as evaluation metrics. Our method is compared with recently proposed methods for coloring line art, including:

[0098] Pix2pix[P.Isola,J.-Y.Zhu,T.Zhou,and A.A.Efros,“Image-to-imagetranslation with conditional adversarial networks,”in IEEE Conference onComputer Vision and Pattern Recognition,CVPR,June,2017,pp.1125–1134.],

[0099] DeepIA[J.Liao,Y.Yao,L.Y uan,G.Hua,and S.B.Kang,“Visual attributetransfer through deep image analogy,”ACM Trans.Graph.,vol.36,no.4,pp.120:1–120:15,2017.],

[0100] SGA[Z.Li,Z.Geng,Z.Kang,W.Chen,and Y.Yang,“Eliminating gradientconflict in reference-based line-art colorization,”in European Conference onComputer Vision,ECCV,October,2022,vol.13677,2022,pp.579–596.],

[0101] TCVC[H.Thasarathan,K.Nazeri,and M.Ebrahimi,“Automatic temporallycoherent video colorization,”in IEEE Conference on Computer and Robot Vision,CRV,May,2019,pp.189–194.],

[0102] LCMFTN [Z.Qian, W.Bo, W.Wei, L.Hai, and LJHui, "Line art correlationmatching network for automatic animation colorization," in Winter Conference on Applications of Computer Vision, WACV, June, 2021, pp.189–194.],

[0103] DLA VC [M.Shi, J.Zhang, S.Chen, L.Gao, Y.Lai, and F.Zhang, "Deep line artvideo colorization with a few references," IEEE transactions on visualization and computer graphics, TVCG, 2022. [Online].Available: http: / / doi.org / 10.1109 / TVCG.2022.3146000.].

[0104] Throughout the experiment, only the first frame of the color image is needed to colorize the entire line art image sequence. We compared our method with other methods in scenes with large geometric motion, small geometric motion, long scenes (greater than 25 frames), and short scenes (less than or equal to 25 frames), and calculated the average results across all scenes. The table shows that our method significantly improves performance on the three evaluation metrics: SSMI, PSNR, and FID.

[0105] Since existing methods ignore inter-frame motion information, they can lead to semantic misalignment and poor generation results. The method in this paper focuses on inter-frame motion information. Through the construction of three modules and the constraints of the loss function, the line art coloring effect is significantly improved.

Claims

1. A method for coloring line art in animated videos based on deep learning, characterized in that, The method for coloring line art in animated videos is implemented through three modules, as follows: The first module is used to improve the accuracy of semantic alignment of movement information between line art frames by distilling the movement information between color frames. The first module comprises two sub-networks: a RAFT motion estimation network for color images and a RAFT motion estimation network for line art images; both sub-networks use the same Encoder feature extractor. The specific steps are as follows: Step 1: Two color frames are processed by a RAFT motion estimation network for color images to obtain inter-frame motion information features. Color reference frames and color target frames are input into the RAFT motion estimation network for color images; the color reference frames and color target frames are training frame pairs extracted from single scenes in different animated films; the RAFT motion estimation network for color images is supervised by Warp Loss and Structural Loss to obtain the motion information feature T′ from the color reference frame to the color target frame. i The obtained T′ i Optical flow TF is obtained through multiple iterations of a RAFT motion estimation network based on color images. i Using the obtained optical flow TF i The color reference frame is warped using the grid sample function to obtain the distorted image w. Tc , that is, the target color reference frame; Step 2: The motion information features T′ between color frames in the RAFT motion estimation network of the color image are... i Distilled into the RAFT motion estimation network of the line drawing image The RAFT motion estimation network for line art images takes a reference frame and a target frame as input. The reference frame and target frame are line art pairs extracted from real color animation frames using a line art extractor. The RAFT motion estimation network then passes the reference frame and target frame through the line art images to obtain the inter-frame motion information feature S′. i The RAFT motion estimation network structure for the line drawing images is the same as that for the color images, but the two networks do not share parameters. The motion information features T′ between color frames from step 1 are used. i As a soft label, the inter-frame motion information feature S′ of the line art image generated by the RAFT motion estimation network is used for the line art image. i Supervision was performed using mean squared error loss, Warp Loss, and Structural Loss to extract the motion information features T′ between color frames from step 1. i Distilled to the movement information feature S′ between line drawing frames i The obtained S′ i Optical flow SF is obtained through multiple iterations of a RAFT motion estimation network based on line art images. i Using the obtained optical flow SF i The color reference frame from step 1 is warped using the grid sample function to obtain the distorted image w. Sc That is, the initial color layout map of the target frame; To ensure the accuracy of semantic color alignment, the RAFT motion estimation network for color images is first pre-trained using Warp Loss and Structural Loss. After the RAFT motion estimation network for color images is stabilized, Warp Loss, Structural Loss, and mean squared error loss are used to jointly train the RAFT motion estimation network for line art images and the RAFT motion estimation network for color images, so that the RAFT motion estimation network for line art images can converge faster. The loss function used in this module is: Mean squared error loss: The purpose is to force the inter-frame motion information features of the RAFT motion estimation network for line art images and the RAFT motion estimation network for color images to be the same; MSE=||ρ(S′ i )-((T′ i )||2 Formula 1 in, i represents the i-th level, S′ i T represents the motion information features between the online frames of the i-th level. i ' represents the motion information feature between color frames at the i-th level, and ρ and ζ are convolutional layers used to map these two feature maps onto a unified feature space; Distortion loss (Warp Loss): Constrains the network to learn a semantically aligned color layout. L warp =||w Tc (j, k)-T C (j, k)||1 Formula 2 Among them, w Tc T represents the generated distorted image. C (j, k) represents the color target frame image; (j, k) represents the spatial location of the distorted image. Structural Loss: The purpose is to preserve the overall outline structure of the line art, so that the line art extracted by the SketchKeras line art extractor is closer to the real line art. L structure =||φ i (T c ′)-φ i (T c )||1 Formula 3 Among them, T′ c It is the target color reference frame generated in step 1, T c It is a color target frame, φ i (T′ c ) represents the i-th layer line art feature generated by the SketchKeras line art extractor, φ i (T c ) represents the i-th layer line art feature generated by the SketchKeras line art extractor after processing the color target frame, φ i This represents the i-th layer feature extracted by the pre-trained SketchKeras network for extracting line art. The second module obtains an accurate high-confidence color distribution reference map based on confidence assessment of mobile information. Step 3: Obtain the confidence distribution feature map The initial color layout map of the target frame obtained by the RAFT motion estimation network of the line drawing image in step 2 is used. Sc Line drawing frame movement information feature S′ i Optical flow SF i The target frame's initial color layout map w is used as input to this module, along with a color reference frame. Sc The color reference frame and the image were processed by a feature extraction network, which contained a 1×1 convolutional layer and a ReLU layer; the inter-frame motion information feature S′ was used for the line art. i First, the matching rate is calculated using the softmax function. Then, the probability of the top-K matches is extracted using a Top-K pooling layer, removing the low-confidence parts. Finally, a Soft Argmax layer is used to obtain new inter-frame motion information features of the line art. These new inter-frame motion information features are then used by a feature extraction network to extract the initial color layout map w of the target frame. Sc Features extracted from color reference frames and optical flow SF i The features are concatenated at the channel level, and feature fusion is performed using three convolutional layers and a sigmoid activation function to generate a confidence distribution feature map CM(j,k). To identify regions with incorrect initial color layouts, a binarized cross-entropy loss Lc is introduced: in, Let CM^(j,k) be the pseudo confidence map, and yi be the confidence distribution feature map CM(j,k) generated in step 3; The pseudo-confidence map CM^(j,k) is obtained by comparing it with the initial color layout map w of the target frame. Sc and the color target frame image T in step 1 C get; The formula for obtaining a pseudo-confidence plot is: T C w represents the color target frame image in step 1. Sc The initial color layout map of the target frame is represented by (j, k), which represents the image pixel coordinates. α and β represent the maximum and minimum pixel values ​​of the confidence map, and θ is a hyperparameter. Step 4: Obtain a high-confidence color distribution reference image By performing a Hadamard product between the initial color layout map of the target frame obtained in step 2 and the confidence distribution feature map obtained in step 3, a high-confidence color distribution reference map can be obtained. The third module is a colorization network that generates target color frames based on attention mechanisms and multi-scale feature fusion. Step 5: Generate the target color frame An attention module is introduced into the encoder part of the U-Net structure. The line art target frame in step 2 is input into the encoder of the U-Net structure to obtain multi-scale features. At the same time, the color reference frame in step 1 and the high-confidence color distribution reference map obtained in step 4 are input into the attention module. The attention mechanism is used to calculate the cosine similarity between the high-confidence color distribution map features at the corresponding scale and the transpose of the color reference frame features in step 1. The cosine similarity is multiplied by the color reference frame features in step 1 to obtain the attention feature map. Then, the obtained multi-scale attention features are concatenated with the multi-scale features of the line art target frame obtained by the encoder of the U-Net structure at the channel level. Finally, the concatenated feature map is used to generate the target color frame through the decoder of the U-Net structure. The loss function used in this module is: L1 loss aims to ensure that the content of the generated color target frame and the real color target frame are as similar as possible. L1=||T′ c -T c ||1 Formula 7 Among them, T′ c T represents the generated color target frame. c Represents a true color target frame; Perceptual loss: Using perceptual loss to constrain the generated color target frame T′ c and true color target frame T c Feature reconstruction matching between: in, It is the distribution of the generated color target frames. It is the distribution of true color target frames, φ i Let represent the i-th layer features from the pre-trained VGG-19 network, and chw represent the size of the feature map of the i-th layer; Style loss: The Gram matrix is ​​used to measure the differences in high-dimensional feature maps. in, It is the generated color target frame T′ c distributed, It is a true color target frame T c distributed, It is the activation graph φ of the i-th layer of the pre-trained VGG-19 network. i Calculate the Gram matrix, and then calculate the Euclidean distance between corresponding layers; The overall loss function of the model is defined as: Where λ mse , λ warp , λ structure , λ style , λ c , λ perceptual ,and These are the parameters for mean squared error loss, distortion loss, structure loss, style loss, binary cross-entropy loss, perceptual loss, and L1 loss, respectively.

Images