Face image colorization method based on residual structure-parsing task assistance

Abstract

The present invention proposes a facial image colorization method based on residual structure-analysis task assistance, and the implementation steps are: 1. Generate a training set; 2. Construct a residual structure-multi-task neural network; 3. Train the residual structure-multi-task neural network; 4. Colorize the facial image. The present invention introduces a parsing task into the colorization task on the basis of the residual structure, thereby constructing a residual structure-multi-task network. The residual structure enhances the learning ability of the network, and the parsing task assists the colorization task so that it can perform more semantic facial image colorization, effectively improving the quality of facial image colorization.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to a residual structure-analysis task-assisted facial image coloring method in the field of facial image processing technology. The present invention can be used to color facial images in a more semantically appropriate manner. Background Art

[0002] Colorization, especially colorization of face images, is an important issue in the field of low-level image processing. It can be used to restore the color of colorless face images, which can then be used for downstream tasks such as face recognition and detection, or for appreciation. With the development of the times, people can now digitize face photos and then color them manually. However, manual colorization usually takes a lot of time and energy, which makes it very difficult to color a large number of face images. Therefore, there have been some methods that try to design efficient automatic image colorization or face image colorization methods. The high efficiency and high performance of deep learning has made it play an important role in this field.

[0003] Beijing Feisou Technology Co., Ltd. disclosed a deep learning-based image colorization method in its patent application "Image colorization method and device based on generative adversarial network" (application number: 201811595178X, application date: 2018-12-25, application publication number: CN 109859288 A). The steps are as follows: 1. Construct a generative adversarial network model with an attention mechanism; 2. Input the image sample to be colored into the generator of the generative adversarial network model with an attention mechanism for training to obtain the color image of the image sample to be colored, and input Gaussian white noise into the generator for training to obtain the virtual image of the image sample to be colored; 3. Calculate the loss between the virtual image of the image sample to be colored and the color image sample corresponding to the image sample to be colored to obtain the loss result of the generator, and use the back propagation algorithm to update the parameters of the generator based on the loss result of the generator; 4. Input the color image of the image sample to be colored and the color image sample corresponding to the image sample to be colored into the discriminator of the generative adversarial network model with an attention mechanism for discrimination to obtain the discrimination loss of the discriminator, and use the back propagation algorithm to update the parameters of the discriminator based on the discrimination loss. 5. When the loss result of the generator and the discrimination loss are both maintained within the preset threshold range and maintained for a period of time, a trained image colorization model based on the generative adversarial network is obtained. The disadvantage of this method is that although the generative adversarial network model can improve the quality of colorization, the possibility of training collapse is high, which eventually leads to the collapse of colorization quality. This is because the training objectives of the generator and the discriminator in the generative adversarial network model are opposite, which can easily lead to the problem of mode collapse in network training fluctuations and then the collapse of colorization quality. At the same time, this method also lacks category information, that is, semantic guidance, which may lead to the assignment of wrong colors to areas with different semantics on the image. For example, for a face image, the same color may be assigned to the skin and lips, the same color may be assigned to the background and hair, and the eyes may be incorrectly colored, resulting in a serious decline in colorization quality.

[0004] The steps in the patent document "Grayscale Image Colorization Method Based on Convolutional Neural Network and Conditional Generative Adversarial Network" (Application Number: 2020104279531 Application Date: 2020-05-20 Application Publication Number: CN 111627080 A) applied by Guangxi Normal University are as follows: 1. Classify the image using a transfer learning method based on a residual network. 2. Input the original grayscale image and the label of the image into the image automatic colorization network. The image automatic colorization network matches the corresponding colorization model for the image according to the label of the image to automatically colorize the image. The disadvantage of this method is that the colorization network in this method adopts an ordinary convolution structure. This design has limited modeling ability for high-pass information, which limits the final colorization quality of this method. At the same time, although this method uses semantic information to enhance the colorization quality, semantic information must be used as a condition in the colorization process, that is, the performance of the colorization task is overly dependent on the quality of semantic information, which may directly lead to the collapse of colorization quality. Summary of the invention

[0005] The purpose of the present invention is to propose a facial image colorization method based on residual structure-parsing task assistance in view of the defects of the above-mentioned prior art, which is used to solve the problems of limited colorization ability due to insufficient network learning ability and failure to semantically colorize different areas on the face due to neglect or improper use of semantic information.

[0006] The idea of ​​realizing the purpose of the present invention is to construct a network using a residual structure. Specifically, the network is constructed by residual layers constructed by residual blocks, which uses the idea of ​​residuals in residuals. Such step-by-step construction of skip branches allows a large amount of low-pass information in the image to pass through the network quickly, allowing the network to reduce the learning of unnecessary information and pay more attention to the information it needs to learn, thereby improving the learning ability of the network to solve the problem of limited colorization ability caused by insufficient network learning ability. On the basis of constructing the colorization task network, the parsing task is introduced in parallel to assist in forming a multi-task network. The parsing task is to classify and learn semantics for different areas on the face image. Both the colorization task and the parsing task convert the image from one space to another. They are both image translation tasks and have an intrinsic connection. In fact, the two tasks implicitly promote each other. Better colorization of the colorization task can enable the parsing task to perform more accurate parsing. The parsing task distinguishes the categories of different areas on the face image to assist the colorization task in performing more semantic colorization. This overcomes the problem of the existing technology ignoring semantic information. At the same time, because the colorization task and the parsing task are designed in parallel, the quality of the colorization task will not be overly dependent on semantic information, thereby solving the problem of improper use of semantic information in the existing technology.

[0007] The specific steps of the present invention are as follows:

[0008] Step 1: Generate training set:

[0009] (1a) Select at least 24,000 color face images;

[0010] (1b) Each color face image is inferred through the trained face parsing network BiSeNet to generate a parsed image containing a parsed category label corresponding to the color face image;

[0011] (1c) generating a grayscale image of each color face image through linear mapping;

[0012] (1d) combining each color face image with its corresponding parsed image and grayscale image to generate a data sample;

[0013] (1e) All data samples are combined into a training set;

[0014] Step 2: Build a residual structure-multi-task neural network:

[0015] (2a) The structure of the encoder is constructed as follows: head, first downsampling layer, first residual layer, second downsampling layer, second residual layer, third downsampling layer, third residual layer, fourth downsampling layer, fourth residual layer, fifth downsampling layer;

[0016] The head consists of a convolution layer, a BatchNorm layer, a ReLU activation layer, and a maximum pooling layer. The convolution layer consists of 64 7×7 convolution kernels, and the maximum pooling layer consists of 64 3×3 pooling kernels.

[0017] A backbone branch and a skip-connection branch are set in each residual layer, two residual blocks connected in series are set in the backbone branch of the residual layer, and a backbone branch and a skip-connection branch are set in each residual block; the backbone branch of the residual block is composed of a first convolution layer with a convolution kernel size of 3×3, a first BatchNorm layer, a ReLU activation layer, a second convolution layer with a convolution kernel size of 3×3, and a second BatchNorm layer connected in series; the skip-connection branch of the residual block adopts a convolution layer with a convolution kernel size of 1×1; the backbone branch and the skip-connection branch of the residual block are connected through a position-by-position adder and then connected to the ReLU activation layer; the skip-connection branch of the residual layer adopts a convolution layer with a convolution kernel size of 1×1; the backbone branch and the skip-connection branch of the residual layer are connected through a position-by-position adder;

[0018] The number of convolution kernels in the first to fourth residual layers are 64, 128, 256, and 512 respectively;

[0019] The first to fifth downsampling layers all use the bilinear interpolation method;

[0020] (2b) Construct a decoder consisting of a coloring branch and a parsing branch:

[0021] The structure of the coloring branch in the decoder is constructed as follows: the first residual coloring layer, the first upsampling layer, the second residual coloring layer, the second upsampling layer, the third residual coloring layer, the third upsampling layer, the fourth residual coloring layer, the fourth upsampling layer, the fifth residual coloring layer, and the fifth upsampling layer; a backbone branch and a skip branch are set in each residual coloring layer, and the structure and parameters of the backbone branch and the skip branch of the residual coloring layer are consistent with the backbone branch and the skip branch of the residual layer of the encoder described in step (2a); the backbone branch and the skip branch of the residual coloring layer are connected through a position-by-position adder; the number of convolution kernels of the convolution layers in the first to fifth residual coloring layers are 256, 128, 64, 64, and 64 respectively; the first to fifth upsampling layers all use a bilinear interpolation method;

[0022] The structure of the parsing branch in the decoder is constructed as four parallel branches for parsing features of different sizes, each branch is composed of a residual parsing layer and an upsampling layer connected in series; a backbone branch and a skip branch are set in each residual parsing layer, and the structure and parameters of the backbone branch and the skip branch of the residual parsing layer are consistent with the backbone branch and the skip branch of the residual layer of the encoder described in step (2a); the backbone branch and the skip branch of the residual parsing layer are connected through a position-by-position adder; the number of convolution kernels of the convolution layer in each residual parsing layer is 19; each upsampling layer adopts a bilinear interpolation method; the four branches in the parsing branch are connected through a position-by-position adder;

[0023] (2c) Connect the coloring branch and parsing branch of the encoder and decoder:

[0024] The fifth down-sampling layer of the encoder is connected to the first residual color layer of the color branch of the decoder, the fourth down-sampling layer of the encoder is connected to the first up-sampling layer of the color branch of the decoder through a position-by-position adder, the third down-sampling layer of the encoder is connected to the second up-sampling layer of the color branch of the decoder through a position-by-position adder, the second down-sampling layer of the encoder is connected to the third up-sampling layer of the color branch of the decoder through a position-by-position adder, and the first down-sampling layer of the encoder is connected to the fourth up-sampling layer of the color branch of the decoder through a position-by-position adder;

[0025] Connect the first upsampling layer in the coloring branch of the decoder to the first branch in the parsing branch of the decoder; connect the second upsampling layer to the second branch in the parsing branch of the decoder; connect the third upsampling layer to the third branch in the parsing branch of the decoder; connect the fourth upsampling layer to the fourth branch in the parsing branch of the decoder; complete the construction of the residual structure-multi-task neural network;

[0026] Step 3, training residual structure-multi-task neural network:

[0027] The training set is input into the residual structure-multi-task neural network, and the network parameters are iteratively updated using the gradient descent back-propagation method until the total loss function converges to obtain a trained residual structure-multi-task neural network;

[0028] Step 4: Colorize the face image:

[0029] Input a grayscale face image to be colored into the trained residual structure-multi-task neural network and output the colored face image.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] First, the present invention constructs a network based on a residual structure in building a residual structure-multi-task neural network. A large number of skip branches in the residual structure can filter low-pass information and allow the network to pay more attention to the information it needs to learn, overcoming the defect of limited colorization ability caused by insufficient network learning ability in the prior art. Compared with the previous network based on ordinary convolution operations, the present invention further improves the learning performance and improves the colorization quality of facial images.

[0032] Second, the present invention constructs a decoder including a coloring branch and a parsing branch in building a residual structure-multi-task neural network, and introduces a parsing task to parallel assist the learning of the coloring task, thereby overcoming the deficiency of the prior art in failing to semantically color different areas of the face due to the neglect or improper use of semantic information, thereby enabling the present invention to achieve a more semantically consistent and effective face image coloring effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 It is a schematic diagram of the structure of the residual structure-multi-task neural network of the present invention;

[0035] Figure 3 It is a structural schematic diagram of a residual block and a residual layer in a residual structure-multi-task neural network of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0037] Reference Figure 1 , the implementation steps of the present invention are further described in detail.

[0038] Step 1: Generate a training set.

[0039] The first step is to select at least 24,000 color face images.

[0040] In the second step, each color face image is inferred through the trained face parsing network BiSeNet to generate a parsed image containing the parsed category label corresponding to the color face image.

[0041] The third step is to generate a grayscale image of each color face image through linear mapping.

[0042] The fourth step is to combine each color face image with its corresponding parsed image and grayscale image to generate data samples.

[0043] The fifth step is to form a training set from all data samples.

[0044] Step 2: Build a residual structure-multi-task neural network.

[0045] Reference Figure 2 , the present invention constructs a residual structure-multi-task neural network for further detailed description.

[0046] The present invention constructs a residual structure-multi-task neural network composed of an encoder and a decoder.

[0047] The structure of the constructed encoder is: head, first downsampling layer, first residual layer, second downsampling layer, second residual layer, third downsampling layer, third residual layer, fourth downsampling layer, fourth residual layer, fifth downsampling layer.

[0048] The head consists of a convolutional layer, a BatchNorm layer, a ReLU activation layer, and a maximum pooling layer. The convolutional layer consists of 64 7×7 convolution kernels, and the maximum pooling layer consists of 64 3×3 pooling kernels.

[0049] Reference Figure 3 , the residual block and the residual layer in the residual structure-multi-task neural network constructed by the present invention are further described in detail. The structures of the residual layer, the residual coloring layer and the residual parsing layer in the network are consistent.

[0050] A backbone branch and a skip branch are set in each residual layer, and two residual blocks connected in series are set in the backbone branch of the residual layer, and a backbone branch and a skip branch are set in each residual block. The backbone branch of the residual block is composed of a first convolutional layer with a convolution kernel size of 3×3, a first BatchNorm layer, a ReLU activation layer, a second convolutional layer with a convolution kernel size of 3×3, and a second BatchNorm layer connected in series. The skip branch of the residual block adopts a convolution layer with a convolution kernel size of 1×1, and the backbone branch and the skip branch of the residual block are connected through a position-by-position adder and then connected to the ReLU activation layer. The skip branch of the residual layer adopts a convolution layer with a convolution kernel size of 1×1. The backbone branch and the skip branch of the residual layer are connected through a position-by-position adder.

[0051] The numbers of convolution kernels in the first to fourth residual layers are 64, 128, 256, and 512, respectively.

[0052] The first to fifth downsampling layers all use the bilinear interpolation method.

[0053] The sizes of the output features corresponding to the first downsampling layer, the second downsampling layer, the third downsampling layer, the fourth downsampling layer, and the fifth downsampling layer in the three dimensions of channel, height, and width are 64×128×128, 64×64×64, 128×32×32, 256×16×16, and 512×8×8, respectively.

[0054] The constructed decoder consists of a coloring branch and a parsing branch.

[0055] The structure of the coloring branch in the decoder is constructed as follows: the first residual coloring layer, the first upsampling layer, the second residual coloring layer, the second upsampling layer, the third residual coloring layer, the third upsampling layer, the fourth residual coloring layer, the fourth upsampling layer, the fifth residual coloring layer, and the fifth upsampling layer. A backbone branch and a skip branch are set in each residual coloring layer, and the structure and parameters of the backbone branch and the skip branch of the residual coloring layer are consistent with the backbone branch and the skip branch of the residual layer of the encoder. The backbone branch and the skip branch of the residual coloring layer are connected by a position-by-position adder. The number of convolution kernels of the convolution layers in the first to fifth residual coloring layers are 256, 128, 64, 64, and 64, respectively. The first to fifth upsampling layers all use a bilinear interpolation method.

[0056] The sizes of the output features corresponding to the first upsampling layer, the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer in the coloring branch in the three dimensions of channel, height and width are 256×16×16, 128×32×32, 64×64×64, 64×128×128 and 3×256×256 respectively.

[0057] The structure of the parsing branch in the decoder is four parallel branches, which are used to parse features of different sizes. Each branch is composed of a residual parsing layer and an upsampling layer in series. A backbone branch and a skip branch are set in each residual parsing layer. The structure and parameters of the backbone branch and the skip branch of the residual parsing layer are consistent with the backbone branch and the skip branch of the residual layer of the constructed encoder; the backbone branch and the skip branch of the residual parsing layer are connected through a position-by-position adder. The number of convolution kernels of the convolution layer in each residual parsing layer is 19; each upsampling layer adopts a bilinear interpolation method. The four branches in the parsing branch are connected through a position-by-position adder.

[0058] Connecting the coloring branch and parsing branch of the encoder and decoder obtains the residual structure - a multi-task neural network.

[0059] The fifth downsampling layer of the encoder is connected to the first residual color layer of the decoder color branch, the fourth downsampling layer of the encoder is connected to the first upsampling layer of the decoder color branch through a position-by-position adder, the third downsampling layer of the encoder is connected to the second upsampling layer of the decoder color branch through a position-by-position adder, the second downsampling layer of the encoder is connected to the third upsampling layer of the decoder color branch through a position-by-position adder, and the first downsampling layer of the encoder is connected to the fourth upsampling layer of the decoder color branch through a position-by-position adder.

[0060] The first upsampling layer in the coloring branch of the decoder is connected to the first branch in the parsing branch of the decoder; the second upsampling layer is connected to the second branch in the parsing branch of the decoder; the third upsampling layer is connected to the third branch in the parsing branch of the decoder; the fourth upsampling layer is connected to the fourth branch in the parsing branch of the decoder; and the construction of the residual structure-multi-task neural network is completed.

[0061] Based on the above structure, the features output by the fifth downsampling layer of the encoder are input to the first residual coloring layer and the first upsampling layer of the decoder coloring branch. The output features are consistent with the feature size output by the fourth downsampling layer in the encoder, and then the two are summed position by position to complete the fusion. The fused features are input to the second residual coloring layer and the second upsampling layer of the decoder coloring branch. The output features are consistent with the feature size output by the third downsampling layer in the encoder, and the two are summed position by position to complete the fusion. The fused features are input to the third residual coloring layer and the third upsampling layer of the decoder coloring branch. The output features are consistent with the feature size output by the second downsampling layer in the encoder, and the two are summed position by position to complete the fusion. The fused features are input to the fourth residual coloring layer and the fourth upsampling layer of the decoder coloring branch. The output features are consistent with the feature size output by the first downsampling layer in the encoder, and the two are summed position by position to complete the fusion. The fused features are input to the fifth residual coloring layer and the fifth upsampling layer of the decoder coloring branch, and the output features are the predicted color face images.

[0062] The output of the first upsampling layer in the decoder coloring branch corresponds to the first residual parsing layer and the first upsampling layer of the input decoder parsing branch; the output of the second upsampling layer in the decoder coloring branch corresponds to the second residual parsing layer and the second upsampling layer of the input decoder parsing branch; the output of the third upsampling layer in the decoder coloring branch corresponds to the third residual parsing layer and the third upsampling layer of the input decoder parsing branch; the output of the fourth upsampling layer in the decoder coloring branch corresponds to the fourth residual parsing layer and the fourth upsampling layer of the input decoder parsing branch; then, the outputs of the four parallel branches in the above decoder parsing branches are summed position by position to complete the fusion, which is the predicted parsing image.

[0063] Step 3: Train the residual structure-multi-task neural network.

[0064] The training set is input into the residual structure-multi-task neural network, and the network parameters are iteratively updated using the gradient descent back-propagation method until the total loss function converges to obtain a trained residual structure-multi-task neural network.

[0065] The total loss function is as follows:

[0066]

[0067] Among them, L represents the total loss function, ∑ represents the summation operation, N represents the total number of training set samples, and L c (·) represents the loss function of the decoder colorization branch, L p (·) represents the loss function of the decoder parsing branch, α is the balance coefficient, which is set according to the decoder coloring branch and the decoder parsing branch relative to the application scenario. The grayscale image representing the nth color face image in the training set is output by the coloring branch in the residual structure-multitask neural network. represents the nth color face image in the training set, The grayscale image of the nth color face image in the training set is the parsed image output by the parsing branch in the residual structure-multitask neural network. represents the parsed image of the nth color face image in the training set;

[0068] The L c (·) is calculated by the following formula:

[0069]

[0070] in, Express The average value of the absolute value of the difference is calculated position by position, which is the loss of the decoder coloring branch. Express The average value of the absolute value of the difference is calculated at each position. They represent the features of the output of the mth layer of the VGG19 network trained with the face recognition dataset VggFace2, and the grayscale image of the nth color face image in the training set through the coloring branch in the residual structure-multitask neural network;

[0071] The L p (·) is calculated by the following formula:

[0072]

[0073] Among them, C represents the total number of parsed category labels in the parsed image, c represents the sequence number of the parsed category labels in the parsed image, The grayscale image of the nth color face image in the training set is output by the parsing branch in the residual structure-multitask neural network, and the label of the cth category of the parsed image. represents the label of the cth category in the parsed image of the nth color face image in the training set, and log represents the logarithm operation with base 2.

[0074] Step 4: Colorize the face image.

[0075] Input a grayscale face image to be colored into the trained residual structure-multi-task neural network and output the colored face image.

[0076] The effect of the present invention can be further illustrated by the following simulation experiment.

[0077] 1. Simulation experiment conditions:

[0078] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-8700K CPU, the main frequency is 3.70GHz, the memory is 64G, the graphics card is NVIDIA GEFORCE GTX 1080Ti, and the video memory is 11GB.

[0079] The software platform of the simulation experiment of the present invention is: Ubuntu operating system and python 3.7.

[0080] The data used in the simulation experiment of the present invention comes from all the data in the high-definition face dataset CelebA-HQ, which is generated by NVIDIA based on the face dataset CelebA and contains 30,000 face images with a resolution of 1024×1024.

[0081] 2. Simulation experiment content and result analysis:

[0082] The steps of the simulation experiment of the present invention are: (1) downsampling the data in the CelebA-HQ data set to a resolution of 256×256 by bilinear interpolation; (2) performing inference on each color face image by using the face parsing network BiSeNet trained on the CelebAMask-HQ data set to generate a parsing image corresponding to the color face image and containing a parsing category label; (3) generating a grayscale image of the color face image by using the cvtColor function in the opencv-python library for each color face image; (4) combining each color face image with its corresponding parsing image and grayscale image to generate a data sample; (5) forming all the data samples into an overall data set; and then randomly dividing the overall data set into a training set, a validation set and a test set according to a ratio of 8:1:1. The training set data is input into a residual structure-multitask neural network for training, and a total of 50 epochs are trained. During the training, the performance of the current model is evaluated on the validation set using an evaluation index every 5 epochs. After the training, the performance of the model that achieves the best validation effect is evaluated on the test set.

[0083] The evaluation indicators include peak signal-to-noise ratio PSNR, structural similarity SSIM, and Fréchet initial distance FID, and the formulas are as follows:

[0084]

[0085] Among them, x is the predicted color image, y is the real color image, MAX I is the maximum possible pixel value in the image, which is 255 in the present invention, and L2(·) indicates calculating the average value after calculating the square of the difference at each position.

[0086]

[0087] Among them, x is the predicted color image, y is the real color image, μ x is the mean of x, μ y is the mean of y; is the variance of x, is the variance of y; σ xy is the covariance of x and y; C1=(k1MAX I ) 2 、C2=(k2MAX I ) 2 Avoid division by zero for two constants, where MAX I is the maximum possible pixel value in the image, which is 255 in the present invention, k1=0.01, and k2=0.03 are the default values.

[0088]

[0089] in, Indicates obtaining the activation output of the last layer of the Inception v3 network; is the mean of the distribution of x obtained on Inception v3, is the mean of the distribution of y obtained on Inception v3; is the covariance of the distribution of x obtained on Inception v3, is the covariance of the distribution of y obtained on Inception v3; Trace represents the trace of the calculation matrix.

[0090] In the simulation experiment, the performance comparison of the method of the present invention, the single-task baseline method based on ordinary convolution, and the single-task baseline method based on residual structure on the test set is shown in Table 1 below.

[0091] Table 1. Comparison of coloring performance of the method of the present invention and the baseline method

[0092] PSNR SSIM FID Single-task baseline method based on ordinary convolution 25.59 0.8685 18.11 Single-task baseline method based on residual structure 25.70 0.8702 15.99 Method of the present invention 28.03 0.8786 12.88

[0093] Combined with Table 1, the larger the PSNR and SSIM values ​​are, the higher the quality of colorization is, and the smaller the FID value is, the higher the quality of colorization is. Comparing the first and second rows of Table 1, the use of the residual structure has a significant improvement in the FID indicator that is more in line with human perception, and there are corresponding improvements in other indicators. Comparing the second and third rows of Table 1, the parsing task introduced by the present invention has achieved significant effective improvements in the PSNR, SSIM and FID indicators. The above experimental results show that with the introduction of the residual structure and the parsing task, the performance of the method is gradually improved, which proves the effectiveness of the residual structure and the parallel introduction of the parsing task proposed by the present invention, and can effectively improve the colorization quality of facial images.

Claims

1. A facial image colorization method based on residual structure-analysis task assistance, characterized in that: The network is constructed using the residual structure, and the parsing task is introduced to assist coloring. The specific steps of this method are as follows: Step 1: Generate training set: (1a) Select at least 24,000 color face images; (1b) Each color face image is inferred through the trained face parsing network BiSeNet to generate a parsed image containing a parsed category label corresponding to the color face image; (1c) generating a grayscale image of each color face image through linear mapping; (1d) combining each color face image with its corresponding parsed image and grayscale image to generate a data sample; (1e) All data samples are combined into a training set; Step 2: Build a residual structure-multi-task neural network: (2a) The structure of the encoder is constructed as follows: head, first downsampling layer, first residual layer, second downsampling layer, second residual layer, third downsampling layer, third residual layer, fourth downsampling layer, fourth residual layer, fifth downsampling layer; The head consists of a convolution layer, a BatchNorm layer, a ReLU activation layer, and a maximum pooling layer. The convolution layer consists of 64 7×7 convolution kernels, and the maximum pooling layer consists of 64 3×3 pooling kernels. A backbone branch and a skip-connection branch are set in each residual layer, two residual blocks connected in series are set in the backbone branch of the residual layer, and a backbone branch and a skip-connection branch are set in each residual block; the backbone branch of the residual block is composed of a first convolution layer with a convolution kernel size of 3×3, a first BatchNorm layer, a ReLU activation layer, a second convolution layer with a convolution kernel size of 3×3, and a second BatchNorm layer connected in series; the skip-connection branch of the residual block adopts a convolution layer with a convolution kernel size of 1×1; the backbone branch and the skip-connection branch of the residual block are connected through a position-by-position adder and then connected to the ReLU activation layer; the skip-connection branch of the residual layer adopts a convolution layer with a convolution kernel size of 1×1; the backbone branch and the skip-connection branch of the residual layer are connected through a position-by-position adder; The number of convolution kernels in the first to fourth residual layers are 64, 128, 256, and 512 respectively; The first to fifth downsampling layers all use the bilinear interpolation method; (2b) Construct a decoder consisting of a coloring branch and a parsing branch: The structure of the coloring branch in the decoder is constructed as follows: the first residual coloring layer, the first upsampling layer, the second residual coloring layer, the second upsampling layer, the third residual coloring layer, the third upsampling layer, the fourth residual coloring layer, the fourth upsampling layer, the fifth residual coloring layer, and the fifth upsampling layer; a backbone branch and a skip branch are set in each residual coloring layer, and the structure and parameters of the backbone branch and the skip branch of the residual coloring layer are consistent with the backbone branch and the skip branch of the residual layer of the encoder described in step (2a); the backbone branch and the skip branch of the residual coloring layer are connected through a position-by-position adder; the number of convolution kernels of the convolution layers in the first to fifth residual coloring layers are 256, 128, 64, 64, and 64 respectively; the first to fifth upsampling layers all use a bilinear interpolation method; The structure of the parsing branch in the decoder is constructed as four parallel branches for parsing features of different sizes, each branch is composed of a residual parsing layer and an upsampling layer connected in series; a backbone branch and a skip branch are set in each residual parsing layer, and the structure and parameters of the backbone branch and the skip branch of the residual parsing layer are consistent with the backbone branch and the skip branch of the residual layer of the encoder described in step (2a); the backbone branch and the skip branch of the residual parsing layer are connected through a position-by-position adder; the number of convolution kernels of the convolution layer in each residual parsing layer is 19; each upsampling layer adopts a bilinear interpolation method; Connect the four branches in the parsing branch through a position-by-position adder; (2c) Connect the coloring branch and parsing branch of the encoder and decoder: The fifth down-sampling layer of the encoder is connected to the first residual color layer of the color branch of the decoder, the fourth down-sampling layer of the encoder is connected to the first up-sampling layer of the color branch of the decoder through a position-by-position adder, the third down-sampling layer of the encoder is connected to the second up-sampling layer of the color branch of the decoder through a position-by-position adder, the second down-sampling layer of the encoder is connected to the third up-sampling layer of the color branch of the decoder through a position-by-position adder, and the first down-sampling layer of the encoder is connected to the fourth up-sampling layer of the color branch of the decoder through a position-by-position adder; Connect the first upsampling layer in the coloring branch of the decoder to the first branch in the parsing branch of the decoder; connect the second upsampling layer to the second branch in the parsing branch of the decoder; connect the third upsampling layer to the third branch in the parsing branch of the decoder; connect the fourth upsampling layer to the fourth branch in the parsing branch of the decoder; complete the construction of the residual structure-multi-task neural network; Step 3, training residual structure-multi-task neural network: The training set is input into the residual structure-multi-task neural network, and the network parameters are iteratively updated using the gradient descent back-propagation method until the total loss function converges to obtain a trained residual structure-multi-task neural network; Step 4: Colorize the face image: Input a grayscale face image to be colored into the trained residual structure-multi-task neural network and output the colored face image.

2. The method for colorizing facial images based on residual structure-analysis task assistance according to claim 1, characterized in that: The output features corresponding to the first downsampling layer, the second downsampling layer, the third downsampling layer, the fourth downsampling layer, and the fifth downsampling layer described in step (2a) have dimensions of 64×128×128, 64×64×64, 128×32×32, 256×16×16, and 512×8×8 in the three dimensions of channel, height, and width, respectively.

3. The method for colorizing facial images based on residual structure-analysis task assistance according to claim 1, characterized in that: The output features corresponding to the first upsampling layer, the second upsampling layer, the third upsampling layer, the fourth upsampling layer, and the fifth upsampling layer described in step (2b) have dimensions of 256×16×16, 128×32×32, 64×64×64, 64×128×128, and 3×256×256 in terms of channel, height, and width, respectively.

4. The method for colorizing a face image based on residual structure-analysis task assistance according to claim 1, characterized in that: The total loss function described in step 3 is as follows: Among them, L represents the total loss function, ∑ represents the summation operation, N represents the total number of training set samples, and L c (·) represents the loss function of the decoder colorization branch, L p (·) represents the loss function of the decoder parsing branch, α is the balance coefficient, which is set according to the decoder coloring branch and the decoder parsing branch relative to the application scenario. The grayscale image representing the nth color face image in the training set is output by the coloring branch in the residual structure-multitask neural network. represents the nth color face image in the training set, The grayscale image of the nth color face image in the training set is the parsed image output by the parsing branch in the residual structure-multitask neural network. represents the parsed image of the nth color face image in the training set; The L c (·) is calculated by the following formula: in, Express The average value of the absolute value of the difference is calculated position by position, which is the loss of the decoder coloring branch. Express The average value of the absolute value of the difference is calculated at each position. They represent the features of the output of the mth layer of the VGG19 network trained with the face recognition dataset VggFace2, and the grayscale image of the nth color face image in the training set through the coloring branch in the residual structure-multitask neural network; The L p (·) is calculated by the following formula: Among them, C represents the total number of parsed category labels in the parsed image, c represents the sequence number of the parsed category labels in the parsed image, The grayscale image of the nth color face image in the training set is output by the parsing branch in the residual structure-multitask neural network, and the label of the cth category of the parsed image. represents the label of the cth category in the parsed image of the nth color face image in the training set, and log represents the logarithm operation with base 2.

Images