A method for coloring a face grayscale image based on a double-scale cycle generative adversarial network

Abstract

The application discloses a kind of based on double scale cycle generation confrontation face grayscale image coloring method, its implementation steps are: data collection and pre-processing, model construction, model training and image coloring.The application utilizes cycle generation network, adopts double scale convolution, and fuses CBAM attention module in skip connection to construct face grayscale image coloring model, inputs grayscale image into generator, focuses on important information of the region to be colored, and suppresses unnecessary region mapping learning.PatchGAN is used on the discriminator to achieve more detailed discrimination.High-efficiency end-to-end automatic coloring is realized, which better alleviates the problems of edge color overflow, detail loss and dull coloring commonly existing in existing methods, and finally generates color images with excellent coloring effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a face grayscale image colorization method based on dual-scale cyclic generation adversarial. Background Technology

[0002] In the field of grayscale image colorization, early methods relied primarily on manual pixel-by-pixel coloring, which was not only inefficient but also extremely resource-intensive. Later, the advent and widespread use of computers enabled image processing, greatly simplifying grayscale image colorization.

[0003] Based on the source of image colors, computer-based image colorization can be mainly divided into three categories: colorization methods based on local color expansion, colorization methods based on reference images, and colorization methods based on deep learning. The first two were proposed earlier and usually require user interaction and a large amount of manual operation. The latter appeared later, and users only need to train a network model to achieve end-to-end fully automatic colorization, but the effect is not stable enough and is prone to problems such as color boundary overflow, loss of detail, and monotonous coloring.

[0004] Facial images are one type of image, with relatively clear areas to be colored. Furthermore, due to limitations in early photographic technology, a large number of old black-and-white photographs still exist today. Colorizing these old facial images can greatly restore their vitality. Summary of the Invention

[0005] The purpose of this invention is to provide a face grayscale image coloring method based on dual-scale cyclic generation adversarial method, which can automatically color the input face grayscale image and alleviate the problems of color boundary overflow, detail loss and dull coloring.

[0006] To achieve the above objectives, this invention uses grayscale images as conditional inputs into the generator, extracts shallow, deep, and salient feature information of the image using dual-scale convolution and attention mechanisms, maintains the consistency of the image space through a recurrent generative adversarial network, and finally generates a color image with excellent coloring effect.

[0007] The specific steps are as follows:

[0008] Step 1: Data Collection and Preprocessing: Acquire a large number of color face images and standardize the image size; divide the dataset into training and validation sets; perform data augmentation on the training set data by adding a random flipping operation; use cv library functions to convert the images to the CIE Lab color space and extract the L channel as the input to the model.

[0009] Step 2: Constructing a face grayscale image colorization model: This model uses a recurrent generative network structure, including two pairs of generator-discriminator structures. An improved U-Net is used as the generator, employing a dual-scale convolutional module for feature extraction, increasing the model's adaptability to different scales and extracting multi-dimensional feature information. In skip connections, information with attention weights is extracted through the CBAM attention module and fused with the upsampling stage, focusing on salient regions of the image to be colored and suppressing unnecessary regions. For the discriminator, PatchGAN is used, employing a fully convolutional approach, ultimately outputting a feature map representing the true / false probabilities of multiple regions in the input image, considering the colorization effects of more regions.

[0010] Step 3: Train the face grayscale image colorization model: Use the L-channel grayscale image extracted in Step 1 as the model input, and the remaining ab channels as the model labels. Combine adversarial loss, cycle consistency loss, authentication loss, and grayscale loss, and calculate the final loss function through weighted summation to optimize the model training. The model training follows the strategy of training the discriminator first, and then training the generator.

[0011] Step 4: Colorize the grayscale image of the face: Input the grayscale image of the face to be colored into the trained model, and the colored face image will be output.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] First, this invention combines a recurrent generative network, which maintains the consistency between grayscale and colored images while achieving better fitting results. In the generator, a dual-scale convolutional module with kernels of different sizes is used to extract features from the feature map, adaptively fusing global semantics with local features. Compared to ordinary 3×3 convolutional kernels, this further improves the model's performance and enhances the quality of colored images, resulting in more saturated color images than existing methods.

[0014] Second, this invention integrates an attention mechanism by inserting a CBAM module with a serial structure of channel attention and spatial attention into the generator skip connections. This effectively focuses on salient regions of the feature map and alleviates the problems of color boundary overflow and detail loss that are common in existing methods.

[0015] Third, the model described in this invention is specifically designed for coloring grayscale images of human faces, and it can also achieve good results in coloring some old photos, providing certain practical significance for the restoration of old photos in the color dimension. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention;

[0017] Figure 2 This is a diagram of the cyclic generation network structure of the present invention;

[0018] Figure 3 This is a diagram of the generator network structure of the present invention;

[0019] Figure 4 This is a structural diagram of the dual-scale convolution module of the present invention;

[0020] Figure 5 This is a structural diagram of the CBAM attention module of the present invention;

[0021] Figure 6 This is a diagram of the discriminator network structure of the present invention. Detailed Implementation

[0022] The specific implementation steps of the grayscale image colorization method for human faces of the present invention are described in detail below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, a face grayscale image colorization method based on dual-scale cyclic generative adversarial methods specifically includes the following steps:

[0024] Step 1: Data Collection and Preprocessing

[0025] The first step is to randomly select 30,000 images from the high-resolution face dataset CelebA-HQ.

[0026] The second step is to unify the resolution of all images to 256×256.

[0027] The third step involves dividing the dataset into 90% and 10% segments, resulting in a training set of 27,000 images and a validation set of 3,000 images.

[0028] The fourth step is to convert the image to the CIE Lab color space using cv library functions, extract the L channel as the model input, and the ab channels as label values.

[0029] Fifth step: Before reading the training set, randomly flip the images.

[0030] Step 2: Construct a grayscale image colorization model for the face:

[0031] like Figure 2 As shown, the face grayscale image colorization model is a recurrent generative network structure, including four sub-networks. The G network is the generator, responsible for converting image A into image B, D... B The discriminator is responsible for determining the probability of images being real or fake generated by the G network; the F network is also a generator, responsible for converting image B into images A and D. A It is the discriminator, responsible for determining the probability of an image being real or fake generated by the F network;

[0032] like Figure 3 As shown, the generator uses U-Net as its basic structure. The left side of the U-Net is the encoder part, which extracts image features through downsampling. The resolution of the feature map gradually decreases, while the number of channels gradually increases. The right side is the decoder part, which restores the resolution of the image layer by layer.

[0033] Information is shared between the encoder and decoder through skip connections. When downsampling through the encoder, the features of the image are extracted layer by layer. Due to the existence of skip connections, low-level features can be fused in the upsampling stage, which helps to achieve feature sharing and restore the information loss caused by downsampling.

[0034] The number of convolutional kernels in the generator downsampling stage are 16, 32, 64, 128, and 256, respectively. That is, after passing through the convolution module, the number of image channels changes from 1 to 16, 32, 64, 128, and 256. After two convolutions in the dual-scale convolution module, the number of image channels increases, and then a pooling layer reduces the image resolution to half of its original value.

[0035] The upsampling stage uses transposed convolution to restore the image size, gradually reducing the number of image channels from 256 to 2.

[0036] like Figure 4 As shown, the dual-scale convolution module consists of two convolutional kernels of different sizes, namely 3×3 and 7×7. During the sampling process of the network model, the input feature map is subjected to two convolutional operations of different sizes in parallel, and the final results are fused in a concatenation manner. Subsequently, a 1×1 convolutional kernel is used to achieve effective dimensionality reduction.

[0037] The 3×3 convolutional block consists of a 3×3 convolutional layer, Batch Normalization, and ReLU activation function. The kernel stride is set to 1, and the pixel padding is set to 1.

[0038] The 7×7 convolutional block consists of a 7×7 convolutional layer, Batch Normalization, and ReLU activation function. The kernel stride is set to 1, and the pixel padding is set to 3.

[0039] The stride of the 1×1 convolution kernel is set to 1, and the pixel padding is set to 0.

[0040] The dual-scale convolution module includes two consecutive 3×3 convolution blocks and two 7×7 convolution blocks, and the above convolution operations are in parallel structure.

[0041] Based on the above structure, in the first layer on the left side of the generator, the 256×256×1 input image is transformed into 256×256×16 through two 3×3 convolutional blocks, and then further transformed into 256×256×16 through two 7×7 convolutional blocks. Afterwards, through a concatenation operation, the number of channels is increased to 32, resulting in an image size of 256×256×32. After a 1×1 convolution, the image dimension is reduced, but the width and height remain unchanged at 256×256×16. The subsequent pooling layer reduces the image size to half its original size, i.e., 128×128×16.

[0042] The dual-scale convolution module extracts richer feature information from images, including global and local features, enabling cross-channel feature fusion, increasing the nonlinearity of the model, helping to achieve more complex mapping relationships, and bringing a significant improvement to grayscale image colorization.

[0043] like Figure 5 As shown, the CBAM module (convolutional attention module) is a serial structure consisting of a channel attention module and a spatial attention module.

[0044] In the channel attention module, each channel participates in feature detection, focusing on "what" in the input image is meaningful. The channel attention module first performs pooling operations on the feature map using both max pooling and average pooling, then inputs each into the same shared multilayer perceptron. Finally, the vectors are merged through vector-wise summation to obtain the final channel attention map. The calculation formula for the entire process is shown below. Where σ is the sigmoid function and F is the input feature map. denoted as the weights of the MLP (Multilayer Perceptron), and r is the compression ratio.

[0045]

[0046] The spatial attention module focuses on "where" in the input image is meaningful, i.e., which regions of the image should receive attention. Spatial attention first performs max pooling and average pooling (Avgpool) operations on the feature map, then concatenates the two pools along the channel dimension. Finally, it performs a convolution operation with a 7×7 kernel to obtain the final spatial attention channel map. The calculation formula for the entire process is shown below, where σ is the sigmoid function.

[0047]

[0048] The skip connections contain CBAM (convolutional attention module) attention modules, which share low-level features with certain attention weights. The coloring model will pay more attention to salient regions and learn less color information of unnecessary regions, which improves the coloring effect of the model to a certain extent.

[0049] like Figure 6 As shown, the discriminator uses PatchGAN to distinguish between real and fake images. This structure is a fully convolutional architecture, using a total of 5 convolutional layers. The first three convolutional layers have a kernel size of 4, a stride of 2, and pixel padding of 1, downsampling the image to be judged. After each convolution, the number of channels doubles, while the image size is reduced to half its original size. The kernel size and pixel padding remain unchanged in the last two convolutional layers, with a stride set to 1.

[0050] Given an input image of size 256×256×3, the discriminant network outputs a 30×30 matrix. Each value in the matrix corresponds to the probability of a region of size 70×70 in the input image being true or false.

[0051] The PatchGAN described above considers the real and fake regions of the input image, thus enabling more detailed discrimination.

[0052] Step 3: Training the face grayscale image colorization model.

[0053] Set the total epoch size to 200, the batch size to 1, and use a dynamic learning rate, initializing the learning rate to 0.00002.

[0054] The loss function for the face grayscale image colorization model is as follows:

[0055] Adversarial loss

[0056] Cycle consistency loss Identity authentication loss Grayscale loss

[0057] Where x is the grayscale image, y is the corresponding color image, G and F are the generators, D is the discriminator, and Gray is the grayscale calculation function: Gray(r,g,b)=0.299r+0.587g+0.114b.

[0058] The final total loss function L mix =L GAN +λ1·L c6nsistency +λ2·L id；ntify +λ T ·L gray

[0059] In this embodiment, λ1 = 10, λ2 = 5, and λ3 = 10.

[0060] Step 4: Colorize the grayscale image of the face.

[0061] By inputting the grayscale image to be colored into a trained face grayscale image colorization model, a color image can be obtained.

[0062] Actual testing revealed the following coloring results for the grayscale face image coloring model: the overall coloring effect of the image is good, the face area is given reasonable and full color, the facial features are distinct, and the problems of color boundary overflow, loss of detail and dull coloring are greatly alleviated.

[0063] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A face grayscale image colorization method based on dual-scale cyclic generative adversarial methods, characterized in that: Specifically, the steps include the following: Step 1, Data Collection and Preprocessing: Acquire a large number of color facial images and standardize the image size; divide the dataset into training and validation sets; perform data augmentation on the training set data; Use cv library functions to convert the image to the CIE Lab color space and extract the L channel as input to the model; Step 2: Construct a face grayscale image colorization model: This model adopts a recurrent generative network structure, including two pairs of generator-discriminator pairs; an improved U-Net is used as the generator, and a dual-scale convolution module is used for feature extraction to increase the model's adaptability to information at different scales and extract multi-dimensional feature information; in the skip connections, information with attention weights is extracted through the CBAM attention module and fused with the upsampling stage to focus on the salient regions of the image to be colored and suppress unnecessary regions; in the discriminator, PatchGAN is used, adopting a fully convolutional form, and finally outputting a feature map representing the true and false probability values ​​of multiple regions of the input image, taking into account the colorization effect of more regions; Step 3: Train the face grayscale image colorization model: Use the L-channel grayscale image extracted in Step 1 as the input of the model, and the remaining ab channels as the model's labels; combine adversarial loss, cycle consistency loss, identity authentication loss and grayscale loss, and calculate the final loss function through weighted calculation to optimize the model training, and train the model according to the strategy of training the discriminator first and then the generator. Step 4: Colorize the grayscale image of the face: Input the grayscale image of the face to be colored into the trained model, and the colored face image will be output.

2. The face grayscale image colorization method based on dual-scale cyclic generative adversarial methods as described in claim 1, characterized in that: The recurrent generative network comprises two pairs of generator-discriminator pairs, i.e., four sub-networks. Network G is the generator, responsible for converting image A into image B, and network D... B The discriminator is responsible for determining the probability of images being real or fake generated by the G network; the F network is also a generator, responsible for converting image B into images A and D. A It is the discriminator, responsible for determining the probability of an image being real or fake generated by the F network.

3. The face grayscale image colorization method based on dual-scale cyclic generative adversarial methods as described in claim 1, characterized in that: The dual-scale convolution module uses a combination of 3×3 and 7×7 convolution kernels. After the input feature map is processed by two convolution operations of different sizes, it is fused in the channel dimension. Then, a 1×1 convolution kernel is used for dimensionality reduction to reduce the efficiency reduction caused by the extra model parameters brought by the large convolution kernel.

4. The face grayscale image colorization method based on dual-scale cyclic generative adversarial methods as described in claim 1, characterized in that: In the aforementioned skip connection, the CBAM attention module, which combines channel attention and spatial attention in parallel, focuses on what is meaningful in the feature map and where it is meaningful. It shares useful information from the downsampling stage with the upsampling stage, reduces information loss caused by sampling, suppresses unnecessary information, and improves the coloring effect.

Images