Gray scale image coloring method based on combination of dual-scale convolutional network and attention fusion strategy

Abstract

The invention discloses a grayscale image coloring method based on combination of a dual-scale convolutional network and an attention fusion strategy. According to the method, an attention module is fused on a U-Net network structure with good feature extraction capability, feature extraction is carried out by adopting dual-scale convolution, and features such as texture and resolution are additionally considered on a loss function; according to the method, the problems of color leakage, detail loss and boring coloring of the colored image are relieved, the accuracy and rationality of gray level image coloring are improved, the coloring effect is good, and a new method is provided for color restoration of an old photo.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a grayscale image coloring method combining a dual-scale convolutional network with an attention fusion strategy. Background Art

[0002] Early grayscale image colorization mainly relied on manual pixel-by-pixel processing, which was inefficient and consumed a lot of manpower and material resources. With the popularization of computers, digital image processing has become more convenient. In the fields of medicine, satellite remote sensing, animation, etc., grayscale image colorization can improve the visibility of image features, the classification effect of objects, and the efficiency of painting. The colorization of historical photos can also restore the historical appearance and show humanistic feelings. With the rise of artificial intelligence and deep learning, grayscale image colorization has entered a new research stage. Deep learning can extract rich feature information from images, greatly reduce the amount of manual operations, and achieve good colorization effects. However, grayscale image colorization based on deep learning still has the following defects:

[0003] (1) Grayscale image colorization is a multimodal problem. The same grayscale value may correspond to multiple colors, and deep learning is difficult to achieve 100% accurate restoration.

[0004] (2) Since color is uncertain in some situations, it is necessary to ensure that the generated color image is reasonable overall, which increases the complexity of the calculation.

[0005] (3) Some grayscale image colorization methods based on deep learning may be too smooth because the model is too smooth. For example, grayscale image colorization methods based on convolutional neural networks may lose some information when dealing with complex textures and details, and Deep Koalarization (a colorization method based on deep learning) will cause the model to tend to produce average colors rather than more subtle color changes.

[0006] Therefore, a grayscale image colorization method combining a dual-scale convolutional network with an attention fusion strategy is proposed to improve the accuracy and rationality of grayscale image colorization. Summary of the invention

[0007] In order to solve the problems of color leakage, detail loss and dull coloring in the existing grayscale image coloring methods, the present invention provides a grayscale image coloring method combining a dual-scale convolutional network with an attention fusion strategy. In order to achieve the above purpose, the technical solution adopted by the present invention is: an attention module is integrated into a U-Net network structure with good feature extraction capability, and dual-scale convolution is used for feature extraction. In addition, features such as texture and resolution are additionally considered in the loss function, thereby improving the accuracy and rationality of grayscale image coloring, achieving good coloring effect, and providing a new method for color restoration of old photos.

[0008] The specific steps include: Step 1: Collect face images to build a dataset: Randomly select multiple 1024×1024 high-resolution color RGB face images from the CelebAHQ face dataset, and process the images to modify the resolution to 256×256 to balance computational efficiency and image quality; Preferably, at least 30,000 facial images are selected.

[0009] Step 2: Divide the images processed in step 1 into a training data set, a verification data set, and a test data set; Preferably, 5 / 6 of the face images processed in step 1 are selected as a training data set, 1 / 10 as a verification data set, and 1 / 15 as a test data set, and random flipping is performed in the process of loading images in the training data set, so as to improve the generalization ability of the model and reduce the risk of overfitting; so that the coloring algorithm model can process various types of face images, such as face images with different expressions, lighting conditions and postures; Step 3: Construct a dual-scale convolutional neural network grayscale image colorization model integrating the attention module; The grayscale image colorization model uses U-Net as the basic architecture and improves the original two 3×3 convolution kernels into 3×3 and 7×7 convolution kernels. During the sampling process, the input feature map is convolved twice in parallel with different sizes, the output results are concatenated, and then downscaled through a 1×1 convolution kernel to obtain the output feature map after the double-scale convolution. Batch normalization is performed after each convolution so that the distribution of the data obeys the standard normal distribution with a mean of 0 and a variance of 1. The double-scale convolution improves the network's adaptability to different scales and obtains richer feature information; at the same time, in the process of jump connection, the CBAM attention module based on the serial structure of channel attention and spatial attention is integrated, which can fuse the areas with attention weights of the underlying features with the upsampled feature map, and focus on the salient areas of the grayscale image to be colored during the image colorization process, thereby reducing color leakage and loss of details.

[0010] Step 4: Train the grayscale image colorization model constructed in step 3: The grayscale image colorization model is trained using the multi-scale structural similarity MS_SSIM loss function and the L1 loss function. The loss function is obtained by weighted summing the two. ;in is the Gaussian distribution parameter, and α is the weight value.

[0011] For the training dataset, the batch size is set to 32, the dynamic learning rate is used to set the initial value to 1e-3, and it is decayed to 0.1 times after 30 and 40 iterations. The initial number of channels is set to 16, reducing the number of training parameters.

[0012] Furthermore, considering the colorization of grayscale images, the L1 loss function, which is more sensitive to color, should have a higher weight when coloring. The α values ​​were taken as 0.84, 0.16, 0.08, etc. for comparative experiments. Finally, 0.08 was taken as the α value for better coloring effect.

[0013] Step 5: Input the data in the validation data set and the test data set into the model trained in step 4 to evaluate and verify the model.

[0014] Step 6: Input any grayscale image to be colored into the grayscale image colorization model trained in step 4, and output the colored grayscale image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 This is the improved U-Net colored network structure diagram of the present invention; Figure 3 Comparison chart of different α values ​​for the loss function; Figure 4 This is a comparison chart of different coloring methods; Figure 5 A comparison of the coloring effects of the ablation experiment; It should be noted that due to Figure 2-Figure 5 Regarding human face images, in order to avoid infringement of human portrait rights, the eyes of the human face in the image are mosaiced; comparing the coloring effects of other parts can also reflect the coloring effect of the present invention. DETAILED DESCRIPTION

[0016] The main feature of face images is that the area to be colored is prominent, covering hair, face, facial features, etc. The colorization method proposed in this implementation is mainly designed for face datasets, and can better achieve the grayscale face colorization task.

[0017] like Figure 1 As shown in FIG. 1 , a grayscale image colorization method using a dual-scale convolutional network combined with an attention fusion strategy specifically includes the following steps: Step 1: Use the open source CelebA-HQ face dataset to select 30,000 high-definition color face images with an original resolution of 1024 × 1024. Through data preprocessing, the image resolution selected in step 1 is modified to 256 × 256, which can greatly reduce the network training time while maintaining the colorization effect.

[0018] Step 2: Randomly select 25,000 face images as training data set, 3,000 images as validation data set, and 2,000 images as test data set.

[0019] When reading an image, the library function is used to convert the image represented in the RGB space into the CIE Lab color space. The L brightness channel information will be used as the input of the network, and the a and b chromaticity channel information will be predicted through the network. Finally, the three channels will be fused together to obtain the predicted color image.

[0020] In addition, random flipping is added during the reading of training set images, which can effectively enhance the data, thereby improving the performance and generalization ability of the model and reducing the risk of overfitting.

[0021] Step 3: Build a dual-scale convolutional neural network grayscale image colorization model integrating the attention module: In the process of face image colorization, the area where the face is located has a higher weight, which will inevitably cause some interfering noise in the fused features, thus affecting the colorization effect. Therefore, in order to make the network focus on important areas, the CBAM attention module is introduced to adjust the weights of different regional positions in each feature layer while suppressing noise, and learn the interdependence between each channel.

[0022] like Figure 2 As shown in the figure, the grayscale image colorization model uses U-Net as the basic architecture and improves the original convolution module into a dual-scale convolution module that combines 3×3 and 7×7 convolution kernels to extract features layer by layer. During the network sampling process, the input feature map is subjected to two convolution operations of different sizes in parallel, and the final results are fused together in a cascade (Concatenation) manner. A 1×1 convolution kernel is used to achieve effective dimensionality reduction. After this convolution operation, the output feature map after the dual-scale convolution is obtained. The fusion of features obtained by convolution of two different sizes of convolution kernels helps the neural network obtain richer feature information of different scales, which not only increases the network's understanding of the global semantic information of the image to be colored, but also allows the network to focus on the local area of ​​the image.

[0023] At the same time, in the jump connection process, the CBAM attention module based on the serial structure of channel attention and spatial attention is integrated, which can fuse the areas with attention weights of the underlying features with the upsampled feature map, and focus on the salient areas of the grayscale image to be colored during the image colorization process, thereby reducing color leakage and loss of details.

[0024] The structural details of the dual-scale convolution module are shown in Table 1. Assume that the width and height of the input feature map are 256, the number of channels is 3, and the number of output channels is 16.

[0025] Table 1 Step 4: training the grayscale image colorization model constructed in step 3; In the loss function design, the L1 absolute value loss function compares the difference pixel by pixel, and calculates the sum of the absolute values ​​of the differences between the two. The formula is as follows: ;Although the L1 loss function has good robustness, it ignores the semantic information of the global image and has deficiencies in image edges and processing details.

[0026] Therefore, MS_SSIM_L1 is used as the loss function of the coloring network. SSIM is the abbreviation of structural similarity, which evaluates the similarity of two images from three dimensions: contrast, brightness, and structure. For two images x and y, the calculation formula is as follows: ;

[0027] Among them, u x and u y represents the average value of x and y, σ x and σ y represents the standard deviation, σ xy represents the covariance of the two. c1 and c2 represent constants. Thus, we can get: ; Where P represents the image set, N is the number of images in the image set, and SSIM(p) represents the structural similarity index of each image. MS_SSIM is multi-scale structural similarity, which combines multiple scales on the basis of structural similarity and has greater flexibility.

[0028] ; ; in represents the brightness multi-scale structural similarity of image p, represents the weight parameter, represents the difference in contrast and structure of image p at scale j, Represents the image after the transformation.

[0029] L1 can keep brightness and color unchanged. LMS_SSIM is consistent with human visual perception and takes resolution as an indicator. It performs well on the edges and details of the image. By taking the weighted sum of the two, we get the final loss function Lmix in this chapter: ;in is the Gaussian distribution parameter, and α is the weight value. Figure 3 As shown in the figure, the α values ​​are taken as 0.84, 0.16, 0.08, etc. for comparative experiments. Finally, 0.08 is taken as the α value for better coloring effect, which can make the L1 loss function, which is more sensitive to color, occupy a larger weight, which is more conducive to the coloring of grayscale images.

[0030] For the training data set, the number of training iterations was set to 50, the batch size was set to 32, the initial number of channels was set to 16, and a dynamic learning rate was used to initialize the learning rate to 1e-3. After 30 and 40 epochs, the learning rate decayed to 0.1 times the original value.

[0031] Step 5: Model verification: input the data in the test data set into the model to test the model, and evaluate the model from the qualitative visual coloring effect and quantitative evaluation indicators; input the data in the verification data set into the model to test the model, and after completing the evaluation and testing of the model, output the model.

[0032] Step 6: Input any image to be colored into the model evaluated and tested in step 5, and output the colored image.

[0033] In order to demonstrate the colorization effect of this method, several existing colorization methods were selected for comparison. The colorization networks proposed by Zhang, Pix2Pix, Johnson and Su were used as the control group. They were trained on the CelebA face image dataset and tested on the same test set. Finally, a comparison was made from two perspectives: qualitative visual colorization effect and quantitative evaluation indicators.

[0034] like Figure 4 As shown, through visual comparison, it can be found that this colorization method can better solve the problem of face image colorization. The color and saturation of the overall image are closer to the original image, and to a certain extent, it also alleviates the problem of color penetration and lack of details in the area to be colored.

[0035] In addition to the qualitative analysis, the evaluation indicators of each coloring method were quantitatively analyzed, and the comparison results are shown in Table 2: Table 2 The above colorization results are all performed on the test set constructed in step 2, and the image style and quality are similar to those of the training set and the validation set. In addition, by using colorization models trained with different algorithms, the black-and-white photos of historical figures in the Historical WikiDataset are colorized, proving that this colorization method can provide a solution for the color dimension of old photo restoration.

[0036] In order to verify the impact of using modules and loss functions on the coloring network effect, a simulation experiment of the model was carried out with Python in Anaconda3 under the running environment of 11th Gen Intel(R)Core(TM) i7-11700 @ 2.50 GHz 32GB memory.

[0037] The specific ablation experiments are as follows: Experiment A: original U-Net network + L1 loss function; Experiment B: original U-Net network + MS_SSIM_L1 loss function; Experiment C: U-Net + dual-scale convolution module + MS_SSIM_L1 loss function; Experiment D: U-Net + CBAM + MS_SSIM_L1 loss function; Experiment E (this method): U-Net + dual-scale convolution module + CBAM + MS_SSIM_L1 loss function. Experiments A and B can verify the influence of MS_SSIM_L1 loss function, and experiments B, C, D, and E can respectively verify the influence of dual-scale convolution module, CBAM attention module and the combination of the two modules on the coloring effect. The ablation experiments were all carried out in the same dataset, and the comparison results are shown in Figure 2. Figure 5 As shown in Table 3:

[0038] Table 3 From the data in the table above, we can see that the use of the MS_SSIM_L1 loss function improves PSNR by 0.543dB and SSIM by 0.4%, which also shows that it is more conducive to generating higher quality images than the L1 loss function. In addition, the addition of the dual-scale convolution module and the CBAM attention module has also improved the coloring performance to a certain extent, which once again verifies that the network improved by this colorization method has a certain improvement effect on colorization.

[0039] The present invention proposes a grayscale image colorization method which combines a dual-scale convolutional network with an attention fusion strategy. A large number of experiments are carried out on the CelebA face dataset. By comparing with the existing colorization methods, it is found that the algorithm can achieve higher colorization effects on the test set and the dataset of old photos of historical figures. The ablation experiment also verifies that the added module can bring a certain improvement effect to the original network.

Claims

1. A grayscale image colorization method combining a dual-scale convolutional network with an attention fusion strategy, characterized in that: The specific steps include: Step 1: Collect face images to build a dataset: Randomly select multiple 1024×1024 high-resolution color RGB face images from the CelebAHQ face dataset, and process the images to modify the resolution to 256×256 to balance computational efficiency and image quality; Step 2: Divide the images processed in step 1 into a training data set, a verification data set, and a test data set; Step 3: Construct a dual-scale convolutional neural network grayscale image colorization model integrating the attention module; The grayscale image colorization model uses U-Net as the basic architecture and improves the original two 3×3 convolution kernels into 3×3 and 7×7 convolution kernels. During the sampling process, the input feature map is convolved twice in parallel with different sizes, the output results are concatenated, and then downscaled through a 1×1 convolution kernel to obtain the output feature map after double-scale convolution. Batch normalization is performed after each convolution to make the data distribution obey the standard normal distribution with a mean of 0 and a variance of 1. Double-scale convolution improves the network's adaptability to different scales and obtains richer feature information. At the same time, in the process of skip connection, the CBAM attention module based on the serial structure of channel attention and spatial attention is integrated, which can fuse the areas with attention weights of the underlying features with the upsampled feature map, and focus on the salient areas of the grayscale image to be colored during the image colorization process, thereby reducing color leakage and loss of details; Step 4: Train the grayscale image colorization model constructed in step 3: The grayscale image colorization model is trained using the multi-scale structural similarity MS_SSIM loss function and the L1 loss function. The loss function is obtained by weighted summing the two. ;in is the Gaussian distribution parameter, α is the weight value; For the training dataset, the batch size is set to 32, the initial value is set to 1e-3 using a dynamic learning rate, and it decays to 0.1 times after 30 and 40 iterations. The initial number of channels is set to 16, reducing the number of training parameters; Step 5: Input the data in the validation data set and the test data set into the model trained in step 4 to evaluate and verify the model; Step 6: Input any grayscale image to be colored into the grayscale image coloring model evaluated and verified in step 5, and output the colored grayscale image.

2. The grayscale image colorization method of the dual-scale convolutional network combined with the attention fusion strategy as claimed in claim 1, characterized in that: In step 1, at least 30,000 1024×1024 high-resolution color RGB face images are randomly selected from the CelebAHQ face dataset; in step 2, 5 / 6 of the face images processed in step 1 are used as the training dataset, 1 / 10 as the validation dataset, and 1 / 15 as the test dataset.

3. The grayscale image colorization method using a dual-scale convolutional network combined with an attention fusion strategy as claimed in claim 1 or 2, characterized in that: In step 2, random flipping is performed during the process of loading images in the training dataset to effectively enhance the data and improve the generalization ability of the model.

4. The grayscale image colorization method of the dual-scale convolutional network combined with the attention fusion strategy as claimed in claim 1, characterized in that: In step 4, considering the colorization of the grayscale image, the loss function adopts the weighted sum of the multi-scale structural similarity MS_SSIM loss function and the L1 loss function; the L1 loss function, which is more sensitive to color, should have a higher weight when coloring. Different α values ​​are compared and experiments are carried out, and finally 0.08 is taken as the α value.

Images